Provision SR-IOV Interfaces using the CLI¶

SR-IOV interfaces must be provisioned on at least one host before using SR-IOV network attachments in a container.

Note

For the purpose of enabling and using needVhostNet, SR-IOV needs to be configured on a worker host.

By default, a Kubernetes container is started with a default network virtual interface for cluster networking. For accelerated networking between containers or external networks, additional SR-IOV backed interfaces (network attachments) can be added to the container.

You can use the CLI to provision SR-IOV interfaces on a StarlingX system.

The SR-IOV device plugin discovers and advertises SR-IOV network virtual functions (VFs) in a Kubernetes host. To enable the device plugin, all hosts on which accelerated networking pods will be enabled should have the sriovdp label applied.

The SR-IOV device plugin can also expose the /dev/vhost-net device to a container by using the needVhostNet parameter. To enable it, apply the sriovdp-vhostnet label.

For information about creating and using network attachments, see StarlingX User Tasks: Create Network Attachment Definitions and StarlingX User Tasks: Use Network Attachment Definitions in a Container.

Prerequisites

You must create data networks before you can provision the SR-IOV interfaces. See StarlingX Data Networks: Add Data Networks using the CLI.

Procedure

Lock the host to which you will assign the label.

Note

AIO-SX hosts do not need to be locked to provision an SR-IOV interface and assign it to a data network.

For example:
```
~(keystone_admin)$ system host-lock compute-0
```

Use the host-label-assign command to assign the sriovdp label to the node.

For example, to set sriovdp on worker-0 you would do the following:

~(keystone_admin)$ system host-label-assign compute-0 sriovdp=enabled
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| uuid        | a2c5d21b-f91a-4b8b-8dbc-d40b2f3bdaa9 |
| host_uuid   | 772df330-6a42-4b8e-9a18-ae9a9f3f2336 |
| label_key   | sriovdp                              |
| label_value | enabled                              |
+-------------+--------------------------------------+

Optional step: To expose the vhost-net device to a container, use the host-label-assign command to assign the sriovdp-vhostnet label to the node.

For example, to set sriovdp-vhostnet on worker-0:

[sysadmin@controller-0 ~(keystone_admin)]$ system host-label-assign compute-0 sriovdp-vhostnet=enabled
+-------------+--------------------------------------+
| Property    | Value                                |
+-------------+--------------------------------------+
| uuid        | d2eff38a-889d-46e2-ae4f-43acb1c957a9 |
| host_uuid   | 8c28fe10-8849-4597-8a5d-89a97ea8febf |
| label_key   | sriovdp-vhostnet                     |
| label_value | enabled                              |
+-------------+--------------------------------------+

Identify the interfaces to be configured.

To list all interfaces, use the system host-if-list command and include the -a flag.

For example:

~(keystone_admin)$ system host-if-list -a compute-0
+---...+----------+----------+...+---------------+...+-------------------+
| uuid | name     | class    |   | ports         |   | data networks     |
+---...+----------+----------+...+---------------+...+-------------------+
| 68...| ens787f3 | None     |   | [u'ens787f3'] |   | []                |
| 79...| data0    | data     |   | [u'ens787f0'] |   | [u'group0-data0'] |
| 78...| cluster0 | platform |   | []            |   | []                |
| 89...| ens513f3 | None     |   | [u'ens513f3'] |   | []                |
| 97...| ens803f1 | None     |   | [u'ens803f1'] |   | []                |
| d6...| pxeboot0 | platform |   | [u'eno2']     |   | []                |
| d6...| mgmt0    | platform |   | []            |   | []                |
+---...+----------+----------+...+---------------+...+-------------------+

Use the system host-if-modify and interface-datanetwork-assign commands to configure the SR-IOV interfaces.

~(keystone_admin)$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov -N <numvfs> --vf-driver=<drivername> compute-0 <ethname>
~(keystone_admin)$ system interface-datanetwork-assign compute-0 <interface> <networks>

where the following options are available:

hostname

This is the host name or ID of the compute (worker) node.

numvfs

The number of virtual functions to enable on the device.

drivername

An optional virtual function driver to use. Valid choices are ‘vfio’ and ‘netdevice’. The default value is netdevice, which will cause SR-IOV virtual function interfaces to appear as kernel network devices in the container. A value of ‘vfio’ will cause the device to be bound to the vfio-pci driver. Vfio based devices will not appear as kernel network interfaces, but may be used by DPDK based applications.

Note

Applications backed by Mellanox NICs should use the netdevice VF driver.
If a mix of netdevice and vfio based containers is required, a separate data network should be created for each type.

ethname

The name or UUID of the Ethernet interface to use.

networks

A list of data networks that are attached to the interface, delimited by quotes and separated by commas; for example, “data-net-a,data-net-b”. To specify a single data network, omit the quotes.

For example, to attach Ethernet interface ens787f3 to data network datanet-a configured with 16 virtual functions, do the following:

~(keystone_admin)$ system host-if-modify -m 1500 -n sriov1 -c pci-sriov -N 16 compute-0 ens787f3
+----------------+--------------------------------------+
| Property        | Value                                |
+-----------------+--------------------------------------+
| ifname          | sriov1                               |
| iftype          | ethernet                             |
| ports           | [u'ens787f3']                        |
| imac            | 68:05:ca:3a:18:4b                    |
| imtu            | 1500                                 |
| ifclass         | pci-sriov                            |
| networks        |                                      |
| aemode          | None                                 |
| schedpolicy     | None                                 |
| txhashpolicy    | None                                 |
| uuid            | 68544dbc-244c-4d24-a629-ca8e4543c6f8 |
| ihost_uuid      | 54c28c7c-5b53-4191-97b5-9ddde3cbec81 |
| vlan_id         | None                                 |
| uses            | []                                   |
| used_by         | []                                   |
| created_at      | 2019-05-14T00:12:56.673418+00:00     |
| updated_at      | 2019-05-14T00:16:56.864997+00:00     |
| sriov_numvfs    | 16                                   |
| sriov_vf_driver | None                                 |
| accelerated     | [True]                               |
+-----------------+--------------------------------------+

~(keystone_admin)$ system interface-datanetwork-assign compute-0 sriov1 datanet-a

Note

It is not possible to have multiple data networks assigned to the same SR-IOV interface. If you want to use multiple data network, it is possible to create a new SR-IOV interface on top of another and have different data networks assigned to each of them.

Unlock the host.

Note

AIO-SX hosts do not need to be locked to provision an SR-IOV interface and assign it to a data network.
```
~(keystone_admin)$ system host-unlock compute-0
```

Postrequisites

When launching an SR-IOV-enabled Kubernetes deployment, pods will only be scheduled on hosts with the sriovdp label enabled.