Testing SRIOV functionality

Testing SRIOV functionality

Following the steps explained on How to configure SR-IOV ports make sure that you have already created and applied a NetworkAttachmentDefinition containing a sriov driverType. Also make sure that sriov-device-plugin is enabled on the nodes.

NetworkAttachmentDefinition containing a sriov driverType might look like:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: "net-sriov"
  annotations:
    openstack.org/kuryr-config: '{
    "subnetId": "88d0b025-2710-4f02-a348-2829853b45da",
    "driverType": "sriov"
    }'

Here 88d0b025-2710-4f02-a348-2829853b45da is an id of precreated subnet that is expected to be used for SR-IOV ports:

$ neutron subnet-show 88d0b025-2710-4f02-a348-2829853b45da
+-------------------+--------------------------------------------------+
| Field             | Value                                            |
+-------------------+--------------------------------------------------+
| allocation_pools  | {"start": "192.168.2.2", "end": "192.168.2.254"} |
| cidr              | 192.168.2.0/24                                   |
| created_at        | 2018-11-21T10:57:34Z                             |
| description       |                                                  |
| dns_nameservers   |                                                  |
| enable_dhcp       | True                                             |
| gateway_ip        | 192.168.2.1                                      |
| host_routes       |                                                  |
| id                | 88d0b025-2710-4f02-a348-2829853b45da             |
| ip_version        | 4                                                |
| ipv6_address_mode |                                                  |
| ipv6_ra_mode      |                                                  |
| name              | sriov_subnet                                     |
| network_id        | 2f8b9103-e9ec-47fa-9617-0fb9deacfc00             |
| project_id        | 92a4d7734b17486ba24e635bc7fad595                 |
| revision_number   | 2                                                |
| service_types     |                                                  |
| subnetpool_id     |                                                  |
| tags              |                                                  |
| tenant_id         | 92a4d7734b17486ba24e635bc7fad595                 |
| updated_at        | 2018-11-21T10:57:34Z                             |
+-------------------+--------------------------------------------------+

1. Create deployment definition <DEFINITION_FILE_NAME> with one SR-IOV interface (apart from default one). Deployment definition file might look like:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-sriov
spec:
  replicas: 1
  template:
    metadata:
      name: nginx-sriov
      labels:
        app: nginx-sriov
      annotations:
        k8s.v1.cni.cncf.io/networks: net-sriov
    spec:
      containers:
      - name: nginx-sriov
        image: nginx
        resources:
          requests:
            intel.com/sriov: '1'
            cpu: "1"
            memory: "512Mi"
          limits:
            intel.com/sriov: '1'
            cpu: "1"
            memory: "512Mi"

Here net-sriov is the name of NetworkAttachmentDefinition created before.

  1. Create deployment with the following command:
$ kubectl create -f <DEFINITION_FILE_NAME>
  1. Wait for the pod to get to Running phase.
$ kubectl get pods
NAME                                    READY   STATUS      RESTARTS    AGE
nginx-sriov-558db554d7-rvpxs            1/1     Running     0           1m

4. If your image contains iputils (for example, busybox image), you can attach to the pod and check that the correct interface has been attached to the Pod.

$ kubectl get pod
$ kubectl exec -it nginx-sriov-558db554d7-rvpxs -- /bin/bash
$ ip a

You should see default and eth1 interfaces. eth1 is the SR-IOV VF interface.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
3: eth0@if43: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether fa:16:3e:1a:c0:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.9/24 scope global eth0
        valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe1a:c043/64 scope link
        valid_lft forever preferred_lft forever
13: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:b3:2e:70 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.6/24 scope global eth1
        valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fea8:55af/64 scope link
        valid_lft forever preferred_lft forever

4.1. Alternatively you can login to k8s worker and do the same from the host system. Use the following command to find out ID of running SR-IOV container:

$ docker ps

Suppose that ID of created container is eb4e10f38763. Use the following command to get PID of that container:

$ docker inspect --format {{.State.Pid}} eb4e10f38763

Suppose that output of previous command is bellow:

$ 32609

Use the following command to get interfaces of container:

$ nsenter -n -t 32609 ip a

You should see default and eth1 interfaces. eth1 is the SR-IOV VF interface.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
        valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
        valid_lft forever preferred_lft forever
3: eth0@if43: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether fa:16:3e:1a:c0:43 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.0.9/24 scope global eth0
        valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe1a:c043/64 scope link
        valid_lft forever preferred_lft forever
13: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether fa:16:3e:b3:2e:70 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.6/24 scope global eth1
        valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fea8:55af/64 scope link
        valid_lft forever preferred_lft forever

In our example sriov interface has address 192.168.2.6

  1. Use neutron CLI to check the port with exact address has been created on neutron:
$ openstack port list | grep 192.168.2.6

Suppose that previous command returns a list with one openstack port that has ID 545ec21d-6bfc-4179-88c6-9dacaf435ea7. You can see its information with the following command:

$ openstack port show 545ec21d-6bfc-4179-88c6-9dacaf435ea7
+-----------------------+----------------------------------------------------------------------------+
| Field                 | Value                                                                      |
+-----------------------+----------------------------------------------------------------------------+
| admin_state_up        | UP                                                                         |
| allowed_address_pairs |                                                                            |
| binding_host_id       | novactl                                                                    |
| binding_profile       |                                                                            |
| binding_vif_details   | port_filter='True'                                                         |
| binding_vif_type      | hw_veb                                                                     |
| binding_vnic_type     | direct                                                                     |
| created_at            | 2018-11-26T09:13:07Z                                                       |
| description           |                                                                            |
| device_id             | 7ab02cf9-f15b-11e8-bdf4-525400152cf3                                       |
| device_owner          | compute:kuryr:sriov                                                        |
| dns_assignment        | None                                                                       |
| dns_name              | None                                                                       |
| extra_dhcp_opts       |                                                                            |
| fixed_ips             | ip_address='192.168.2.6', subnet_id='88d0b025-2710-4f02-a348-2829853b45da' |
| id                    | 545ec21d-6bfc-4179-88c6-9dacaf435ea7                                       |
| ip_address            | None                                                                       |
| mac_address           | fa:16:3e:b3:2e:70                                                          |
| name                  | default/nginx-sriov-558db554d7-rvpxs                                       |
| network_id            | 2f8b9103-e9ec-47fa-9617-0fb9deacfc00                                       |
| option_name           | None                                                                       |
| option_value          | None                                                                       |
| port_security_enabled | False                                                                      |
| project_id            | 92a4d7734b17486ba24e635bc7fad595                                           |
| qos_policy_id         | None                                                                       |
| revision_number       | 5                                                                          |
| security_groups       | 1e7bb965-2ad5-4a09-a5ac-41aa466af25b                                       |
| status                | DOWN                                                                       |
| subnet_id             | None                                                                       |
| updated_at            | 2018-11-26T09:13:07Z                                                       |
+-----------------------+----------------------------------------------------------------------------+

The port would have the name of the pod, compute::kuryr::sriov for device owner and ‘direct’ vnic_type. Verify that IP and MAC addresses of the port match the ones on the container. Currently the neutron-sriov-nic-agent does not properly detect SR-IOV ports assigned to containers. This means that direct ports in neutron would always remain in DOWN state. This doesn’t affect the feature in any way other than cosmetically.

Creative Commons Attribution 3.0 License

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.