SR-IOV guide for OVN

The purpose of this page is to describe how SR-IOV works with OVN. Prior to reading this document, it is recommended to first read the basic guide for SR-IOV.

External ports

In order for SR-IOV to work with the Neutron driver we are leveraging the external ports feature from the OVN project. When virtual machines are booted on hypervisors supporting SR-IOV nics, the local ovn-controllers are unable to reply to the VM’s DHCP, internal DNS, IPv6 router solicitation requests, etc… since the hypervisor is bypassed in the SR-IOV case. OVN then introduced the idea of having external ports which are able to reply on behalf of those VM ports external to the hypervisor that they are running on.

The OVN Neutron driver will create a port of the type external for ports with the following VNICs set:

  • direct

  • direct-physical

  • macvtap

Also, ports of the type external will be scheduled on the gateway nodes (controller or networker nodes) in HA mode by the OVN Neutron driver. Check the OVN Database information section for more information.

Environment setup for OVN SR-IOV

There are a very few differences between setting up an environment for SR-IOV for the OVS and OVN Neutron drivers. As mentioned at the beginning of this document, the instructions from the the basic guide for SR-IOV are required for getting SR-IOV working with the OVN driver.

The only differences required for an OVN deployment are:

  • When configuring the mechanism_drivers in the ml2_conf.ini file we should specify ovn driver instead of the openvswitch driver

  • Disabling the Neutron DHCP agent

  • Deploying the OVN Metadata agent on the gateway nodes (controller or networker nodes)

OVN Database information

Before getting into the ports information, the previous sections talks about gateway nodes, the OVN Neutron driver identifies a gateway node by the ovn-cms-options=enable-chassis-as-gw and ovn-bridge-mappings options in the external_ids column from the Chassis table in the OVN Southbound database:

$ ovn-sbctl list Chassis
_uuid               : 12b13aff-a821-4cde-a4ac-d9cf8e2c91bc
external_ids        : {ovn-cms-options=enable-chassis-as-gw, ovn-bridge-mappings="public:br-ex", ...}
hostname            : controller-0
name                : "1a462946-ccfd-46a6-8abf-9dca9eb558fb"
...

For more information about both of these options, please take a look at the ovn-controller documentation.

These options can be set by running the following command locally on each gateway node (note, the ovn-bridge-mappings will need to be adapted to your environment):

$ ovs-vsctl set Open_vSwitch . external-ids:ovn-cms-options=\"enable-chassis-as-gw\" external-ids:ovn-bridge-mappings=\"public:br-ex\"

As mentioned in the External ports section, every time a Neutron port with a certain VNIC is created the OVN driver will create a port of the type external in the OVN Northbound database. These ports can be found by issuing the following command:

$ ovn-nbctl find Logical_Switch_Port type=external
_uuid               : 105e83ae-252d-401b-a1a7-8d28ec28a359
ha_chassis_group    : [43047e7b-4c78-4984-9788-6263fcc69885]
type                : external
...

The ha_chassis_group column indicates which HA Chassis Group that port belongs to, to find that group do:

# The UUID is the one from the ha_chassis_group column from
# the Logical_Switch_Port table
$ ovn-nbctl list HA_Chassis_Group 43047e7b-4c78-4984-9788-6263fcc69885
_uuid               : 43047e7b-4c78-4984-9788-6263fcc69885
external_ids        : {}
ha_chassis          : [3005bf84-fc95-4361-866d-bfa1c980adc8, 72c7671e-dd48-4100-9741-c47221672961]
name                : default_ha_chassis_group

Note

For now, the OVN driver only has one HA Chassis Group created called default_ha_chassis_group. All external ports in the system will belong to this group.

The chassis that are members of the default_ha_chassis_group HA Chassis Group are listed in the ha_chassis column. Those are the gateway nodes (controller or networker nodes) in the deployment and it’s where the external ports will be scheduled. In order to find which gateway node the external ports are scheduled on use the following command:

# The UUIDs are the UUID members of the HA Chassis Group
# (ha_chassis column from the HA_Chassis_Group table)
$ ovn-nbctl list HA_Chassis 3005bf84-fc95-4361-866d-bfa1c980adc8 72c7671e-dd48-4100-9741-c47221672961
_uuid               : 3005bf84-fc95-4361-866d-bfa1c980adc8
chassis_name        : "1a462946-ccfd-46a6-8abf-9dca9eb558fb"
external_ids        : {}
priority            : 32767

_uuid               : 72c7671e-dd48-4100-9741-c47221672961
chassis_name        : "a0cb9d55-a6da-4f84-857f-d4b674088c8c"
external_ids        : {}
priority            : 32766

Note the priority column from the previous command, the chassis with the highest priority from that list is the chassis that will have the external ports scheduled on it. In our example above, the chassis with the UUID 1a462946-ccfd-46a6-8abf-9dca9eb558fb is the one.

Whenever the chassis with the highest priority goes down, the ports will be automatically scheduled on the next chassis with the highest priority which is alive. So, the external ports are HA out of the box.

Known limitations

The current SR-IOV implementation for the OVN Neutron driver has a few known limitations that should be addressed in the future:

  1. At the moment, all external ports will be scheduled on a single gateway node since there’s only one HA Chassis Group for all of those ports.

  2. Routing on VLAN tenant network will not work with SR-IOV. This is because the external ports are not being co-located with the logical router’s gateway ports, for more information take a look at bug #1875852.