Wallaby Series Release Notes

18.6.0-151

New Features

  • Address scope is now added to all OVN LSP port registers in the northbound. Northd then writes the address scope from the northbound to the southbound so it can be used there by the ovn-bgp-agent.

  • OVN mechanism driver refuses to bind a port to a dead agent.

Known Issues

  • The high availability of metadata service on isolated networks is limited or non-existent. IPv4 metadata is redundant when the DHCP agent managing it is redundant, but recovery is tied to the renewal of the DHCP lease, making most recoveries very slow. IPv6 metadata is not redundant at all as the IPv6 metadata address can only be configured in a single place at a time as it is link-local. Multiple agents trying to configure it will generate an IPv6 duplicate address detection failure.

    Administrators may observe the IPv6 metadata address in “dadfailed” state in the DHCP namespace for this reason, which is only an indication it is not highly available. Until a redesign is made to the isolated metadata service there is not a better deployment option. See bug 1953165 for information.

  • Until the OVN bug (https://bugzilla.redhat.com/show_bug.cgi?id=2162756) is fixed, setting the “reside-on-redirect-chassis” to true for the logical router port associated to vlan provider network is needed. This workaround makes the traffic centrallized, but not tunneled, through the node with the gateway port, thus avoiding MTU issues.

  • The redirect-type=bridged option is only used if all the tenant networks connected to the router are of type VLAN or FLAT. In this case their traffic will be distributed. However, if there is a mix of VLAN/FLAT and geneve networks connected to the same router, the redirect-type option is not set, and therefore the traffic for the VLAN/FLAT networks will also be centralized but not tunneled.

  • When using ML2/OVN, during an upgrade procedure, the OVS system-id stored value can be changed. The ovn-controller service will create the “Chassis” and “Chassis_Private” registers based on this OVS system-id. If the ovn-controller process is not gracefully stopped, that could lead to the existence of duplicated “Chassis” and “Chassis_Private” registers in the OVN Southbound database.

Bug Fixes

  • The config option agent_down_time is now limited to a maximum value of 2147483, as neutron-server will fail to start if it is configured higher. See bug 2028724 for more information.

  • 1986003 Fixed an issue with concurrent requests to activate the same port binding where one of the requests returned a 500 Internal Server Error. With the fix one request will return successfully and the other will return a 409 Conflict (Binding already active). This fixes errors in nova live-migrations where those concurrent requests might be sent. Nova handles the 409/Conflict response gracefully.

  • [bug 2003455] It is added an extra checking to ensure the “reside-on-redirect-chassis” is set to true for the logical router port associated to vlan provider network despite having the “ovn_distributed_floating_ip” enabled or not. This is needed as there is an OVN bug (https://bugzilla.redhat.com/show_bug.cgi?id=2162756) making it not work as expected. Until that is fixed, we need these workaround that makes the traffic centrallized, but not tunneled, through the node with the gateway port, thus avoiding MTU issues.

  • [bug 2022914] Neutron-API supports using relays as the southbound connection in a ML2/OVN setup. Before the maintenance worker of the API required a leader_only connection, which was removed.

  • Fixed the scenario where the DHCP agent is deployed in conjunction with the OVN metadata agent in order to serve metadata for baremetal nodes. In this scenario, the DHCP agent would not set the route needed for the OVN metadata agent service resulting in baremetal nodes not being able to query the metadata service. For more information see bug 1982569.

  • Normalise OVN agent heartbeat timestamp format to match other agent types. This fixes parsing of GET /v2.0/agents for some clients, such as gophercloud.

  • For OVN versions v22.09.0 and above, the mcast_flood_reports option is now set to false on all ports except “localnet” types. In the past, this option was set to true as a workaround for a bug in core OVN multicast implementation.

  • Fix an issue in the OVN driver where network metadata could become unavailable if the metadata port was ever deleted, even if accidental. To re-create the port, a user can now disable, then enable, DHCP for one of the subnets associated with the network using the Neutron API. This will try and create the port, similar to what happens in the DHCP agent for ML2/OVS. For more information, see bug 2015377.

  • Now the ML2/OVN trunk driver prevents a trunk creation if the parent port is already bound. In the same way, if a parent port being used in a trunk is bound, the trunk cannot be deleted.

  • During the port bulk creation, if an IPAM allocation fails (for example, if the IP address is outside of the subnet CIDR), the other IPAM allocations already created are deleted before raising the exception. Fixes bug 2039550.

  • [bug 2003455] As part of a previous commit (https://review.opendev.org/c/openstack/neutron/+/875644) the redirect-type=bridged option was set in all the router gateway ports (cr-lrp ovn ports). However this was breaking the N/S traffic for geneve tenant networks connected to the provider networks through those routers with the redirect-type option enabled. To fix this we ensure that the redirect-type option is only set if all the networks connected to the router are of VLAN or FLAT type, otherwise we fall back to the default option. This also means that if there is a mix of VLAN and geneve tenant networks connected to the same router, the VLAN traffic will be centralized (but not tunneled). If the traffic for the VLAN/FLAT needs to be distributed, then it should use a different router.

  • A new OVN maintenance method remove_duplicated_chassis_registers is added. This method will periodically check the OVN Southbound “Chassis” and “Chassis_Private” tables looking for duplicated registers. The older ones (based on the “Chassis_Private.nb_cfg_timestamp” value) will be removed when more than one register has the same hostname, that should be unique.

  • Neutron can record full connection using log-related feature introduced in OVN 21.12. For more info see bug LP#<https://bugs.launchpad.net/neutron/+bug/2003706>

Other Notes

  • The OVN migration performs validation by default. This validation means an instance is spawned and is tested by simple ping after the migration is finished. Also it tries to create new workload post migration. This is useful for very simple scenarios when migration is tested but is not really useful in production since likely the production envrionments already have running workloads. It makes more sense to require the validation explicitly rather than implicitly run it as the migration is mostly intended for production. The VALIDATE_MIGRATION now defaults to False and needs to be changed to True if validation upon request.

  • The external_mac entry in the NAT table is used to distribute/centralize the traffic to the FIPs. When there is an external_mac set the traffic is distributed (DVR). When it is empty it is centralized through the gateway port (no DVR). Upon port status transition to down, the external_mac was removed regardless of DVR being enabled or not, leading to centralize the FIP traffic for DVR – though it was for down ports that won’t accept traffic anyway.

  • Adds a maintenance task that runs once a day and is responsible for cleaning up Hash Ring nodes that haven’t been updated in 5 days or more. See LP #2033281 for more information.

  • Added the missing extension uplink-status-propagation to the ML2/OVN mechanism driver. This extension is used by the ML2/SR-IOV mechanism driver, that could be loaded with ML2/OVN. Now it is possible to create ports with the “uplink-status-propagation” flag defined.

  • Since OVN 20.06, the “Chassis” register configuration is stored in the “other_config” field and replicated into “external_ids”. This replication is stopped in OVN 22.09. The ML2/OVN plugin tries to retrieve the “Chassis” configuration from the “other_config” field first; if this field does not exist (in OVN versions before 20.06), the plugin will use “external_ids” field instead. Neutron will be compatible with the different OVN versions (with and without “other_config” field).

18.6.0

New Features

  • After the port is considered as provisioned, the Nova port binding update could have not been received, leaving the port as not bound. Now the port provisioning method has an active wait that will retry several times, waiting for the port binding update. If received, the port status will be set as active if the admin state flag is set.

  • A new script to remove the duplicated port bindings was added. This script will list all ml2_port_bindings records in the database, finding those ones with the same port ID. Then the script removes those ones with status=INACTIVE. This script is useful to remove those leftovers that remain in the database after a failed live migration. It is important to remark that this script should not be executed during any live migration process.

  • Add use_random_fully setting to allow an operator to disable the iptables random-fully property on an iptable rules.

Known Issues

  • If the use_random_fully setting is disabled, it will prevent random fully from being used and if there’re 2 guests in different networks using the same source_ip and source_port and they try to reach the same dest_ip and dest_port, packets might be dropped in the kernel do to the racy tuple generation . Disabling this setting should only be done if source_port is really important such as in network firewall ACLs and that the source_ip are never repeating within the platform.

Upgrade Notes

  • The default value for the metadata_workers configuration option has changed to 0 for the ML2/OVN driver. Since [OVN] Allow to execute “MetadataProxyHandler” in a local thread, the OVN metadata proxy handler can be spawned in the same process of the OVN metadata agent, in a local thread. That reduces the number of OVN SB database connections to one.

Bug Fixes

  • Support for the extensions dns_domain_ports and subnet_dns_publish_fixed_ip belonging to the DNS integration is now properly announced by the OVN driver. See bug 1947127

  • Fixes an issue in the ML2/OVN driver where the network segment tag was not being updated in the OVN Northbound database. For more information, see bug 1944708.

18.3.0

Bug Fixes

  • Enforce policy for ‘qos_policy_id’ attribute of Floating IP so only authorized users can set/unset it. For more info see bug LP#1957175.

  • For IPv4 subnets when dns_nameservers is not set in the subnet, servers defined in ‘ovn/dns_servers’ config option or system’s resolv.conf are used, but for IPv6 subnets these are not used. The same will now be used for IPv6 subnets too. Additionally dns servers added in ‘ovn/dns_servers’ config option or system’s resolv.conf will be filtered as per the subnet’s IP version. For more info see the bug report 1951816.

Other Notes

  • OVN mechanism driver allows only to have one physical network per bridge.

18.2.0

Bug Fixes

  • Changes the API behaviour while using OVN driver to enforce that it’s not possible to delete all the IPs from a router port. For more info see bug LP#1948457

  • The agent reporting state to the server now uses a RPC timeout set to the report_interval configuration option value. See 1948676.

18.1.1

Security Issues

  • Fix bug 1939733 by dropping from the dhcp extra option values everything what is after first newline (\n) character before passing them to the dnsmasq.

18.1.0

New Features

  • When noauth auth_strategy is used, neutron no longer requires a resource creation request to include a dummy ‘project_id’ in request body. A default project_id fake_project_id would be populated automatically in that case and would make the use of noauth usage simpler.

Known Issues

  • When using the minimim-bandwidth QoS feature due to bug https://launchpad.net/bugs/1921150 physical NIC resource providers were for some time created with the wrong parent (i.e. the hypervisor RP). This is now partially fixed and new resource providers are created now with the expected parent (i.e. the agent RP). However Placement does not allow re-parenting an already existing resource provider, therefore the following Placement DB update may be needed after the fix for bug 1921150 is applied: neutron/tools/bug-1921150-re-parent-device-rps.sql Until all resource providers have the proper parent, neutron-server will retry the re-parenting update, which will be rejected every time, therefore expect polluted logs and some wasted load on Placement. However please note that the bandwidth-aware scheduling is supposed to work even with the wrongly parented resource providers.

Bug Fixes

  • 1926693 The logic to detect the hypervisor hostname, which was introduced by change 69660, has been fixed and now returns the result consistent with libvirt.

  • The new resource_provider_defualt_hypervisor option has been added, to replace the default hypervisor name to locates the root resource provider without giving a complete list of interfaces or bridges in the resource_provider_hypervisors option. This option is located in the [ovs] ini-section for ovs-agent and [sriov_nic] ini-section for sriov-agent.

18.0.0

New Features

  • Security group rule has now new, read only attribute normalized_cidr which contains network address from the CIDR provided in the remote_ip_prefix attribute. This new attribute shows actual CIDR used by backend firewall drivers.

  • Support for network logging based on security groups added to OVN backend. For more information see bug 1914757.

  • Now it is possible to define a gateway IP when creating a subnet using a subnet pool. If the gateway IP can be allocated in one of the subnet pool available subnets, this subnet is created; otherwise a Conflict exception is raised.

  • A new subnet of type network:routed has been added. If such a subnet is used, the IPs of that subnet will be advertized with BGP over a provider network, which itself can use segments. This basically achieves a BGP-to-the-rack feature, where the L2 connectivity can be confined to a rack only, and all external routing is done by the switches, using BGP. In this mode, it is still possible to use VXLAN connectivity between the compute nodes, and only floating IPs and router gateways are using BGP routing.

  • Added support for the vlan-transparent in the OVN mechanism driver.

  • Introduce the attribute port_device_profile to ports that specifies the device profile needed per port. This parameter is a string. This parameter is passed to Nova and Nova retrieves the requested profile from Cyborg: Device profiles.

    Operators can turn on this feature via the configuration option:

    [ml2]
extension_drivers = port_device_profile

  • Neutron now experimentally supports new API policies with the system scope and the default roles (member, reader, admin).

  • Added support in SR-IOV agent for accelerator-direct VNIC type. This type represents a port that supports any kind of hardware acceleration and is provided by Cyborg (https://wiki.openstack.org/wiki/Cyborg). RFE: 1909100. accelerator-direct-physical is still not supported.

  • A new API resource address group and its CRUD operations are introduced to represent a group of IPv4 and IPv6 address blocks. A new option --remote-address-group is added to the security group rule create command to allow network connectivity with a group of address blocks. And the backend support is added to the openvswitch firewall. When IP addresses are updated in the address groups, changes will also be reflected in the firewall rules of the associated security group rules. For more information, see RFE: 1592028

  • Add support for deleting ML2/OVN agents. Previously, deleting an agent would return a Bad Request error. In addition to deleting the agent, this change also drastically improves the scalability of the ML2/OVN agent handling code.

  • Update of an already bound port with a QoS minimum_bandwidth rule with a new QoS policy with a minimum_bandwidth rule now changes the allocations in placement as well.

    Note

    Updating the minimum_bandwidth rule of a QoS policy that is attached to a port which is bound to a VM is still not possible.

  • A new vnic type vdpa has been added to allow requesting port that utilize a vHost-vDPA offload. The ML2/OVS and ML2/OVN mech drivers now have support for the vHost-vDPA vnic type. vHost-vDPA is similar to vHost-user or kernel vhost offload but utilizes the newly added vDPA bus introduced in the Linux 5.7 kernel. vDPA interface can be implemented in software or hardware, when implemented in hardware they provide equivalent performance to SR-IOV or hardware offloaded OVS while providing two main advantages over both SR-IOV and hardware offloaded OVS. Unlike the alternatives, vHost-vDPA enables live migration of instance transparently and provides a standard virtio-net interface to the guest avoiding the need to install vendor specific drivers in the guest.

  • OVN driver now supports VXLAN type for networks. This requires OVN version to be 20.09 or newer.

Known Issues

  • Even with the “igmp_snooping_enable” configuration option stating that traffic would not be flooded to unregistered VMs when this option was enabled, the ML2/OVN driver didn’t follow that behavior. This has now been fixed and ML2/OVN will no longer flood traffic to unregistered VMs when this configuration option is set to True.

  • Support for new policies and system scope context is experimentatal in Neutron. When config option enforce_new_defaults is enabled in Neutron, new default rules will be enforced and things may not work properly in some cases.

Upgrade Notes

  • Address group now has standard attributes. In the alembic migration, the original description column of address_groups is dropped after data migrated to the standardattributes table. The description field is also removed from the address group object and DB model. This change requires a restart of neutron-server service after the DB migration otherwise users will get server errors when making calls to address group APIs.

  • The default value of [oslo_policy] policy_file config option has been changed from policy.json to policy.yaml. Operators who are utilizing customized or previously generated static policy JSON files (which are not needed by default), should generate new policy files or convert them in YAML format. Use the oslopolicy-convert-json-to-yaml tool to convert a JSON to YAML formatted policy file in backward compatible way.

Deprecation Notes

  • Use of JSON policy files was deprecated by the oslo.policy library during the Victoria development cycle. As a result, this deprecation is being noted in the Wallaby cycle with an anticipated future removal of support by oslo.policy. As such operators will need to convert to YAML policy files. Please see the upgrade notes for details on migration of any custom policy files.

  • Deprecate keepalived_use_no_track config option, as keepalived version check is a safe source to decide if no_track can be used in keepalived configuration file.

  • Removed XenAPI support in Neutron. This driver is no longer supported in Nova and Neutron. The configuration options have been marked as “deprecated for removal” and will be removed in X release.

  • Old API policies are deprecated now. They will be removed in future.

Bug Fixes

  • Stop sending agent heartbeat from ovs agent when it detects OVS is dead. This helps to alarm cloud operators that there is something wrong on the given node.

  • Fixed a MAC learning issue when OVS offload is enabled. The OVS firewall reduces the usage of normal actions to reduce CPU utilization. This causes insertion of a flood rule because there is no MAC learning on ingress traffic. While this is okay for the non-offload case, when using OVS offload the flood rule is not being offloaded. This fixes the MAC learning in the offload case, so we avoid the flood rule. For more information, see bug 1897637.

  • Fixes a configuration problem in the OVN driver that prevented external IGMP queries from reaching the Virtual Machines. See bug 1918108 for details.

Other Notes

  • Added a new config option enable_traditional_dhcp for neutron server, if it is set to False, neutron server will disable DHCP provisioning block, DHCP scheduler API extension, network scheduling mechanism and DHCP RPC/notification. This option can be used with the dhcp extension of the OVS agent to enable distributed DHCP, or for a deployment which needs to disable the DHCP agent related functions permanently.

  • To improve performance of the DHCP agent, it will no longer configure the DHCP server for every port type created in Neutron. For example, for floating IP or router HA interfaces there is no need since a client will not make a DHCP request for them

  • The OVN Metadata Agent now creates the network namespaces including the Neutron network UUID in its name. Previously, the OVN datapath UUID was used and it was not obvious for operators and during debugging to figure out which namespace corresponded to what Neutron network.

  • As defined in Migrate from oslo.rootwrap to oslo.privsep, all OpenStack proyects should migrate from oslo.rootwrap to oslo.privsep because “oslo.privsep offers a superior security model, faster and more secure”. This migration will end with the deprecation and removal of oslo.rootwrap from Neutron. To ensure the quality of the Neutron code, this migration will be done sequentially in several patches, checking none of them breaks the current functionality. In order to easily migrate to execute all external commands inside a privsep context, a new input variable “privsep_exec”, that defaults to “False”, is added to neutron.agent.linux.utils.execute. That will divert the code to a privsep decorated executor. Once the migration finishes, this new input parameter will be removed.

  • When new default values for API policies are enabled, some API requests may not be available for project admin users anymore as they are possible only for system scope users. Please note that system scope tokens don’t have project_id included so for example creation of the provider network, with specified physical network details will now require from system scope admin user to explicitly set project_id.