Xena Series Release Notes

19.7.0-84

New Features

  • OVN mechanism driver refuses to bind a port to a dead agent.

Known Issues

  • The high availability of metadata service on isolated networks is limited or non-existent. IPv4 metadata is redundant when the DHCP agent managing it is redundant, but recovery is tied to the renewal of the DHCP lease, making most recoveries very slow. IPv6 metadata is not redundant at all as the IPv6 metadata address can only be configured in a single place at a time as it is link-local. Multiple agents trying to configure it will generate an IPv6 duplicate address detection failure.

    Administrators may observe the IPv6 metadata address in “dadfailed” state in the DHCP namespace for this reason, which is only an indication it is not highly available. Until a redesign is made to the isolated metadata service there is not a better deployment option. See bug 1953165 for information.

  • When using ML2/OVN, during an upgrade procedure, the OVS system-id stored value can be changed. The ovn-controller service will create the “Chassis” and “Chassis_Private” registers based on this OVS system-id. If the ovn-controller process is not gracefully stopped, that could lead to the existence of duplicated “Chassis” and “Chassis_Private” registers in the OVN Southbound database.

Bug Fixes

  • The config option agent_down_time is now limited to a maximum value of 2147483, as neutron-server will fail to start if it is configured higher. See bug 2028724 for more information.

  • [bug 2022914] Neutron-API supports using relays as the southbound connection in a ML2/OVN setup. Before the maintenance worker of the API required a leader_only connection, which was removed.

  • Fixed the scenario where the DHCP agent is deployed in conjunction with the OVN metadata agent in order to serve metadata for baremetal nodes. In this scenario, the DHCP agent would not set the route needed for the OVN metadata agent service resulting in baremetal nodes not being able to query the metadata service. For more information see bug 1982569.

  • For OVN versions v22.09.0 and above, the mcast_flood_reports option is now set to false on all ports except “localnet” types. In the past, this option was set to true as a workaround for a bug in core OVN multicast implementation.

  • Fix an issue in the OVN driver where network metadata could become unavailable if the metadata port was ever deleted, even if accidental. To re-create the port, a user can now disable, then enable, DHCP for one of the subnets associated with the network using the Neutron API. This will try and create the port, similar to what happens in the DHCP agent for ML2/OVS. For more information, see bug 2015377.

  • Now the ML2/OVN trunk driver prevents a trunk creation if the parent port is already bound. In the same way, if a parent port being used in a trunk is bound, the trunk cannot be deleted.

  • During the port bulk creation, if an IPAM allocation fails (for example, if the IP address is outside of the subnet CIDR), the other IPAM allocations already created are deleted before raising the exception. Fixes bug 2039550.

  • A new OVN maintenance method remove_duplicated_chassis_registers is added. This method will periodically check the OVN Southbound “Chassis” and “Chassis_Private” tables looking for duplicated registers. The older ones (based on the “Chassis_Private.nb_cfg_timestamp” value) will be removed when more than one register has the same hostname, that should be unique.

Other Notes

  • The external_mac entry in the NAT table is used to distribute/centralize the traffic to the FIPs. When there is an external_mac set the traffic is distributed (DVR). When it is empty it is centralized through the gateway port (no DVR). Upon port status transition to down, the external_mac was removed regardless of DVR being enabled or not, leading to centralize the FIP traffic for DVR – though it was for down ports that won’t accept traffic anyway.

  • Adds a maintenance task that runs once a day and is responsible for cleaning up Hash Ring nodes that haven’t been updated in 5 days or more. See LP #2033281 for more information.

  • Added the missing extension uplink-status-propagation to the ML2/OVN mechanism driver. This extension is used by the ML2/SR-IOV mechanism driver, that could be loaded with ML2/OVN. Now it is possible to create ports with the “uplink-status-propagation” flag defined.

19.7.0

Known Issues

  • The redirect-type=bridged option is only used if all the tenant networks connected to the router are of type VLAN or FLAT. In this case their traffic will be distributed. However, if there is a mix of VLAN/FLAT and geneve networks connected to the same router, the redirect-type option is not set, and therefore the traffic for the VLAN/FLAT networks will also be centralized but not tunneled.

Bug Fixes

  • 1986003 Fixed an issue with concurrent requests to activate the same port binding where one of the requests returned a 500 Internal Server Error. With the fix one request will return successfully and the other will return a 409 Conflict (Binding already active). This fixes errors in nova live-migrations where those concurrent requests might be sent. Nova handles the 409/Conflict response gracefully.

  • [bug 2003455] As part of a previous commit (https://review.opendev.org/c/openstack/neutron/+/875644) the redirect-type=bridged option was set in all the router gateway ports (cr-lrp ovn ports). However this was breaking the N/S traffic for geneve tenant networks connected to the provider networks through those routers with the redirect-type option enabled. To fix this we ensure that the redirect-type option is only set if all the networks connected to the router are of VLAN or FLAT type, otherwise we fall back to the default option. This also means that if there is a mix of VLAN and geneve tenant networks connected to the same router, the VLAN traffic will be centralized (but not tunneled). If the traffic for the VLAN/FLAT needs to be distributed, then it should use a different router.

19.6.0

Known Issues

  • Until the OVN bug (https://bugzilla.redhat.com/show_bug.cgi?id=2162756) is fixed, setting the “reside-on-redirect-chassis” to true for the logical router port associated to vlan provider network is needed. This workaround makes the traffic centrallized, but not tunneled, through the node with the gateway port, thus avoiding MTU issues.

Bug Fixes

  • [bug 2003455] It is added an extra checking to ensure the “reside-on-redirect-chassis” is set to true for the logical router port associated to vlan provider network despite having the “ovn_distributed_floating_ip” enabled or not. This is needed as there is an OVN bug (https://bugzilla.redhat.com/show_bug.cgi?id=2162756) making it not work as expected. Until that is fixed, we need these workaround that makes the traffic centrallized, but not tunneled, through the node with the gateway port, thus avoiding MTU issues.

  • Normalise OVN agent heartbeat timestamp format to match other agent types. This fixes parsing of GET /v2.0/agents for some clients, such as gophercloud.

  • Neutron can record full connection using log-related feature introduced in OVN 21.12. For more info see bug LP#<https://bugs.launchpad.net/neutron/+bug/2003706>

Other Notes

  • The OVN migration performs validation by default. This validation means an instance is spawned and is tested by simple ping after the migration is finished. Also it tries to create new workload post migration. This is useful for very simple scenarios when migration is tested but is not really useful in production since likely the production envrionments already have running workloads. It makes more sense to require the validation explicitly rather than implicitly run it as the migration is mostly intended for production. The VALIDATE_MIGRATION now defaults to False and needs to be changed to True if validation upon request.

19.5.0

New Features

  • Address scope is now added to all OVN LSP port registers in the northbound. Northd then writes the address scope from the northbound to the southbound so it can be used there by the ovn-bgp-agent.

  • After the port is considered as provisioned, the Nova port binding update could have not been received, leaving the port as not bound. Now the port provisioning method has an active wait that will retry several times, waiting for the port binding update. If received, the port status will be set as active if the admin state flag is set.

  • A new script to remove the duplicated port bindings was added. This script will list all ml2_port_bindings records in the database, finding those ones with the same port ID. Then the script removes those ones with status=INACTIVE. This script is useful to remove those leftovers that remain in the database after a failed live migration. It is important to remark that this script should not be executed during any live migration process.

  • Add use_random_fully setting to allow an operator to disable the iptables random-fully property on an iptable rules.

Known Issues

  • If the use_random_fully setting is disabled, it will prevent random fully from being used and if there’re 2 guests in different networks using the same source_ip and source_port and they try to reach the same dest_ip and dest_port, packets might be dropped in the kernel do to the racy tuple generation . Disabling this setting should only be done if source_port is really important such as in network firewall ACLs and that the source_ip are never repeating within the platform.

Upgrade Notes

  • The default value for the metadata_workers configuration option has changed to 0 for the ML2/OVN driver. Since [OVN] Allow to execute “MetadataProxyHandler” in a local thread, the OVN metadata proxy handler can be spawned in the same process of the OVN metadata agent, in a local thread. That reduces the number of OVN SB database connections to one.

Bug Fixes

  • Fixes an issue in the ML2/OVN driver where the network segment tag was not being updated in the OVN Northbound database. For more information, see bug 1944708.

Other Notes

  • Since OVN 20.06, the “Chassis” register configuration is stored in the “other_config” field and replicated into “external_ids”. This replication is stopped in OVN 22.09. The ML2/OVN plugin tries to retrieve the “Chassis” configuration from the “other_config” field first; if this field does not exist (in OVN versions before 20.06), the plugin will use “external_ids” field instead. Neutron will be compatible with the different OVN versions (with and without “other_config” field).

19.3.0

Bug Fixes

  • Support for the extensions dns_domain_ports and subnet_dns_publish_fixed_ip belonging to the DNS integration is now properly announced by the OVN driver. See bug 1947127

19.2.0

Bug Fixes

  • Enforce policy for ‘qos_policy_id’ attribute of Floating IP so only authorized users can set/unset it. For more info see bug LP#1957175.

  • Changes the API behaviour while using OVN driver to enforce that it’s not possible to delete all the IPs from a router port. For more info see bug LP#1948457

  • For IPv4 subnets when dns_nameservers is not set in the subnet, servers defined in ‘ovn/dns_servers’ config option or system’s resolv.conf are used, but for IPv6 subnets these are not used. The same will now be used for IPv6 subnets too. Additionally dns servers added in ‘ovn/dns_servers’ config option or system’s resolv.conf will be filtered as per the subnet’s IP version. For more info see the bug report 1951816.

Other Notes

  • OVN mechanism driver allows only to have one physical network per bridge.

19.1.0

Bug Fixes

  • The agent reporting state to the server now uses a RPC timeout set to the report_interval configuration option value. See 1948676.

19.0.0

New Features

  • When noauth auth_strategy is used, neutron no longer requires a resource creation request to include a dummy ‘project_id’ in request body. A default project_id fake_project_id would be populated automatically in that case and would make the use of noauth usage simpler.

  • Neutron supports creating IPv4 subnet with prefixlen /31 and /32, via disabling dhcp on a subnet. For more information, see bug 1580927.

  • Added a new OVS agent extension dhcp to support distributed DHCP for VMs in compute nodes directly. To enable this just set extensions=dhcp to OVS agent config file under [agent] section. We also add a new config section [dhcp] which has options enable_ipv6 = True/False for indicating whether enable the DHCPv6 for VM ports.

  • Special keywords <project_id>, <project_name>, <user_name> and <user_id> can be used in the network’s, port’s and floating IP’s dns_domain attribute. Those special keywords will be replaced by the corresponding data from the request context. With that cloud admin can define dns_domain for shared network and ports which belongs to the other projects in the way that each project can use separate DNS zones which needs to be pre-created by users. To enable this feature dns_domain_keywords ML2 plugin extension has to be enabled in the Neutron config. Enabling multiple dns_integration extensions at the same time leads to an error.

  • Neutron supports ECMP routes now, with this change, neutron will consolidate multiple routes with the same destination address into a single ECMP route. For more information see bug 1880532.

  • A new quota driver is added: DbQuotaNoLockDriver. This driver, unlike DbQuotaDriver, does not create a unique lock per (resource, project_id). That may lead to a database deadlock state if the number of server requests exceeds the number of resolved resource creations, as described in LP#1926787. This driver relays on the database transactionality isolation and counts the number of used and reserved resources and, if available, creates the new resource reservations in one single database transaction.

  • Added two new API methods to QuotaDriverAPI class. get_resource_usage returns the current resource usage. quota_limit_check checks the current resource usage of several resources against a set of deltas (a dictionary of resource names and resource counters).

  • Adds support for Network Availability Zones to the OVN driver. When Network AZ is used, OVN’s “external” ports will now be scheduled onto nodes belonging to the AZs specified in the network that the port belongs to. This feature also removes the limitation where all “external” ports were part of to a single HA Chassis Group (meaning they would all be bond to a single host) now the “external” ports will be better distributed across different hosts.

  • Support stateless security groups with the latest OVN 21.06+. The stateful=False security groups are mapped to the new “allow-stateless” OVN ACL verb.

  • Added new API extension to QoS service plugin to support CRUD actions for packet rate limit (packet per second) rule in Neutron server side.

  • The port.mac_address field is sanitized to have a common format “xx:xx:xx:xx:xx:xx”. The values stored in the database can be sanitized executing the new script provided neutron-sanitize-port-mac-addresses. This script will read all port registers and fix, if needed, the stored MAC address format. The port API is also modified to sanitize the user input. This change was added to neutron-lib 2.12.0 in 788300.

  • SR-IOV agent now can handle ports from different networks with the same MAC addresses. This feature implies an upgrade in the agent and the server RPC version (see neutron.plugins.ml2.rpc.RpcCallbacks version 1.9). Some agent RPC methods have been updated to pass not only the device MAC address but the PCI slot too. In case of having more than one port with the same MAC address, the PCI slot will discriminate the requested port.

  • Reject any router route or gateway update if not all route nexthops have connectivity with any gateway subnets CIDRs; in other words, all route nexthops IP addresses should belong to one gateway subnet CIDR.

Known Issues

  • When using the minimim-bandwidth QoS feature due to bug https://launchpad.net/bugs/1921150 physical NIC resource providers were for some time created with the wrong parent (i.e. the hypervisor RP). This is now partially fixed and new resource providers are created now with the expected parent (i.e. the agent RP). However Placement does not allow re-parenting an already existing resource provider, therefore the following Placement DB update may be needed after the fix for bug 1921150 is applied: neutron/tools/bug-1921150-re-parent-device-rps.sql Until all resource providers have the proper parent, neutron-server will retry the re-parenting update, which will be rejected every time, therefore expect polluted logs and some wasted load on Placement. However please note that the bandwidth-aware scheduling is supposed to work even with the wrongly parented resource providers.

  • When using Linux Bridge mechanism driver in newer operating systems that use nftables by default, it is needed to switch back to the legacy tool, as documented in the admin documentation for Linux bridge mechanism driver.

Upgrade Notes

  • The way the ML2 plugin filters out API extensions which are not supported by loaded mechanism drivers has changed. Before, the API extension was on the list if at least one of the mechanism drivers supported it, but now the extension needs to be supported by all the mechanism drivers. If at least one of them filters it out, it will be removed from the final list of enabled API extensions. Currently, only the OVN mechanism driver is filtering out some of the ML2 API extensions, thus if that mechanism driver is loaded in Neutron with any other mechanism driver, the list of the enabled API extensions may be smaller than it was before.

  • The configuration options for XenAPI support has been removed, because these options were already ineffective.

  • Both the server and the agent RPC versions have been bumped to 1.9; to provide a smooth upgrade transition, the Upgrade Procedure should be followed, upgrading first the servers and then the agents. The agent RPC methods returned values are not modified to keep compatibility with other agents (Linux Bridge, Open vSwitch). The RPC server side is capable of attending calls from agent API < 1.9, in order to provide backwards compatibility. If the device PCI slot is not provided, the behavior will be the previous one.

Deprecation Notes

  • The following parameters in the designate section have been deprecated and will be removed in a future release. The [designate] auth_type parameter and required keystoneauth parameters should be used instead.

    • admin_username

    • admin_password

    • admin_tenant_id

    • admin_tenant_name

    • admin_auth_url

Security Issues

  • Fix bug 1939733 by dropping from the dhcp extra option values everything what is after first newline (\n) character before passing them to the dnsmasq.

Bug Fixes

  • Report external dns service OverQuota exception as new neutron ConflictException (409) i.e. ExternalDNSOverQuota. Report the failure as “External DNS Quota exceeded for resources: recordset”.

  • Ensures that OVN’s mechanism driver does not start when [ml2_type_geneve]/max_header_size is set below the required 38. LP#1868137

  • 1926693 The logic to detect the hypervisor hostname, which was introduced by change 69660, has been fixed and now returns the result consistent with libvirt.

  • Introduced config option for RPC agent step size customization: rpc_resources_processing_step - Number of resources for neutron to divide the large RPC call data sets. It can be reduced if RPC timeout occurred. Default value equals 20. The best value can be determined empirically in your environment.

  • The new resource_provider_defualt_hypervisor option has been added, to replace the default hypervisor name to locates the root resource provider without giving a complete list of interfaces or bridges in the resource_provider_hypervisors option. This option is located in the [ovs] ini-section for ovs-agent and [sriov_nic] ini-section for sriov-agent.

Other Notes

  • Neutron resource tags can now be 255 characters long, previously resource tags was limited to 60 characters.