Newton Series Release Notes¶
9.4.1¶
Prelude¶
IPv6 addresses in DHCP namespaces will now be (correctly) statically configured by the DHCP agent.
Upgrade Notes¶
On upgrade, IPv6 addresses in the DHCP namespaces that have been created dynmically via SLAAC will be removed, and a static IPv6 address will be added instead.
Bug Fixes¶
There is a race condition when adding ports in DHCP namespaces where an IPv6 address could be dynamically created via SLAAC from a Router Advertisement sent from the L3 agent, leading to a failure to start the DHCP agent. This bug has been fixed, but care must be taken on an upgrade dealing with any possibly stale dynamic addresses. For more information, see bug 1627902.
9.4.0¶
Prelude¶
Keepalived VRRP health check functionality to enable verification of connectivity from the “master” router to all gateways.
New Features¶
Activation of this feature enables gateway connectivity validation and rescheduling of the “master” router to another node when connectivity is lost. If all routers lose connectivity to the gateways, the election process will be repeated round-robin until one of the routers restores its gateway connection. In the mean time, all of the routers will be reported as “master”.
9.2.0¶
New Features¶
A new mechanism has been added to netns_cleanup to kill processes that are listening on any port/unix socket within the namespace. This will try to kill them gracefully via SIGTERM and, if they don’t die, then a SIGKILL will be sent to the remaining processes to ensure a proper cleanup.
Known Issues¶
In kernels < 3.19 net.ipv4.ip_nonlocal_bind was not a per-namespace kernel option. L3 HA sets this option to zero to avoid sending gratuitous ARPs for IP addresses that were removed while processing. If this happens then gratuitous ARPs are going to be sent which might populate ARP caches of peer machines with the wrong MAC address.
Deprecation Notes¶
The iptables firewall driver will no longer enable bridge firewalling in next versions of Neutron. If your distribution overrides the default value for any of relevant sysctl settings (
net.bridge.bridge-nf-call-arptables,net.bridge.bridge-nf-call-ip6tables, andnet.bridge.bridge-nf-call-iptables) then make sure you set them back to upstream kernel default (1) using /etc/sysctl.conf or /etc/sysctl.d/* configuration files.
Bug Fixes¶
A special case has been added to allow the creation of DHCP ports on Service Subnets that do not have the service type “network:dhcp”, provided that the subnet has ‘enable_dhcp’ set to ‘True’. This fixes the recurring error seen when neutron attempts to automatically create a DHCP port on a dhcp-enabled subnet after the subnet is created. See bug report 1636963 for more details.
Versions of keepalived < 1.2.20 don’t send gratuitous ARPs when keepalived process receives SIGHUP signal. These versions are not packaged in some Linux distributions like RHEL, CentOS or Ubuntu Xenial. Not sending gratuitous ARPs may lead to peer ARP caches containing wrong information about floating IP addresses until the entry is invalidated. Neutron now sends gratuitous ARPs for all new IP addresses that appear on non-HA interfaces in router namespace which simulates behavior of new versions of keepalived.
9.1.0¶
New Features¶
Middleware was added to parse the X-Forwarded-Proto HTTP header or the Proxy protocol in order to help neutron respond with the correct URL refs when it’s put behind a TLS proxy (such as HAProxy). This adds http_proxy_to_wsgi middleware to the pipeline. This middleware is disabled by default, but can be enabled via a configuration option in the oslo_middleware group.
Upgrade Notes¶
The api-paste.ini configuration file for the paste pipeline was updated to add the http_proxy_to_wsgi middleware.
A new option
ha_keepalived_state_change_server_threadshas been added to configure the number of concurrent threads spawned for keepalived server connection requests. Higher values increase the CPU load on the agent nodes. The default value is half of the number of CPUs present on the node. This allows operators to tune the number of threads to suit their environment. With more threads, simultaneous requests for multiple HA routers state change can be handled faster.
9.0.0¶
Prelude¶
Add options to designate external dns driver of neutron for SSL based connections. This makes it possible to use neutron with designate in scenario where endpoints are SSL based. Users can specify to skip cert validation or specify path to a valid cert in [designate] section of neutron.conf file.
Call dhcp_release6 command line utility when releasing unused IPv6 leases for DHCPv6 stateful subnets. dhcp_release6 first appeared in dnsmasq 2.76
Add ip_allocation attribute to port resources
The default value for ‘external_network_bridge’ in the L3 agent is now ‘’.
Support configuration of greenthreads pool for WSGI.
A new rule has been added to the API that allows for tagging traffic with DSCP values. This is currently supported by the Open vSwitch QoS driver.
The Neutron server no longer needs to be configured with a firewall driver and it can support mixed environments of hybrid iptables firewalls and the pure OVS firewall.
Prior to Newton, the neutron-openvswitch-agent used ‘ovs-ofctl’ of_interface driver by default. In Newton, ‘of_interface’ defaults to ‘native’. This mostly eliminates spawning ovs-ofctl and improves performance a little.
Properly calculate overlay (tunnel) protocol overhead for environments using IPv4 or IPv6 endpoints. The ML2 plug-in configuration file contains a new configuration option, ‘overlay_ip_version’, in the ‘[ml2]’ section that indicates the IP version of all overlay network endpoints. Use ‘4’ for IPv4 and ‘6’ for IPv6. Defaults to ‘4’. Additionally, all layer-2 agents must use the same IP version for endpoints.
Support for IPv6 addresses as tunnel endpoints in OVS.
Prior to Newton, the default option for ‘ovsdb_interface’ was ‘vsctl’. In Newton ‘ovsdb_interface’ defaults to ‘native’. This change switches the way of communication with OVSDB from the ovs-vsctl tool to Open vSwitch python api to improve out-of-the-box performance for typical deployments.
Schedule networks on dhcp-agents with access to network
The internal pluggable IPAM implementation – added in the Liberty release – is now the default for both old and new deployments. Old deployments are unconditionally switched to pluggable IPAM during upgrade. Old non-pluggable IPAM is deprecated and removed from code base.
OFAgent has been removed in the Newton cycle.
Remove ‘quota_items’ configuration option from neutron.conf file. This option was deprecated since Liberty release and has no effect now.
Remove ‘router_id’ configuration option from the l3_agent.ini file. ‘router_id’ option has been defined in order to associate an l3-agent to a specific router when use_namespaces=False. It was deprecated after use_namespaces was removed in Mitaka release.
By default, the QoS driver for the Open vSwitch and Linuxbridge agents calculates the burst value as 80% of the available bandwidth.
Several NICs per physical network can be used with SR-IOV.
The created_at and updated_at fields available on Neutron resources now include a timezone indicator at the end. Because this is a change in format, the old ‘timestamp_core’ extension has been removed and replaced with a ‘timestamp’ extension.
The “vlan-aware-vms” feature allows Nova users to launch VMs on a single port (trunk parent port) that connects multiple Neutron logical networks together.
New Features¶
Return code for quota delete for a tenant whose quota has not been previously defined has been changed from 204 to 404.
Two new options are added to [designate] section to support SSL.
First option insecure allows to skip SSL validation when creating a keystone session to initate a designate client. Default value is False, which means to always verify connection.
Second option ca_cert allows setting path to a valid cert file. Default is None.
SR-IOV now supports egress minimum bandwidth configuration.
The port resource now has an ip_allocation attribute. The value of this attribute will be set to ‘immediate’, ‘deferred’, or ‘none’ at the time the port is created. It will not be changed when the port is updated. ‘immediate’ means that the port is expected to have an IP address and Neutron attempted IP allocation on port creation. ‘deferred’ means that the port is expected to have an IP address but Neutron deferred IP allocation until a port update provides the host to which the port will be bound. ‘none’ means that the port was created explicitly with no addresses by passing [] in fixed_ips when creating it.
Subnets now have a new property ‘service_types’. This is a list of port device owners, such that only ports with a matching device owner will be given an IP from this subnet. If no matching service subnet exists for the given device owner, or no service subnets have been defined on the network, the port will be assigned an IP from a subnet with no service-types. This preserves backwards compatibility with older deployments.
Neutron can apply a QoS rule to ports that mark outgoing traffic’s type of service packet header field.
The Open vSwitch Neutron agent has been extended to mark the Type of Service IP header field of packets egressing from the VM when the QoS rule has been applied.
The Neutron server now learns the appropriate firewall wiring behavior from each OVS agent so it no longer needs to be configured with the firewall_driver. This means it also supports multiple agents with different types of firewalls.
net-mtu extension now recalculates network MTU on each network access, not just on creation. It now allows operators to tweak MTU related configuration options and see them applied to all network resources right after controller restart, both old and new.
The new l2_adjacency extension adds an l2_adjacency field to the network, to indicate whether or not there is guaranteed L2 adjacency between the ports on that Network. Routed network implementations would typically set l2_adjacency to False.
The neutron L3 agent now has the ability to load agent extensions, which allows other services to integrate without additional agent changes. An API for exposing the l3 agent’s router info data to the extensions is also provided so that extensions can remain consistent with router state.
Neutron switched to using oslo.cache library to cache port state in metadata agent. With it, more caching backends are now available, including Memcached and Mongo. More details in oslo.cache documentation.
The local_ip value in ml2_conf.ini can now be set to an IPv6 address configured on the system.
DHCP schedulers use “filter_host_with_network_access” plugin method to filter hosts with access to dhcp network. Plugins can overload it to define their own filtering logic. In particular, ML2 plugin delegates the filtering to mechanism drivers.
The Networking API now supports the ‘project_id’ field in requests and responses, for compatibility with the Identity (Keystone) API V3. A new API extension, ‘project-id’, has been added to allow API users to detect if the ‘project_id’ field is supported. Note that the ‘tenant_id’ field is still supported, and the two fields are functionally equivalent.
Users can now apply a QoS rule to a port or network to setup the minimum egress bandwidth per queue and port. The minimum egress bandwidth rule is applied to each port individually.
New API extensions, ‘sorting’ and ‘pagination’, have been added to allow API users to detect if sorting and pagination features are enabled. These features are controlled by
allow_sortingandallow_paginationconfiguration options.
The feature “vlan-aware-vms” is available. To enable it, a service plugin named ‘trunk’ must be added to the option
service_pluginsin your neutron.conf. The plugin exposes two new extensionstrunkandtrunk_details. The plugin can work with multiple backends and in particular Neutron has support for ML2/openvswitch and ML2/linuxbridge. Even though Neutron API compatibility should be preserved for ports associated to trunks, since this is the first release where the feature is available, it is reasonable to expect possible functionality gaps for one or both drivers. These will be filled over time as being reported. The CLI is available via openstackclient, and python-neutronclient 5.1.0 or above. For more details, please check the networking guide.
Known Issues¶
Upgrade Notes¶
A version of dnsmasq that includes dhcp_release6 should be installed on systems running the DHCP agent. Failure to do this could cause DHCPv6 stateful addressing to not function properly.
The rootwrap filters file dhcp.filters must be updated to include dhcp_release6, otherwise trying to run the utility will result in a NoFilterMatched exception.
OSprofiler support was introduced. To allow its usage the api-paste.ini file needs to be modified to contain osprofiler middleware. Also [profiler] section needs to be added to the neutron.conf file with enabled, hmac_keys and trace_sqlalchemy flags defined.
All existing ports are considered to have ‘immediate’ IP allocation. Any ports that do not have this attribute should also be considered to have immediate IP allocation.
A new table ‘subnet_service_types’ has been added to cater for this feature. It uses the ID field from the ‘subnets’ table as a foreign key.
The default value for ‘external_network_bridge’ has been changed to ‘’ since that is the preferred way to configure the L3 agent and will be the only way in future releases. If you have not explicitly set this value and you use the L3 agent, you will need to set this value to ‘br-ex’ to match the old default. If you are using ‘br-ex’, you should switch to ‘’, ensure your external network has a flat segment and ensure your L2 agent has a bridge_mapping entry between the external network’s flat segment physnet and ‘br-ex’ to get the same connectivity. If the external network did not already have the flat segment, you will need to detach all routers from the external networks, delete the incorrect segment type, add the flat segment, and re-attach the routers.
The configuration option dhcp_lease_time was deprecated in the Havana cycle. This option is no longer supported. The option was replaced by dhcp_lease_duration.
The configuration option dnsmasq_dns_server was deprecated in the kilo cycle. This value is no longer supported.
API sorting and pagination features are now enabled by default.
Existing networks with MTU values that don’t reflect configuration will receive new MTU values after controller upgrade. Note that to propagate new correct MTU values to your backend, you may need to resync all agents that set up ports, as well as re-attach VIFs to affected instances.
To retain the old default for neutron-openvswitch-agent, use ‘of_interface = ovs-ofctl’ in the ‘[ovs]’ section of your openvswitch agent configuration file.
By default, the native interface will have the Ryu controller listen on 127.0.0.1:6633. The listen address can be configured with of_listen_address and of_listen_port options. Ensure that the controller has permission to listen at the configured address.
Define the ‘overlay_ip_version’ option and value appropriate for the environment. Only required if not using the Default of ‘4’.
To keep the old default value use ‘ovsdb_interface = vsctl’ in ‘[ovs]’ section of openvswitch_agent.ini (common path ‘/etc/neutron/plugins/ml2/openvswitch_agent.ini’) if there is a separate openvswitch agent configuration file; otherwise apply changes mentioned above to ml2_conf.ini (common path ‘/etc/neutron/plugins/ml2/ml2_conf.ini’).
The native interface configures ovsdb-server to listen for connections on 127.0.0.1:6640 by default. The address can be configured with the ovsdb_connection config option. Ensure that ovsdb-server has permissions to listen on the configured address.
In case you rely on the default ML2 path_mtu value of 1500 to cap MTU used for new network resources, please set it explicitly in your ml2_conf.ini file.
During upgrade ‘internal’ ipam driver becomes default for ‘ipam_driver’ config option and data is migrated to new tables using alembic migration.
The network_device_mtu option is removed. Existing users of the option are advised to adopt new configuration options to accommodate for their underlying physical infrastructure. The relevant options are global_physnet_mtu for all plugins, and also path_mtu and physical_network_mtus for ML2.
Remove ‘quota_items’ configuration option from neutron.conf file.
Remove ‘router_id’ configuration option from the l3_agent.ini file.
The configuration options for
default_ipv4_subnet_poolanddefault_ipv6_subnet_poolhave been removed. Please use theis_defaultoption of the create/update subnetpool API instead.
tenant_id column has been renamed to project_id. This database migration is required to be applied as offline migration.
The ‘timestamp_core’ extension has been removed and replaced with the ‘standard-attr-timestamp’ extension. Objects will still have timestamps in the ‘created_at’ and ‘updated_at’ fields, but they will have the timestamp appended to the end of them to be consistent with other OpenStack projects.
Deprecation Notes¶
The ‘advertise_mtu’ option is deprecated and will be removed in Ocata. There should be no use case to disable the feature, hence the option is considered redundant. DHCP and L3 agents will continue advertising MTU values to instances. Other plugins not using those agents are also encouraged to advertise MTU to instances. The actual implementation of MTU advertisement depends on the plugin in use, but it’s assumed that at least DHCP option for IPv4 clients and Router Advertisements for IPv6 clients is supported.
The
allow_sortingandallow_paginationconfiguration options are deprecated and will be removed in a future release.
Neutron controller service currently allows to load
service_providersoptions from some files that are not passed to it via –config-dir or –config-file CLI options. This behaviour is now deprecated and will be disabled in Ocata. Current users are advised to switch to aforementioned CLI options.
The option min_l3_agents_per_router is deprecated and will be removed for the Ocata release where the scheduling of new HA routers will always be allowed.
The ‘supported_pci_vendor_devs’ option is deprecated in Newton and will be removed in Ocata. The validation of supported pci vendors is done in nova-scheduler through the pci_passthrough_whitelist option when it selects a suitable hypervisor, hence the option is considered redundant.
The tool neutron-debug is now deprecated, to be replaced with a new set of troubleshooting and diagnostic tools. There is no plan for removal in the immediate term, and not until comparable tools will be adequate enough to supplant neutron-debug altogether. For more information, please see https://blueprints.launchpad.net/neutron/+spec/troubleshooting
The option
[AGENT] prevent_arp_spoofinghas been deprecated and will be removed in Ocata release. ARP spoofing protection should always be enabled unless its explicitly disabled via the port security extension via the API. The primary reason it was a config option was because it was merged at the end of Kilo development cycle so it was not considered stable. It has been enabled by default since Liberty and is considered stable and there is no reason to keep this configurable.
The cache_url configuration option is deprecated as of Newton, and will be removed in Ocata. Please configure metadata cache using [cache] group, setting enable = True and configuring your backend.
The non-pluggable ipam implementatios is deprecated and will be removed in Newton release cycle.
Security Issues¶
OSprofiler support requires passing of trace information between various OpenStack services. This information is securely signed by one of HMAC keys, defined in neutron.conf configuration file. To allow cross-project tracing user should use the key, that is common among all OpenStack services he or she wants to trace.
When working with the ML2/openvswitch driver, the “vlan-aware-vms” feature has the following limitations:
security groups do not work in conjunction with the iptables-based firewall driver.
if security groups are desired, the use of the stateful OVS firewall is required, however, that prevents the use of the DPDK datapath for OVS versions 2.5 or lower.
Bug Fixes¶
Missing OSprofiler support was added. This cross-project profiling library allows to trace various OpenStack requests through all OpenStack services that support it. To initiate OpenStack request tracing –profile <HMAC_KEY> option needs to be added to the CLI command. This key needs to present one of the secret keys defined in neutron.conf configuration file with hmac_keys option under the [profiler] configuration section. To enable or disable Neutron profiling the appropriate enabled option under the same section needs to be set either to True or False. By default Neutron will trace all API and RPC requests, but there is an opportunity to trace DB requests as well. For this purpose trace_sqlalchemy option needs to be set to True. As a prerequisite OSprofiler library and its storage backend needs to be installed to the environment. If so (and if profiling is enabled in neutron.conf) the trace can be generated via command - $ neutron –profile SECRET_KEY <subcommand>. At the end of output there will be message with <trace_id>, and to plot nice HTML graphs the following command should be used - $ osprofiler trace show <trace_id> –html –out result.html
In order to fix the communication issues between SR-IOV instances and regular instances the FDB population extension is added to the OVS or linuxbridge agent. the cause was that messages from SR-IOV direct port instance to normal port instances located on the same hypervisor were sent directly to the wire because the FDB table was not yet updated. FDB population extension tracks instances boot/delete operations using the handle_port delete_port extension interface messages and update the hypervisor’s FDB table accordingly. Please note this L2 agent extension doesn’t support allowed address pairs extension.
The default value for ML2 path_mtu option is changed from 1500 to 0, effectively disabling its participation in network MTU calculation unless it’s overridden in the ml2_conf.ini configuration file.
Fixes Bug 1548193, removing ‘force_gateway_on_subnet’ configuration option. This will always allow adding gateway outside the subnet, and gateway cannot be forced onto the subnet range.
Fixes bug 1572670
Allow SR-IOV agent to run with 0 vfs
The ‘physical_device_mappings’ of sriov_nic configuration now can accept more than one NIC per physical network. For example, if ‘physnet2’ is connected to enp1s0f0 and enp1s0f1, ‘physnet2:enp1s0f0,physnet2:enp1s0f1’ will be a valid option.
Bug 1561200 has been fixed by including the timezone with Neutron ‘created_at’ and ‘updated_at’ fields.
Other Notes¶
In order to use QoS egress minimum bandwidth limit feature, ‘ip-link’ must support the extended VF management parameter
min_tx_rate. Minimum version ofip-linksupporting this parameter isiproute2-ss140804, git tagv3.16.0.
Operators may want to tune the
max_overflowandwsgi_default_pool_sizeconfiguration options according to the investigations outlined in this mailing list post. The default value ofwsgi_default_pool_sizeinherits from that of oslo.config, which is currently 100. This is a change in default from the previous Neutron-specific value of 1000.
The value of the ‘overlay_ip_version’ option adds either 20 bytes for IPv4 or 40 bytes for IPv6 to determine the total tunnel overhead amount.
Requires OVS 2.5+ version or higher with linux kernel 4.3 or higher. More info at OVS github page.
At the time of writing, Neutron bandwidth booking is not integrated with Compute scheduler, which means that minimal bandwidth is not guaranteed but provided as best effort.
The Openflow Agent(OFAgent) mechanism driver and its agent have been removed in favor of OpenvSwitch mechanism driver with “native” of_interface in the Newton cycle.
The configuration option ‘force_gateway_on_subnet’ is removed. This will always allow adding gateway outside the subnet, and gateway cannot be forced onto the subnet range.
