Wallaby Series Release Notes

8.0.1-89

Known Issues

  • When using a distribution with a recent SELinux release such as CentOS 8 Stream, PING health-monitor does not work as shell_exec_t calls are denied by SELinux.

  • Fixed configuration issue which allowed authenticated and authorized users to inject code into HAProxy configuration using API requests. Octavia API no longer accepts unencoded whitespace characters in url_path values in update requests for healthmonitors.

Upgrade Notes

  • The fix that updates the Netfilter Conntrack Sysfs variables requires rebuilding the amphora image in order to be effective.

Security Issues

  • Filter out private information from the taskflow logs when ‘’INFO’’ level messages are enabled and when jobboard is enabled. Logs might have included TLS certificates and private_key. By default, in Octavia only WARNING and above messages are enabled in taskflow and jobboard is disabled.

Bug Fixes

  • Fixed the ability to use the ‘text/plain’ mime type with the healthcheck endpoint.

  • Fixed an issue when deleting the last listener from a load balancer may trigger a failover.

  • Increased the TCP buffer memory maximum and enabled MTU ICMP black hole detection.

  • The generated RSyslog configuration on the amphora supports now RSyslog failover with TCP if multiple RSyslog servers were specified.

  • In order to avoid hitting the Neutron API hard when batch update with creating many new members, we cache the subnet validation results in batch update members API call. We also change to validate new members only during batch update members since subnet ID is immutable.

  • Ensure that the provided rsyslog configuration file is used by the rsyslog by restarting the service, it fixes the log offloading feature on distributions that start rsyslog before cloud-init.

  • Ensure that the provided rsyslog configuration file is used by rsyslog in the amphora by restarting the service when using the amphorav1 provider, it fixes the log offloading feature on distributions that start rsyslog before cloud-init.

  • The parameters of a taskflow Flow were logged in ‘’INFO’’ level messages by taskflow, it included TLS-enabled listeners and pools parameters, such as certificates and private_key.

  • Fix amphora haproxy_count to return the number of haproxy processes that are running.

  • Fixed issues when building amphora image for Centos Stream 9.

  • Fixed issues when building amphora image for RHEL 9.

  • Fixed a bug in amphorav1, the subnet of a member that was being deleted was not immediately unplugged from the amphora, but only during the next update of the members.

  • Fix an authentication error with Barbican when creating a TERMINATED_HTTPS listener with application credential tokens or trust IDs.

  • Fixed a potential race condition in the member batch update API call, the load balancers might not have been locked properly.

  • Fix an issue that may have occurred when running the amphorav2 with persistence, the ComputeActiveWait was incorrectly executed twice on different controllers.

  • Fixed a “corrupted global server state file” error in Centos 9 Stream when reloading the state of the servers after restarting haproxy. It also fixed the recovering of the operational state of the servers in haproxy after its restart.

  • Fix disabled UDP pools. Disabled UDP pools were marked as “OFFLINE” but the requests were still forwarded to the members of the pool.

  • Correctly detect the member operating status “drain” when querying status data from HAProxy.

  • Fix the shutdown of the driver-agent, the process might have been stuck while waiting for threads to finish. Systemd would have killed the process after a timeout, but some children processes might have leaked on the controllers.

  • Enable required SELinux booleans for CentOS or RHEL amphora image.

  • Fix a bug when full graph of load balancer is created without listeners if jobboard_enabled=False

  • Fixed a bug that prevented Octavia from creating listeners with the fully-populated load balancer API in SINGLE topology mode.

  • Fixed backwards compatibility issue with the feature that preserves HAProxy server states between reloads. HAProxy version 1.5 or below do not support this feature, so Octavia will not to activate it on amphorae with those versions.

  • Fix a bug that prevented the provisioning_state of a health-monitor to be set to ERROR when an error occurred while creating, updating or deleting a health-monitor.

  • Fixed a bug that didn’t set all the active load balancer Health Monitors ONLINE in populated LB single-create calls.

  • Fix an issue with IPv6 members that could have been set in operating_status ERROR just after being added.

  • Fixed an issue with load balancers stuck in a PENDING_* state during database outages. Now when a task fails in Octavia, it retries to update the provisioning_status of the load balancer until the database is back (or it gives up after a really long timeout - around 2h45)

  • Fixes listener creation failure when protocol used is PROXY or PROXYV2 which are pool protocol and not listener protocol.

  • Fix update listener certs failed. The fix ensures that an existing certificate gets overwritten properly.

  • Fix an issue with amphorav2 and persistence, some long tasks executed by a controller might have been released in taskflow and rescheduled on another controller. Octavia now ensures that a task is never released early by using a keepalive mechanism to notify taskflow (and its redis backend) that a job is still running.

  • Fixed an issue with members in ERROR operating status that may have been updated briefly to ONLINE during a Load Balancer configuration change.

  • Fixed a potential error when plugging a member from a new network after deleting another member and unplugging its network. Octavia may have tried to plug the new network to a new interface but with an already existing name. This fix requires to update the Amphora image.

  • Netfilter Conntrack Sysfs variables net.netfilter.nf_conntrack_max and nf_conntrack_expect_max get set to sensible values on the amphora now. Previously, kernel default values were used which were much too low for the configured net.netfilter.nf_conntrack_buckets value. As a result packets could get dropped because the conntrack table got filled too quickly. Note that this affects only UDP and SCTP protocol listeners. Connection tracking is disabled for TCP-based connections on the amphora including HTTP(S).

  • Now the [nova] service_name parameter is effectively used to find the nova endpoint in keystone catalog. The parameter had no effect before it was fixed.

  • Modified default Keepalived LVS persistence granularity configuration value so it would be ipv6 compatible.

  • Fix an issue with PING health-monitors on Centos 8 Stream. Changes in Centos and systemd prevent an unprivileged user from sending ping requests from a network namespace.

  • Fix PING health-monitors with recent haproxy releases (>=2.2), haproxy now requires an additional “insecure-fork-wanted” option to authorize the Octavia PING healthcheck.

  • Fix a bug when adding a member on a subnet that belongs to a network with multiple subnets, an incorrect subnet may have been plugged in the amphora.

  • Fix a bug when deleting the last member plugged on a network, the port that was no longer used was not deleted.

  • Fixed a bug that didn’t set the correct provisioning_status for unattached pools when creating a fully-populated load balancer.

  • Fix a bug when updating a load balancer with a QoS policy after a failover, Octavia attempted to update the VRRP ports of the deleted amphorae, moving the provisioning status of the load balancer to ERROR.

  • Fix a potential race condition when updating a resource in the amphorav2 worker. The worker was not waiting for the resource to be set to PENDING_UPDATE, so the resource may have been updated with old data from the database, resulting in a no-op update.

  • Fixed a race condition in the members batch update API call, the data passed to the Octavia worker service may have been incorrect when quickly sending successive API calls. Then the load balancer was stuck in PENDING_UPDATE provisioning_status.

  • Fix the rescheduling of taskflow tasks that have been resumed after being interrupted.

  • Fixed issue with SELinux and the lvs-masquerade.sh script on the amphora. The script already runs with root permissions, so the use of sudo inside the script is unneeded.

  • Fixed an SELinux issues with TCP-based health-monitor on UDP pools, some specific monitoring ports were denied by SELinux. The Amphora image now enables the keepalived_connect_any SELinux boolean that allows connections to any ports.

  • Fix an issue when Octavia performs a failover of an ACTIVE-STANDBY load balancer that has both amphorae missing. Some tasks in the controller took too much time to timeout because the timeout value defined in [haproxy_amphora].active_connection_max_retries and [haproxy_amphora].active_connection_rety_interval was not used.

  • Fixed a too long timeout when attempting to start the VRRP service in an unreachable amphora during a failover. A specific shorter timeout should be used during the failovers.

  • Fix a serialization issue when using TLSContainer with amphorav2 driver with persistence, a list of bytes type in the data model was not correctly converted to serializable data.

  • Fix a bug that could have triggered a race condition when configuring a member interface in the amphora. Due to a race condition, a network interface might have been deleted from the amphora, leading to a loss of connectivity.

  • Fixed “Could not retrieve certificate” error when updating/deleting the client_ca_tls_container_ref field of a listener after a CA/CRL was deleted.

  • Fix a python3 error that prevented to use the [controller_worker]/user_data_config_drive option when building amphorae.

  • Fixed validations in L7 rule and session cookie APIs in order to prevent authenticated and authorized users to inject code into HAProxy configuration. CR and LF (\r and \n) are no longer allowed in L7 rule keys and values. The session persistence cookie names must follow the rules described in https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie.

  • Fix load balancers stuck in PENDING_UPDATE issues for some API calls (POST /l7rule, PUT /pool) when a provider denied the call.

  • The Octavia API returned an unhelpful message when a constraint failed while creating an object in the DB. The error now contains the name and the value of the parameter that breaks the constraints.

  • When plugging a new member subnet, the amphora sends an IP advertisement of the newly allocated IP. It allows the servers on the same L2 network to flush the ARP entries of a previously allocated IP address.

  • Usage of castellan_cert_manager as cert_manager has been significantly improved. Now you can define configuration options for castellan in octavia.conf and they will be passed properly to castellan beckend. This allows to use allowed castellan backends as for certificate storage.

  • Reduce the duration of the failovers of ACTIVE_STANDBY load balancers. Many updates of an unreachable amphora may have been attempted during a failover, now if an amphora is not reachable at the first update, the other updates are skipped.

  • Reduce the duration of the failovers of ACTIVE_STANDBY load balancers when both amphorae are unreachable.

  • Validate that the creation of L7 policies is compatible with the protocol of the listener in the Amphora driver. L7 policies are allowed for Terminated HTTPS or HTTP protocol listeners, but not for HTTPS, TCP or UDP protocols listeners.

Other Notes

  • The string representation of data base model objects has been improved. Calling str() on them will return a certain subset of fields and calling repr() on them will return all fields. This is helpful for debugging, but it may also change some of the log messages that Octavia emits.

8.0.1

Deprecation Notes

  • The [amphora_agent].agent_server_network_file configuration option is now deprecated, the new Amphora network configuration tool introduced in Xena does not support a single configuration file.

Bug Fixes

  • Amphora network configuration for the VIP interface and the pool member interfaces are now applied with the amphora-interface tool. amphora-interface uses pyroute2 low-level functions to configure the interfaces instead of distribution-specific tools such as “network-scripts” or “/etc/network/interfaces” files.

  • Disable conntrack for TCP flows in the Amphora, it reduces memory usage for HAProxy-based listeners and prevents some kernel warnings about dropped packets.

  • Fix an issue with amphorav2 driver, a failover of an amphora created an amphora with an ERROR status.

  • Fixes loadbalancer creation failure when one of the listener port matches with the octavia generated peer ports and the allowed_cidr is explicitly set to 0.0.0.0/0 on the listener. This is due to creation of two security group rules with remote_ip_prefix as None and remote_ip_prefix as 0.0.0.0/0 which neutron rejects the second request with security group rule already exists.

  • Fix a serialization error when using host_routes in VIP subnets when persistence in the amphorav2 driver is enabled.

  • Fixed MAX_TIMEOUT for timeout_client_data, timeout_member_connect, timeout_member_data, timeout_tcp_inspect API listener. The value was reduced from 365 days to 24 days, which now does not exceed the value of the data type in DB.

  • Fixed an issue with the lo interface in the amphora-haproxy network namespace. The lo interface was down and prevented haproxy to communicate with other haproxy processes (for persistent stick tables) on configuration change. It delayed old haproxy worker cleanup and increased the memory consumption usage after reloading the configuration.

  • Increase the limit value for nr_open and file-max in the amphora, the new value is based on what HAProxy 2.x is expecting from the system with the greatest maxconn value that Octavia can set.

  • Fix an issue with the provisioning status of a load balancer that was set to ERROR too early when an error occurred, making the load balancer mutable while the execution of the tasks for this resources haven’t finished yet.

  • Fix an issue that could set the provisioning status of a load balancer to a PENDING_UPDATE state when an error occurred in the amphora failover flow.

  • Fix weighted round-robin for UDP and SCTP listeners with keepalived and lvs. The algorithm must be specified as ‘wrr’ in order for weighted round-robin to work correctly, but was being set to ‘rr’.

8.0.0

New Features

  • Added support for keystone default roles and system token scopes.

  • Added aarch64/arm64 amphora image support to the disk image create tool and to the devstack plugin.

  • The HTTP/2 protocol is now added to the default ALPN protocol list for listener and pools.

  • CentOS-based amphora images will now install HAProxy version 2.2 maintained by CentOS NFV SIG. Other supported distributions (Ubuntu Bionic, RHEL 8) remain untouched.

  • Added support for TLS extension Application Layer Protocol Negotiation (ALPN) to TLS-enabled pools. A new parameter alpn_protocols was added to the Pool API.

  • Octavia provider drivers can now be extended to support HTTP/2 between TLS-enabled pools and members.

  • Added HTTP/2 over TLS support via ALPN protocol negotiation to the amphora provider driver for TLS-enabled pools.

  • The Octavia amphora driver now supports gRPC protocol when HTTP/2 is enabled for TERMINATED_HTTPS listeners and TLS-enabled pools, and the amphora image is using HAProxy 2.0 or newer.

  • Add support for the SCTP protocol in the Amphora driver. Support for SCTP listeners and pools is implemented using keepalived in the amphora. Support for SCTP health monitors is provided by the amphora-health-checker script and relies on an INIT/INIT-ACK/ABORT sequence of packets.

Upgrade Notes

  • Legacy Octavia Advanced RBAC policies will continue to function as before as long as the [oslo_policy] enforce_scope = False and enforce_new_defaults = False settings are present (this is the current oslo.policy default). However, we highly recommend you update your user roles to follow the new keystone default roles and start using scoped tokens as appropriate. See the Octavia Policies administration guide for more information.

  • Support for new features, such as ALPN on pools, HTTP/2 on pools, gRPC, and SCTP require an updated amphora image.

  • The default value of [oslo_policy] policy_file config option has been changed from policy.json to policy.yaml. Operators who are utilizing customized or previously generated static policy JSON files (which are not needed by default), should generate new policy files or convert them in YAML format. Use the oslopolicy-convert-json-to-yaml tool to convert a JSON to YAML formatted policy file in backward compatible way.

Deprecation Notes

  • Legacy Octavia Advanced RBAC policies without the keystone default roles and/or token scoping are deprecated as of the Wallaby release. The oslo.policy project may change the default settings requiring the keystone default roles and scoped tokens in a future release. Please see the upgrade section in these release notes and the Octavia Policies administration guide for more information.

  • Use of JSON policy files was deprecated by the oslo.policy library during the Victoria development cycle. As a result, this deprecation is being noted in the Wallaby cycle with an anticipated future removal of support by oslo.policy. As such operators will need to convert to YAML policy files. Please see the upgrade notes for details on migration of any custom policy files.

Bug Fixes

  • Fixes an issue with load balancer failover, when the VIP subnet is out of IP addresses, that could lead to the VIP being deallocated.

  • Fixed an issue with batch member updates, that don’t have any changes, not properly rolling back the update.

  • Fixed amphora driver pool ALPN compatibity with older amphora images.

  • Fix an issue when load balancer creation was aborted due to en error on get of amphora VM.

  • Fixed an issue that an amphorav2 LB cannot be reached after loadbalancer failover. The LB security group was not set in the amphora port.

  • Fix default value override for timeout values for listeners. Changing the default timeouts in the configuration file wasn’t correctly applied in the default listener parameters.

  • Fixes an issue where provider drivers may not decrement the load balancer objects quota on delete.

  • Fixed an issue that could cause load balancers, with multiple amphora in a failed state, to be unable to complete a failover.

  • Fix an incorrect operating_status with empty UDP pools. A UDP pool without any member is now ONLINE instead of OFFLINE.

  • Some IPv6 UDP members were incorrectly marked in ERROR status, because of a formatting issue while generating the health message in the amphora.

  • Add missing cloud-utils-growpart RPM to Red Hat based amphora images.

  • Add missing cronie RPM to Red Hat based amphora images.

  • Fix nf_conntrack_buckets sysctl in the Amphora, its value was incorrectly set.

  • Fixed an issue were updating a CRL or client certificate on a pool would cause the pool to go into ERROR.

  • Fixed a bug where pools with PROXYV2 will go into ERROR.

  • Fix load balancers that use customized host_routes in the VIP or the member subnets in amphorav2.

  • Fix an issue when updating tls_versions and tls_ciphers in Pools with empty (None) values, unsetting theses parameters now resets their values to the default values.

  • Fixed the healthcheck endpoint always querying the backends by caching results for a configurable time. The default is five seconds.

  • Fix a bug that allowed a user to create a load balancer on a vip_subnet_id that belongs to another user using the subnet UUID.

  • Add a validation step in the Octavia Amphora driver to ensure that the port_security_enabled parameter is set on the VIP network.

Other Notes

  • The diskimage-create.sh default for Ubuntu is now focal.