Rocky Series Release Notes

3.2.2-5

Upgrade Notes

  • After this upgrade, users will no longer be able use network resources they cannot see or “show” on load balancers. Operators can revert this behavior by setting the “allow_invisible_reourece_usage” configuration file setting to True.

Security Issues

  • Previously, if a user knew or could guess the UUID for a network resource, they could use that UUID to create load balancer resources using that UUID. Now the user must have permission to see or “show” the resource before it can be used with a load balancer. This will be the new default, but operators can disable this behavior via the setting the configuration file setting “allow_invisible_resource_usage” to True. This issue falls under the “Class C1” security issue as the user would require a valid UUID.

3.2.2

Bug Fixes

  • Add listener and pool protocol validation. The pool and listener can’t be combined arbitrarily. We need some constraints on the protocol side.

  • Fixed an issue where the the amphora image create tool would checkout the master amphora-agent code and master upper constraints.

  • Fix a bug that could interrupt resource creation when performing a graceful shutdown of the house keeping service and leave resources such as amphorae in a BOOTING status.

3.2.1

Upgrade Notes

  • A new amphora image is required to fix the potential certs-ramfs race condition.

Security Issues

  • A race condition between the certs-ramfs and the amphora agent may lead to tenant TLS content being stored on the amphora filesystem instead of in the encrypted RAM filesystem.

Bug Fixes

  • Fixed a potential race condition with the certs-ramfs and amphora agent services.

  • Fixes an issue in the selection of vip-subnet-id on multi-subnet networks by checking the IP availability of the subnets, ensuring enough IPs are available for loadbalancer when creating loadbalancer specifying vip-network-id.

  • Fix a bug that could interrupt resource creation when performing a graceful shutdown of the controller worker and leave resources in a PENDING_CREATE/PENDING_UPDATE/PENDING_DELETE provisioning status. If the duration of an Octavia flow is greater than the ‘graceful_shutdown_timeout’ configuration value, stopping the Octavia worker can still interrupt the creation of resources.

3.2.0

Known Issues

  • When a load balancer with a UDP listener is updated, the listener service is restarted, which causes an interruption of the flow of traffic during a short period of time. This issue is caused by a keepalived bug (https://github.com/acassen/keepalived/issues/1163) that was fixed in keepalived 2.0.14, but this package is not yet provided by distributions.

Upgrade Notes

  • To enable UDP listener monitoring when no pool is attached, the amphora image needs to be updated and load balancers with UDP listeners need to be failed over to the new image.

Security Issues

  • Correctly require two-way certificate authentication to connect to the amphora agent API (CVE-2019-17134).

Bug Fixes

  • Fixed an issue with the health manager reporting an UnboundLocalError if it gets an exception attempting to get a database connection.

  • Fixes a potential DB deadlock in allocate_and_associate found in testing.

  • Fixes an issue where, if we were unable to attach the base (VRRP) port to an amphora instance, the revert would not clean up the port in neutron.

  • Add support for monitor_address and monitor_port attributes in UDP members. Previously, monitor_address and monitor_port were ignored and address and protocol_port attributes were used as monitoring address and port.

  • Fix operating_status for pools and members that use UDP protocol. operating_status values are now consistant with the values of non-UDP load balancers.

  • Fix a bug that prevented UDP servers to be restored as members of a pool after removing a health monitor resource.

  • The passphrase for config option ‘server_certs_key_passphrase’ is used as a Fernet key in Octavia and thus must be 32, base64(url) compatible, characters long. Octavia will now validate the passphrase length and format.

  • Adding a member with different IP protocol version than the VIP IP protocol version in a UDP load balancer caused a crash in the amphora. A validation step in the amphora driver now prevents mixing IP protocol versions in UDP load balancers.

3.1.1

Bug Fixes

  • Fixed duplicated IPv6 addresses in Active/Standby mode in CentOS amphorae.

  • Fixed an issue where the listener API would accept null/None values for fields that must have a valid value, such as connection-limit. Now when a PUT call is made to one of these fields with null as the value the API will reset the field value to the field default value.

3.1.0

Upgrade Notes

  • To fix the issue with active/standby load balancers or single topology load balancers with members on the VIP subnet, you need to update the amphora image.

Critical Issues

  • Fixed a bug where active/standby load balancers and single topology load balancers with members on the VIP subnet may fail. An updated image is required to fix this bug.

Security Issues

  • As a followup to the fix that resolved CVE-2018-16856, Octavia will now encrypt certificates and keys used for secure communication with amphorae, in its internal workflows. Octavia used to exclude debug-level log prints for specific tasks and flows that were explicitly specified by name, a method that is susceptive to code changes.

Bug Fixes

  • Fixed an issue creating members on networks with IPv6 subnets.

  • Fixes creating a fully populated load balancer with not REDIRECT_POOL type L7 policy and default_pool field.

  • Fixed a performance issue where the Housekeeping service could significantly and incrementally utilize CPU as more amphorae and load balancers are created and/or marked as DELETED.

  • Fix load balancers that could not be failed over when in ERROR provisioning status.

  • Fixed a bug that caused an excessive number of RabbitMQ connections to be opened.

  • Fixed an error when plugging the VIP on CentOS-based amphorae.

  • Fixed an issue where trying to set a QoS policy on a VIP while the QoS extension is disabled would bring the load balancer to ERROR. Should the QoS extension be disabled, the API will now return HTTP 400 to the user.

  • Fixed an issue where setting a QoS policy on the VIP would bring the load balancer to ERROR when the QoS extension is enabled.

  • Octavia will no longer automatically revoke access to secrets whenever load balancing resources no longer require access to them. This may be added in the future.

Other Notes

  • Added a new option named server_certs_key_passphrase under the certificates section. The default value gets copied from an environment variable named TLS_PASS_AMPS_DEFAULT. In a case where TLS_PASS_AMPS_DEFAULT is not set, and the operator did not fill any other value directly, ‘insecure-key-do-not-use-this-key’ will be used.

3.0.2

Upgrade Notes

  • To resolve the IPv6 VIP issues on active/standby load balancers you need to build a new amphora image.

Security Issues

  • Fixed a debug level logging of Amphora certificates for flows such as ‘octavia-create-amp-for-lb-subflow-octavia-generate-serverpem’ (triggered with loadbalancer failover) and ‘octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration’.

Bug Fixes

  • Fixes issues using IPv6 VIP addresses with load balancers configured for active/standby topology. This fix requires a new amphora image to be built.

  • Add new parameters to specify the number of threads for updating amphora health and stats.

  • This will automatically nova delete zombie amphora when they are detected by Octavia. Zombie amphorae are amphorae which report health messages but appear DELETED in Octavia’s database.

Other Notes

  • Processing zombie amphora is already expensive and this adds another step which could increase the load on Octavia Health Manager, especially during Nova API slowness.

3.0.1

Bug Fixes

  • Fixed a performance regression in the Octavia v2 API when using the “list” APIs.

  • Fixed an issue when Octavia cannot reach the database (all database instances are down) bringing down all running loadbalancers. The Health Manager is more resilient to DB outages now.

3.0.0

New Features

  • Added UDP protocol support to listeners and pools.

  • Adds a health monitor type of UDP-CONNECT that does a basic UDP port connect.

  • Listeners have four new timeout settings:

    • timeout_client_data: Frontend client inactivity timeout

    • timeout_member_connect: Backend member connection timeout

    • timeout_member_data: Backend member inactivity timeout

    • timeout_tcp_inspect: Time to wait for TCP packets for content inspection

    The value for all of these fields is expected to be in milliseconds.

  • Members have a new boolean option backup. When set to true, the member will not receive traffic until all non-backup members are offline. Once all non-backup members are offline, traffic will begin balancing between the backup members.

  • Added ability for Octavia to automatically set Barbican ACLs on behalf of the user. Such enables users to create TLS-terminated listeners without having to add the Octavia keystone user id to the ACL list. Octavia will also automatically revoke access to secrets whenever load balancing resources no longer require access to them.

  • Add sos element to amphora images (Red Hat family only).

  • Adding support for the listener X-Forwarded-Proto header insertion.

  • Octavia now supports provider drivers. This allows third party load balancing drivers to be integrated with the Octavia v2 API. Users select the “provider” for a load balancer at creation time.

  • There is now an API available to list enabled provider drivers.

  • Cloud deployers can set api_settings.allow_ping_health_monitors = False in octavia.conf to disable the ability to create PING health monitors.

  • The new option [haproxy_amphora]/connection_logging will disable logging of connection data if set to False which can improve performance of the load balancer and might aid compliance.

  • You can now update the running configuration of the Octavia control plane processes by sending the parent process a “HUP” signal. Note: The configuration item must support mutation.

  • Amphora API now returns the field image_id which is the ID of the glance image used to boot the amphora.

Known Issues

Upgrade Notes

  • UDP protocol support requires an update to the amphora image to support UDP protocol statistics reporting and UDP-CONNECT health monitoring.

  • Two new options are included with provider driver support. The enabled_provider_drivers option defaults to “amphora, octavia” to support existing Octavia load balancers. The default_provider_driver option defaults to “amphora” for all new load balancers that do not specify a provider at creation time. These defaults should cover most existing deployments.

  • The provider driver support requires a database migration and follows Octavia standard rolling upgrade procedures; database migration followed by rolling control plane upgrades. Existing load balancers with no provider specified will be assigned “amphora” as part of the database migration.

  • The fix for the hmac.compare_digest on python3 requires you to upgrade your health managers before updating the amphora image. The health manager is compatible with older amphora images, but older controllers will reject the health heartbeats from images with this fix.

Deprecation Notes

  • The quota objects named health_monitor and load_balancer have been renamed to healthmonitor and loadbalancer, respectively. The old names are deprecated, and will be removed in the T cycle.

  • The Octavia API handlers are now deprecated and replaced by the new provider driver support. Octavia API handlers will remain in the code to support the Octavia v1 API (used for neutron-lbaas).

  • Provider of “octavia” has been deprecated in favor of “amphora” to clarify the provider driver supporting the load balancer.

  • Finally completely the remove user_group option, as it was deprecated in Pike.

Security Issues

  • Disabling connection logging might make it more difficult to audit systems for unauthorized access, from which IPs it originated, and which assets were compromised.

  • Adds a configuration option, “reserved_ips” that allows the operator to block addresses from being used in load balancer members. The default setting blocks the nova metadata service address.

Bug Fixes

  • Fixes the v2 API returning “DELETED” records until the amphora_expiry_age timeout expired. The API will now immediately return a 404 HTTP status code when deleted objects are requested. The API version has been raised to v2.1 to reflect this change.

  • Fixes an issue where if more than one amphora fails at the same time, failover might not fully complete, leaving the load balancer in ERROR.

  • Fixes an issue where VIP return traffic was always routed, if a gateway was defined, through the gateway address even if it was local traffic.

  • Fixes a bug where unspecified or unlimited listener connection limit settings would lead to a 2000 connection limit when using the amphora/octavia driver. This was the compiled in connection limit in some HAproxy packages.

  • Fixes an issue with hmac.compare_digest on python3 that could cause health manager “calculated hmac not equal to msg hmac” errors.

  • Creating a member on a pool with no healthmonitor would sometimes briefly update their operating status from NO_MONITOR to OFFLINE and back to NO_MONITOR during the provisioning sequence. This flapping will no longer occur.

  • Members that are disabled via admin_state_up=False are now rendered in the HAProxy configuration on the amphora as disabled. Previously they were not rendered at all. This means that disabled members will now appear in health messages, and will properly change status to OFFLINE.

  • Fixes a neutron-lbaas LBaaS v2 API compatibility issue when requesting a load balancer status tree via ‘/statuses’.

Other Notes

  • Health monitors of type UDP-CONNECT may not work correctly if ICMP unreachable is not enabled on the member server or is blocked by a security rule. A member server may be marked as operating status ONLINE when it is actually down.

  • A provider driver developer guide has been added to the documentation to aid driver providers.

  • An operator documentation page has been added to list known Octavia provider drivers and provide links to those drivers. Non-reference drivers, drivers other than the “amphora” driver, will be outside of the octavia code repository but are dynamically loadable via a well defined interface described in the provider driver developers guide.

  • Installed drivers need to be enabled for use in the Octavia configuration file once you are ready to expose the driver to users.

  • As part of GDPR compliance, connection logs might be considered personal data and might need to follow specific data retention policies. Disabling connection logging might aid in making Octavia compliant by preventing the output of such data. As always, consult with an expert on compliance prior to making changes.