Queens Series Release Notes¶
2.1.2-11¶
Upgrade Notes¶
A new amphora image is required to fix the potential certs-ramfs race condition.
Security Issues¶
A race condition between the certs-ramfs and the amphora agent may lead to tenant TLS content being stored on the amphora filesystem instead of in the encrypted RAM filesystem.
Bug Fixes¶
Add listener and pool protocol validation. The pool and listener can’t be combined arbitrarily. We need some constraints on the protocol side.
Fixed an issue where the the amphora image create tool would checkout the master amphora-agent code and master upper constraints.
Fixed a potential race condition with the certs-ramfs and amphora agent services.
Fix a bug that could interrupt resource creation when performing a graceful shutdown of the house keeping service and leave resources such as amphorae in a BOOTING status.
Fixes an issue in the selection of vip-subnet-id on multi-subnet networks by checking the IP availability of the subnets, ensuring enough IPs are available for loadbalancer when creating loadbalancer specifying vip-network-id.
Fix a bug that could interrupt resource creation when performing a graceful shutdown of the controller worker and leave resources in a PENDING_CREATE/PENDING_UPDATE/PENDING_DELETE provisioning status. If the duration of an Octavia flow is greater than the ‘graceful_shutdown_timeout’ configuration value, stopping the Octavia worker can still interrupt the creation of resources.
2.1.2¶
Security Issues¶
Correctly require two-way certificate authentication to connect to the amphora agent API (CVE-2019-17134).
Bug Fixes¶
Fixed an issue with the health manager reporting an UnboundLocalError if it gets an exception attempting to get a database connection.
Fixes a potential DB deadlock in allocate_and_associate found in testing.
Fixed an issue where invalid certificates would trigger an amphora failover loop. Certificates are now validated at API level.
The passphrase for config option ‘server_certs_key_passphrase’ is used as a Fernet key in Octavia and thus must be 32, base64(url) compatible, characters long. Octavia will now validate the passphrase length and format.
2.1.1¶
Bug Fixes¶
Fixed duplicated IPv6 addresses in Active/Standby mode in CentOS amphorae.
Fixed an issue where the listener API would accept null/None values for fields that must have a valid value, such as connection-limit. Now when a PUT call is made to one of these fields with null as the value the API will reset the field value to the field default value.
2.1.0¶
Upgrade Notes¶
To fix the issue with active/standby load balancers or single topology load balancers with members on the VIP subnet, you need to update the Amphora image.
Critical Issues¶
Fixed a bug where active/standby load balancers and single topology load balancers with members on the VIP subnet may fail. An updated image is required to fix this bug.
Security Issues¶
As a followup to the fix that resolved CVE-2018-16856, Octavia will now encrypt certificates and keys used for secure communication with amphorae, in its internal workflows. Octavia used to exclude debug-level log prints for specific tasks and flows that were explicitly specified by name, a method that is susceptible to code changes.
Bug Fixes¶
Fixed an issue creating members on networks with IPv6 subnets.
Fixed a performance regression in the Octavia v2 API when using the “list” APIs.
Fixes creating a fully populated load balancer with not REDIRECT_POOL type L7 policy and default_pool field.
Fix load balancers that could not be failed over when in ERROR provisioning status.
Fixed a bug that caused an excessive number of RabbitMQ connections to be opened.
Fixed an error when plugging the VIP on CentOS-based amphorae.
Fixed an issue where trying to set a QoS policy on a VIP while the QoS extension is disabled would bring the load balancer to ERROR. Should the QoS extension be disabled, the API will now return HTTP 400 to the user.
Fixed an issue where setting a QoS policy on the VIP would bring the load balancer to ERROR when the QoS extension is enabled.
Other Notes¶
Added a new option named server_certs_key_passphrase under the certificates section. The default value gets copied from an environment variable named TLS_PASS_AMPS_DEFAULT. In a case where TLS_PASS_AMPS_DEFAULT is not set, and the operator did not fill any other value directly, ‘insecure-key-do-not-use-this-key’ will be used.
2.0.4¶
Upgrade Notes¶
To resolve the IPv6 VIP issues on active/standby load balancers you need to build a new Amphora image.
Bug Fixes¶
Fixes issues using IPv6 VIP addresses with load balancers configured for active/standby topology. This fix requires a new amphora image to be built.
Creating a member on a pool with no healthmonitor would sometimes briefly update their operating status from NO_MONITOR to OFFLINE and back to NO_MONITOR during the provisioning sequence. This flapping will no longer occur.
Members that are disabled via admin_state_up=False are now rendered in the HAProxy configuration on the Amphora as disabled. Previously they were not rendered at all. This means that disabled members will now appear in health messages, and will properly change status to OFFLINE.
2.0.3¶
Known Issues¶
Octavia Queens release was released with insufficient lower constraints for Jinja2 and pyOpenSSL requirements. Please make sure your environment can install Jinja2>=2.10 and pyOpenSSL>=17.1.0.
Security Issues¶
Fixed a debug level logging of Amphora certificates for flows such as ‘octavia-create-amp-for-lb-subflow-octavia-generate-serverpem’ (triggered with loadbalancer failover) and ‘octavia-create-amp-for-lb-subflow-octavia-update-cert-expiration’.
Bug Fixes¶
Fixed an issue when Octavia cannot reach the database (all database instances are down) bringing down all running loadbalancers. The Health Manager is more resilient to DB outages now.
Add new parameters to specify the number of threads for updating amphora health and stats.
This will automatically nova delete zombie amphora when they are detected by Octavia. Zombie amphorae are amphorae which report health messages but appear DELETED in Octavia’s database.
Other Notes¶
Processing zombie amphora is already expensive and this adds another step which could increase the load on Octavia Health Manager, especially during Nova API slowness.
2.0.2¶
Security Issues¶
Adds a configuration option, “reserved_ips” that allows the operator to block addresses from being used in load balancer members. The default setting blocks the Nova metadata service address.
Bug Fixes¶
Fixes an issue where if more than one Amphora fails at the same time, failover might not fully complete, leaving the load balancer in ERROR.
Fixes an issue where VIP return traffic was always routed, if a gateway was defined, through the gateway address even if it was local traffic.
Fixes a bug where unspecified or unlimited listener connection limit settings would lead to a 2000 connection limit when using the Amphora/Octavia driver. This was the compiled in connection limit in some HAproxy packages.
Fixes a neutron-lbaas LBaaS v2 API compatibility issue when requesting a load balancer status tree via ‘/statuses’.
2.0.0¶
New Features¶
Added hook to plugin.sh: octavia_create_network_interface_device and octavia_delete_network_interface_device. For each of these functions, if they are defined during stack (respectively unstack), they are called to create (respectively delete) the management network interface.
Added a new endpoint /v2.0/octavia/amphorae to expose internal details about amphorae. This endpoint is admin only.
The compute zone (if applicable) is now cached in the database and returned in the Amphora API as cached_zone. Please note that this is only set at the original time of provisioning, and could be stale for various reasons (for example, if live-migrations have taken place due to maintenances). We recommend it be used for reference only, unless you are absolutely certain it is current in your environment. The source of truth is still the system you use for compute.
Added the ‘failover’ sub-resource for the Amphora API. Each amphora can be triggered to failover by sending a PUT (with an empty body) to the resource
/v2.0/octavia/amphorae/<uuid>/failover. It will cause the amphora to be recycled and replaced, in the same way as the health-triggered failover.
Users can now use a reference to a single PKCS12 bundle as their default_tls_container_ref instead of a Barbican container with individual secret objects. PKCS12 supports bundling a private key, certificate, and intermediates. Private keys can no longer be passphrase protected when using PKCS12 bundles. No configuration change is necessary to enable this feature. Users may simply begin using this. Any use of the old style containers will be detected and automatically fall back to using the old Barbican driver.
Certificate bundles can now be stored in any backend Castellan supports, and can be retrieved via a Castellan driver, even if Barbican is not deployed.
It is now possible to completely update a pool’s member list as a batch operation. Using a PUT request on the base member endpoint of a pool, you can specify a list of member objects and the service will perform any necessary creates/deletes/updates as a single operation.
In some environments (e.g. OSA) Neutron and Octavia use different queues (at least different vhosts) and so if Octavia posts to the Octavia queue and Neutron listens on the Neutron queue the events will never make it over.
This adds a way to configure a custom queue for the event streamer thus allowing to post messages to the Neutron queue if needed.
New option in diskimage-create.sh -n to completely disable SSHd on the Amphora image.
Now Octavia API can accept the QoS Policy id from neutron to support the QoS requirements towards Load Balancer VIP port when create/update load balancer.
Known Issues¶
Amphora images with HAProxy older than 1.6 (CentOS 7, etc.) will still use health monitor type TCP when PING is selected by the user.
Upgrade Notes¶
Amphora will need to be updated to a new image with this version of the agent and ping-wrapper.sh script prior to updating the Octavia controllers. If a load balancer is using a health monitor of type PING with an Amphora image that has not been updated, the next configuration change to the load balancer will cause it to go into an ERROR state until it is failed over to an updated image.
Deprecation Notes¶
Config option amp_ssh_access_allowed is deprecated, as it overlaps with amp_ssh_key_name in functionality and is not needed. Simply leave the variable amp_ssh_key_name blank and no ssh key will be installed. This is the same result as using amp_ssh_access_allowed = False.
Security Issues¶
Private keys can no longer be password protected, as PKCS12 does not support storing a passphrase in an explicitly defined way. Note that this is not noticeably less secure than storing a passphrase protected private key in the same place as the passphrase, as was the case with Barbican.
Depending on how the other queue is set up additional passwords for the other queue will be in the Octavia config file. Operators should take care of setting up appropriate users with appropriate restrictions to the topic(s) needed.
It is now possible to completely remove SSHd from the Amphora image, to further lock down access and increase security. If this is set, providing an amp_ssh_key_name in config will install the key, but ssh access will not be possible as SSHd will not be running.
Bug Fixes¶
Fixed an issue where health monitors of type PING were really doing a TCP health check.
Improves error messages returned to the user, such as errors for attempting to add a second health monitor to a pool.
Neutron LBaaS was assigning the VIP port it created the user’s project-id, thus allowing the user to attach Floating-IPs to the VIP port. Octavia, on the other hand, was assigning the Octavia project-id to the port, making it impossible for the user to attach a Floating IP. This patch brings Octavia’s behaviour in line with Neutron LBaaS and assigns the user’s project-id to the VIP port created by Octavia.
