Stein Series Release Notes

8.3.0-8

Bug Fixes

  • Fixes an issue with the kolla-ansible prechecks command with Docker 20.10. LP#1907436

  • Fixes some configuration issues around Barbican logging. LP#1891343

  • Fixes an issue with executing kolla-ansible when installed via pip install --user. LP#1915527

8.3.0

New Features

  • Improves performance of the common role by generating all fluentd configuration in a single file.

  • Improves performance of the common role by generating all logrotate configuration in a single file.

Upgrade Notes

  • The default value of REST_API_REQUIRED_SETTINGS was synchronized with Horizon. You may want to review settings exposed by the updated configuration.

Security Issues

  • The admin-openrc.sh file generated by kolla-ansible post-deploy was previously created with root:root ownership and 644 permissions. This would allow anyone with access to the same directory to read the file, including the admin credentials. The ownership of admin-openrc.sh is now set to the user executing kolla-ansible, and the file is assigned a mode of 600. This change can be applied by running kolla-ansible post-deploy.

Bug Fixes

  • Add support to use bifrost-deploy behind proxy. It uses existing container_proxy variable.

  • Fixes handling of /dev/kvm permissions to be more robust against host-level actions. LP#1681461

  • This patch adds kolla-ansible internal logrotate config for Logstash. Logstash 2.4 uses integrated in container logrotate configuration which tries to rotate logs in /var/log/logstash while kolla-ansible deployed Logstash logs are in /var/log/kolla/logstash. LP#1886787

  • Fixes --configdir parameter to apply to default passwords.yml location. LP#1887180

  • This patch fixes a bug, when kolla_toolbox Ansible module failed due to Python deprecation warnings caused by paramiko/cryptography. LP#1888657

  • Fixes haproxy_single_service_split template to work with default for mode (http). LP#1896591

  • Fixed invalid fernet cron file path on Debian/Ubuntu from /var/spool/cron/crontabs/root/fernet-cron to /var/spool/cron/crontabs/root. LP#1898765

  • Add with_first_found on placement for placement-api wsgi configuration to allow overwrite from users. LP#1898766

  • Fixes issues with some CloudKitty commands trying to connect to an external TLS endpoint using HTTP. LP#1888544

  • The admin-openrc.sh file generated by kolla-ansible post-deploy was previously created with root:root ownership and 644 permissions. This would allow anyone with access to the same directory to read the file, including the admin credentials. The ownership of admin-openrc.sh is now set to the user executing kolla-ansible, and the file is assigned a mode of 600. This change can be applied by running kolla-ansible post-deploy.

  • Fixes an issue with fluentd deployment when there are no changes to the container’s configuration. LP#1904721

  • Fixes an issue where Keystone Fernet key rotation may fail due to permission denied error if the Keystone rotation happens before the Keystone container starts. LP#1888512

  • Fixes LP#1892210 where the number of open connections to Memcached from neutron-server would grow over time until reaching the maximum set by memcached_connection_limit (5000 by default), at which point the Memcached instance would stop working.

  • Fixes an issue with Octavia deployment caused by a reference to an undefined variable, openstack_cacert. LP#1888003

  • An issue where when Kafka default topic creation was used to create a Kafka topic, no redundant replicas were created in a multi-node cluster. LP#1888522. This affects Monasca which uses Kafka, and was previously masked by the legacy Kafka client used by Monasca which has since been upgraded in Ussuri. Monasca users with multi-node Kafka clusters should consultant the Kafka documentation to increase the number of replicas.

  • Fixes an issue where the br_netfilter kernel module was not loaded on compute hosts. LP#1886796

  • Reduce the use of SQLAlchemy connection pooling, to improve service reliability during a failover of the controller with the internal VIP. LP#1896635

  • No longer configures the Prometheus OpenStack exporter to use the prometheus Docker volume, which was never required.

8.2.0

New Features

  • Kolla Ansible checks now that the local Ansible Python environment is coherent, i.e. used Ansible can see Kolla Ansible. LP#1856346

Upgrade Notes

  • Avoids unnecessary fact gathering using the setup module. This should improve the performance of environments using fact caching and the Ansible smart fact gathering policy. See blueprint for details.

  • In the previous stable release, the octavia user was no longer given the admin role in the admin project, and a task was added to remove the role during upgrades. However, the octavia configuration was not updated to use the service project, causing load balancer creation to fail.

    There is also an issue for existing deployments in simply switching to the service project. While existing load balancers appear to continue to work, creating new load balancers fails due to the security group belonging to the admin project. For this reason, Train and Stein have been reverted to use the admin project by default, while from the Ussuri release the service project will be used by default.

    To provide flexibility, an octavia_service_auth_project variable has been added. In the Train and Stein releases this is set to admin by default, and from Ussuri it will be set to service by default. For users of Train and Stein, octavia_service_auth_project may be set to service in order to avoid a breaking change during the Ussuri upgrade.

    To switch an existing deployment from using the admin project to the service project, it will at least be necessary to create the required security group in the service project, and update octavia_amp_secgroup_list to this group’s ID. Ideally the Amphora flavor and network would also be recreated in the service project, although this does not appear to be necessary for operation, and will impact existing Amphorae.

    See bug 1873176 for details.

  • Apache ZooKeeper will now be automatically deployed whenever Apache Storm is enabled.

Bug Fixes

  • Fixes Kibana deployment with the new E*K stack (6+). LP#1799689

  • Removing chrony package and AppArmor profile from docker host if containerized chrony is enabled. LP#1882513

  • Do not require kolla-ansible to be installed (Stein only). LP#1882780

  • Add missing “become: true” on some VMWare related tasks. Fixed on Copying VMware vCenter CA file and Copying over nsx.ini.

  • In line with clients for other services used by Magnum, Cinder and Octavia also use endpoint_type = internalURL. In the same tune, these services also use the globally defined openstack_region_name.

  • Fixes an issue with Cinder upgrades that would cause online schema migration to fail. LP#1880753

  • Fixes an issue where fernet_token_expiry would fail the pre-checks despite being set to a valid value. Please see bug 1856021 for more details.

  • In the previous stable release, the octavia user was no longer given the admin role in the admin project, and a task was added to remove the role during upgrades. However, the octavia configuration was not updated to use the service project, causing load balancer creation to fail. See upgrade notes for details. LP#1873176

  • Improves error reporting in kolla-genpwd and kolla-mergepwd when input files are not in the expected format. LP#1880220.

  • Fixes Magnum trust operations in multi-region deployments.

  • Fixes an issue where host configuration tasks (sysctl, loading kernel modules) could be performed during the kolla-ansible genconfig command. See bug 1860161 for details.

  • Deploys Apache ZooKeeper if Apache Storm is enabled explicitly. ZooKeeper would only be deployed if Apache Kafka was also enabled, which is often done implicitly by enabling Monasca.

8.1.1

Upgrade Notes

  • The octavia user is no longer given the admin role in the admin project. Octavia does not require this role and instead uses octavia user with admin role in service project. During an upgrade the octavia user is removed from the admin project. See bug 1873176 for details.

Bug Fixes

  • Adds necessary region_name to octavia.conf when enable_barbican is set to true. LP#1867926

  • Adds /etc/timezone to Debian/Ubuntu containers. LP#1821592

  • Fixes an issue with Nova live migration not using migration_interface_address even when TLS was not used. When migrating an instance to a newly added compute host, if addressing depended on /etc/hosts and it had not been updated on the source compute host to include the new compute host, live migration would fail. This did not affect DNS-based name resolution. Analogically, Nova live migration would fail if the address in DNS//etc/hosts was not the same as migration_interface_address due to user customization. LP#1729566

  • Fix qemu loading of ceph.conf (permission error). LP#1861513

  • Remove /run bind mounts in Neutron services causing dbus host-level errors and add /run/netns for neutron-dhcp-agent and neutron-l3-agent. LP#1861792

  • Fixes an issue where old fluentd configuration files would persist in the container across restarts despite being removed from the node_custom_config directory. LP#1862211

  • Use more permissive regex to remove the offending 127.0.1.1 line from /etc/hosts. LP#1862739

  • Each Prometheus mysqld exporter points now to its local mysqld instance (MariaDB) instead of VIP address. LP#1863041

  • Cinder Backup has now access to kernel modules to load e.g. iscsi_tcp module. LP#1863094

  • Makes RabbitMQ hostname address resolution precheck stronger by requiring uniqueness of resolution to avoid later issues. LP#1863363

  • Fixes haproxy role to avoid restarting haproxy service multiple times in a single Ansible run. LP#1864810 LP#1875228

  • Fixes failure to deploy telegraf with monitoring of zookeeper due to wrong variable being referenced. LP#1867179

  • Fixes ceph deployment reconfiguration error, when Gathering OSDs step would fail due to Kolla-Ansible user not having access to /var/lib/ceph/osd/_FSID_/whoami. LP#1867946

  • Fixes designate-worker not to use etcd as its coordination backend because it is not supported by Designate (no group membership support available via tooz). LP#1872205

  • Fixes source-IP-based load balancing for Horizon when using the “split” HAProxy service template.

  • Fixes issue where HAProxy would have no backend servers in its config files when using the “split” config template style.

  • Manage nova scheduler workers through openstack_service_workers variable. LP#1873753

  • Remove the meta field of the Swift rings from the default rsync_module template. Having it by default, undocumented, can lead to unexpected behavior when the Swift documentation states that this field is not processed.

  • Fixes an issue with HAProxy prechecks when scaling out using --limit or --serial. LP#1868986.

  • Fixes an issue with the HAProxy monitor VIP precheck when some instances of HAProxy are running and others are not. See bug 1866617.

  • Fixes gnocchi-api script name for Ubuntu/Debian binary deployments. LP#1861688

  • Fixes an issue with port prechecks for the Placement service. See bug 1861189 for details.

  • Removes the [http]/max-row-limit = 10000 setting from the default InfluxDB configuration, which resulted in the CloudKitty v1 API returning only 10000 dataframes when using InfluxDB as a storage backend. See bug 1862358 for details.

  • Skydive’s API and the web UI now rely on Keystone for authentication. Only users in the Keystone project defined by skydive_admin_tenant_name will be able to authenticate. See LP#1870903 <https://launchpad.net/bugs/1870903> for more details.

  • Switch endpoint_type from public to internal for octavia communicating with the barbican service. See bug 1875618 for details.

8.1.0

New Features

  • Add support to Kolla-Ansible for Cloudkitty InfluxDB storage system deployment.

  • HAProxy - Add the ability to define custom HAProxy services in {{ node_custom_config }}/haproxy/services.d/

  • Designate coordination backend can now be configured via the designate_coordination_backend variable. Coordination is mandatory when multiple workers are deployed as in a multinode environment. Possible values are redis or etcd.

  • Adds support for passing extra options to Prometheus.

Upgrade Notes

  • Modifies the default storage backend for Cloudkitty to InfluxDB, to match the default in Cloudkitty from Stein onwards. This is controlled via cloudkitty_storage_backend. To use the previous default, set cloudkitty_storage_backend to sqlalchemy. See bug 1838641 for details.

  • Modifies the path for custom configuration of swift.conf from /etc/kolla/config/swift/<service>.conf to /etc/kolla/config/swift/<service>/swift.conf, to avoid a collision with custom configuration for <service>.conf. Here, <service> may be proxy-server, account-*, container-* or object-*.

  • The default connection limit for HAProxy backends is 2000 however, MariaDB defaults to a max of 10000 conections. This has been changed to match the MariaDB limit.

    ‘haproxy_max_connections’ has also been increased to 40000 to accommodate this.

  • Changes the database backup procedure to use mariabackup which is compatible with MariaDB 10.3. The qpress based compression used previously is now replaced with gzip. The documented restore procedure has been modified accordingly. See the Mariabackup documentation for further information.

  • The Heat role has stopped disabling deprecated plugins. To apply this change to existing deployments, the file `/etc/kolla/heat-engine/_deprecated.yaml is automatically removed during the upgrade.

Deprecation Notes

  • The enable_xtrabackup variable is deprecated in favour of enable_mariabackup.

Bug Fixes

  • When etcd is used with cinder_coordination_backend and/or designate_coordination_backend, the config has been changed to use the etcd3gw (aka etcd3+http) tooz coordination driver instead of etcd3 due to issues with the latter’s availability and stability. etcd3 does not handle well eventlet-based services, such as cinder’s and designate’s. See bugs 1852086 and 1854932 for details. See also tooz change introducing etcd3gw.

  • Adds configuration to set also_notifies within the pools.yaml file when using the Infoblox backend for Designate.

    Pushing a DNS NOTIFY packet to the master does not cause the DNS update to be propagated onto other nodes within the cluster. This means each node needs a DNS NOTIFY packet otherwise users may be given a stale DNS record if they query any worker node. For details please see bug 1855085

  • Fixes an issue with Docker client timeouts where Docker reports ‘Read timed out’. The client timeout may be configured via docker_client_timeout. The default timeout has been increased to 120 seconds. See bug for details.

  • Fixes an issue where a failure in pulling an image could lead to a container being removed and not replaced. See bug 1852572 for details.

  • Fixes Swift volume mounting failing on kernel 4.19 and later due to removal of nobarrier from XFS mount options. See bug 1800132 for details.

  • Fixes an issue with fluentd parsing of WSGI logs for Aodh, Masakari, Qinling, Vitrage and Zun. See bug 1720371 for details.

  • Fixes glance_api to run as privileged and adds missing mounts so it can use an iscsi cinder backend as its store. LP#1855695

  • When upgrading from Rocky to Stein HAProxy configuration moves from using a single configuration to assembling a file from snippets for each service. Applying the HAProxy tag to the entire play ensures that HAProxy configuration is generated for all services when the HAProxy tag is specified. For details please see bug 1855094.

  • Fixes templating of Prometheus configuration when Alertmanager is disabled. In a deployment where Prometheus is enabled and Alertmanager is disabled the configuration for the Prometheus will fail when templating as the variable prometheus_alert_rules does not contain the key files. LP#1854540

8.0.1

New Features

  • Kolla Ansible can now configure deployed docker for Zun. Enable docker_configure_for_zun (disabled by default to retain backwards compatibility).

  • Neutron port_forwarding service plugin, and l3 extension can be enabled with variable enable_neutron_port_forwarding.

  • Merge action plugins (for config/ini and yaml files) now allow relative imports in the same way that upstream template modules does, e.g. one can now include subtemplate from the same directory as base template.

  • Cinder coordination backend can now be configured via cinder_coordination_backend variable. Coordination is optional and can now be set to either redis or etcd.

Upgrade Notes

  • RHEL-based targets no longer require EPEL repository. It can be safely removed from target hosts if not used otherwise.

Deprecation Notes

8.0.0

Prelude

The Kolla Ansible 8.0.0 release is the first release in the Stein cycle. Highlights include full support for the OpenStack Monasca project, support for the Placement service which has been extracted from Nova, and support for performing full or incremental backups of the MariaDB database.

New Features

  • Adds support for deploying a ceilometer_ipmi container for collecting Ceilometer metrics on IPMI.

  • Adds support in Cinder and Nova for Quobyte volumes

  • Adds support for deploying the OpenStack Cyborg service. Cyborg is a service for managing hardware accelerators.

  • Adds support for a dedicated migration network. This is configured via the variables migration_interface and migration_interface_address.

  • Adds support for deploying the Monasca fork of Grafana, which includes Keystone integration.

  • Adds support for deploying the Monasca Log Metrics service. This service is responsible for generating metrics from log files.

  • Add support for deploying the Monasca Notification service. The Notification service is responsible for notifiying users when an alert, as defined via the Monasca API, is generated by the Monasca Thresh topology.

  • Adds support for deploying the Monasca Persister process. The Persister is responsible for reading metrics, alarms and events from Kafka and storing them in a variety of backends.

  • Adds support for deploying the Monasca thresh service, an Apache Storm topology for alerting.

  • Adds support for deploying the Neutron metering agent.

  • Adds support for configuring custom policies in Octavia.

  • Adds support for using a separate network for Octavia. This is configured via octavia_network_interface and octavia_network_interface_address.

  • Adds an option, haproxy_nova_serialconsole_proxy_tunnel_timeout, to configure the nova_serialconsole_proxy tunnel timeout. The default is to keep the websocket connection alive for 10 minutes.

  • Configures Prometheus as a Vitrage datasource automatically.

  • Adds support for deploying the Monasca Agent, which provides host and application specific monitoring data collection and forwarding.

  • Adds support for configuring the maximum files and processes limits in the nova_libvirt container, via the qemu_max_files and qemu_max_processes variables. The default values for these are 32768 and 131072 respectively. This is useful when Nova uses Ceph as a backend, since the default limit of 1024 is often not enough.

  • Adds support for configuring ulimit in containers, extending the dimension support added in Rocky release.

  • Adds a configuration option enable_keepalived to allow disabling the keepalived service. This is useful when using an external load balancer in front of HAProxy.

  • Adds support for configuring vendor info in Nova via the release file. To do this place a file called release in one of the following locations:

    • /etc/kolla/config/nova/release

    • /etc/kolla/config/nova_compute/release

    • /etc/kolla/config/nova_compute/{{ inventory_hostname }}/release

    An example of the file can be seen at https://github.com/openstack/nova/blob/master/etc/nova/release.sample

  • Adds support for installing Docker Community Edition (CE) using the kolla-ansible bootstrap-servers command. Existing support uses the legacy packages from https://dockerproject.org. New packages are distributed via https://download.docker.com, and that location is now supported and used by default. Use of the legacy packages is enabled by setting the variable docker_legacy_packages to true.

    It is also now possible to skip configuration of the Docker repository, by setting the variable enable_docker_repo to false.

  • Adds ability to configure custom fluentd formatting.

    In some scenarios it may be useful to configure custom fluentd formatting to, for example, convert events to JSON.

    Configuration of custom fluentd formatting is possible by placing output configuration files in /etc/kolla/config/fluentd/format/*.conf.

  • Adds ability to configure custom fluentd inputs.

    Configuration of custom fluentd inputs is possible by placing input configuration files in /etc/kolla/config/fluentd/input/*.conf.

  • Adds support for configuring glance-cache, enabled with enable_glance_image_cache. The cache size is configured via glance_cache_max_size.

  • Implements Neutron rolling upgrade logic, applied for Neutron server, VPNaaS and FWaaS because only these projects have support for rolling upgrade database migration.

  • Implements Nova rolling upgrade logic.

  • Implements Swift rolling upgrade logic, enabled via swift_enable_rolling_upgrade, which is true by default.

  • Adds support for the Ironic Inspector dnsmasq PXE filter that provides improved scalability over the default IPTables PXE filter. This is now used by default instead of the iptables PXE filter. The iptables filter can be enabled by setting ironic_inspector_pxe_filter to iptables.

  • Adds a new flag, enable_openstack_core, which defaults to yes. Setting this flag to no will disable the core OpenStack services, including Glance, Heat, Horizon, Keystone, Neutron, and Nova.

  • Improves the default configuration of OpenStack Ironic when used in standalone mode.

  • Adds support for providing custom kibana configuration via /etc/kolla/config/kibana/kibana.yml.

  • Docker logs are no longer allowed to grow unbounded and have been limited to a fixed size per container. Two new variables have been added, docker_log_max_file and docker_log_max_size which default to 5 and 50MB respectively. This means that for each container, there should be no more than 250MB of Docker logs.

  • Adds a symbolic link from the kolla_logs docker volume to /var/log/kolla, making it easier to find log files. The volume path is compatible with docker-engine and docker-ce.

  • Adds support for taking a backup of all MariaDB-hosted databases using Percona XtraBackup.

  • Adds support for loading kernel modules required by containers. This is required since kolla images removed support for loading kernel modules from within the container in the Stein release.

  • opendaylight_release variable is removed, version is discovered automatically while booting features.

  • Exposed a config option to enable the ceph manager prometheus plugin, this also enables the exporter on the prometheus-server configuration for each ceph-mgr host.

  • HAProxy configuration is now split per service, which makes creating and updating service configurations much simpler.

  • Adds support for stopping a service with the kolla-ansible stop command. This feature allows specific services to be stopped with --tags and --limit to limit the changes to a subset of hosts.

  • Added new parameter in kolla_docker to support configuring TTY in containers, value is False by default

  • Adds support to seperate Swift access and replication traffic from other storage traffic.

    In a deployment where both Ceph and Swift have been deployed, this changes adds functionalality to support optional seperation of storage network traffic. This adds two new network interfaces swift_storage_interface and swift_replication_interface which maintain backwards compatibility.

    The Swift access network interface is configured via swift_storage_interface, which defaults to storage_interface. The Swift replication network interface is configured via swift_replication_interface, which defaults to swift_storage_interface.

    If a separate replication network is used, Kolla Ansible now deploys separate replication servers for the accounts, containers and objects, that listen on this network. In this case, these services handle only replication traffic, and the original account-, container- and object- servers only handle storage user requests.

  • Adds configuration variables to enable/disable custom horizon policy files per-service even if the service is not being deployed by kolla-ansible.

Upgrade Notes

  • Updates the minimum required version of Ansible to 2.5.

  • Changes the default path for certificates generated via kolla-ansible certificates from {[ node_config_directory }}/certificates to {{ node_config }}. {{ node_config }} is the directory containing globals.yml, which by default is /etc/kolla/. This makes certificates consistent with other locally generated files, such as admin-openrc.sh.

  • The default value for docker_legacy_packages is false, which means that the Docker Community Edition (CE) should be installed. If the kolla-ansible bootstrap-servers command is used on a previously deployed host that is running a legacy Docker engine, it would result in the Docker engine being upgraded to use the Docker Community Edition packages, which will result in a restart of the Docker engine and the containers running on that host. Use the kolla-ansible --serial or --limit arguments to avoid losing quorum in clustered services such as MariaDB by restarting all containers at once.

  • The Keystone fernet key rotation scheduling algorithm has been modified to avoid issues with over-rotation of keys.

    The variables fernet_token_expiry, fernet_token_allow_expired_window and fernet_key_rotation_interval may be set to configure the token expiry and key rotation schedule.

    By default, fernet_token_expiry is 86400, fernet_token_allow_expired_window is 172800, and fernet_key_rotation_interval is the sum of these two variables. This allows for the minimum number of active keys - 3.

    See bug 1809469 for details.

  • Adds swift as a gnocchi storage option. Here is the list of storage options for gnocchi: a) Use swift if swift is enabled. b) Use ceph if ceph is enabled. c) Default to file if swift and ceph are enabled. User has to explicitly set to swift or ceph if both are enabled.

  • The Bare Metal Inspection service is now configured to store logs from the inspection ramdisk in the kolla_logs Docker volume.

  • The default PXE filter used by Ironic Inspector is now dnsmasq rather than iptables. This change has been made to work around an issue introduced by moving to Docker CE, where the daemon sets the default policy on the iptables FORWARD chain to DROP. This policy can interact with the Ironic Inspector iptables PXE filter to cause DHCP packets from bare metal nodes to get dropped, which prevents provisioning.

  • Previously deprecated compute groups inner-compute and external-compute have now been removed in favor of the more simple compute group. Please be sure to update your inventory. Set neutron_compute_dvr_mode on nodes with which you wish to customise the value for neutron’s agent_mode.

  • All HAProxy-related variables have been moved from the haproxy role to the haproxy-common role, with the exception of the following which were also split and renamed after the move:

    • haproxy_listen_tcp_extra becomes haproxy_frontend_tcp_extra and haproxy_backend_tcp_extra

    • haproxy_listen_http_extra becomes haproxy_frontend_http_extra and haproxy_backend_http_extra

  • The following additional haproxy related variables have been created in the haproxy-common role:

    • haproxy_http_request_timeout: default http request timeout for haproxy

    • haproxy_queue_timeout: default queue timeout for haproxy

    • haproxy_connect_timeout: default connect timeout for haproxy

    • haproxy_check_timeout: default check timeout for haproxy

    • haproxy_health_check: default health check string for haproxy

    • haproxy_service_template: select which haproxy config style to use

  • Rabbitmq has been updated to 3.7.x. This comes with a new config format which is now called rabbitmq.conf rather than rabbitmq.config.

Deprecation Notes

  • Deprecates support for deploying Ceph. In a future release support for deploying Ceph will be removed from Kolla Ansible. Prior to this we will ensure a migration path to another tool such as Ceph Ansible is available. For new deployments it is recommended to use another tool to deploy Ceph to avoid a future migration. This can be integrated with OpenStack by following the external Ceph guide.

  • The cinder_iscsi_helper variable has been renamed to cinder_target_helper. Use of cinder_iscsi_helper is deprecated, and will be removed during or after the Train release.

Security Issues

  • When the MariaDB backup option is enabled, it will create a new database which is used to keep track of backup-related metadata, along with a new backup user with a specific set of permissions limited to backup-related actions only.

Bug Fixes

  • Adds system hostnames to /etc/hosts, if different from short hostnames. This can fix live migration of Nova instances in some contexts. See bug 1830023 for details.

Other Notes

  • While Kolla Ansible now avoids duplicating Nova cells when messaging or database connection information are changed, operators of existing deployments should perform a manual cleanup of duplicate cells using the nova-manage cell_v2 command from a container running the nova_api image, leaving only two cells, one named cell0 and another one with the right connection information.