Operating Kolla¶
Tools versioning¶
Kolla and Kolla Ansible use the x.y.z semver
nomenclature for naming versions, with major version increasing with each
new series, e.g., Wallaby.
The tools are designed to, respectively, build and deploy Docker images of
OpenStack services of that series.
Users are advised to run the latest version of tools for the series they
target, preferably by installing directly from the relevant branch of the Git
repository, e.g.:
pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/zed
Version of deployed images¶
By default, Kolla Ansible will deploy or upgrade using the series name embedded
in the internal config (openstack_release) and it is not recommended to
tweak this unless using a local registry and a custom versioning policy, e.g.,
when users want to control when services are upgraded and to which version,
possibly on a per-service basis (but this is an advanced use case scenario).
Upgrade procedure¶
Note
This procedure is for upgrading from series to series, not for doing updates
within a series.
Inside a series, it is usually sufficient to just update the
kolla-ansible package, rebuild (if needed) and pull the images,
and run kolla-ansible deploy again.
Please follow release notes to check if there are any issues to be aware of.
Note
If you have set enable_cells to yes then you should read the
upgrade notes in the Nova cells guide.
Kolla’s strategy for upgrades is to never make a mess and to follow consistent patterns during deployment such that upgrades from one environment to the next are simple to automate.
Kolla Ansible implements a single command operation for upgrading an existing deployment.
Limitations and Recommendations¶
Warning
Please notice that using the ansible --limit option is not recommended.
The reason is, that there are known bugs with it, e.g. when upgrading parts of nova.
We accept bug reports for this and try to fix issues when they are known.
The core problem is how the register: keyword works and how it
interacts with the --limit option. You can find more information in the above
bug report.
Note
Please note that when the use_preconfigured_databases flag is set to
"yes", you need to have the log_bin_trust_function_creators set to
1 by your database administrator before performing the upgrade.
Note
If you have separate keys for nova and cinder, please be sure to set
ceph_nova_keyring: ceph.client.nova.keyring and ceph_nova_user: nova
in /etc/kolla/globals.yml
Ubuntu Focal 20.04¶
The Victoria release adds support for Ubuntu Focal 20.04 as a host operating system. Ubuntu users upgrading from Ussuri should first upgrade OpenStack containers to Victoria, which uses the Ubuntu Focal 20.04 base container image. Hosts should then be upgraded to Ubuntu Focal 20.04.
Ubuntu Jammy 22.04¶
The Zed release adds support for Ubuntu Jammy 22.04 as a host operating system. Ubuntu Jammy 22.04 support was also added to the Yoga stable release. Ubuntu Focal 20.04 users upgrading from Yoga can thus directly upgrade to Ubuntu Jammy 22.04 on the host and then upgrade to the Zed release.
CentOS Stream 8¶
The Wallaby release adds support for CentOS Stream 8 as a host operating system. CentOS Stream 8 support will also be added to a Victoria stable release. CentOS Linux users upgrading from Victoria should first migrate hosts and container images from CentOS Linux to CentOS Stream before upgrading to Wallaby.
Preparation (the foreword)¶
Before preparing the upgrade plan and making any decisions, please read the release notes for the series you are targeting, especially the Upgrade notes that we publish for your convenience and awareness.
Before you begin, make a backup of your config. On the operator/deployment
node, copy the contents of the config directory (/etc/kolla by default) to
a backup place (or use versioning tools, like git, to keep previous versions
of config in a safe place).
Preparation (the real deal)¶
First, upgrade the kolla-ansible package:
pip3 install --upgrade git+https://opendev.org/openstack/kolla-ansible@stable/zed
Note
If you are running from Git repository, then just checkout the desired
branch and run pip3 install --upgrade with the repository directory.
If upgrading to a Yoga release or later, install or upgrade Ansible Galaxy dependencies:
kolla-ansible install-deps
The inventory file for the deployment should be updated, as the newer sample
inventory files may have updated layout or other relevant changes.
The diff tool (or similar) is your friend in this task.
If using a virtual environment, the sample inventories are in
/path/to/venv/share/kolla-ansible/ansible/inventory/, else they are
most likely in
/usr/local/share/kolla-ansible/ansible/inventory/.
Other files which may need manual updating are:
/etc/kolla/globals.yml/etc/kolla/passwords.yml
For globals.yml, it is best to follow the release notes (mentioned above).
For passwords.yml, one needs to use kolla-mergepwd and kolla-genpwd
tools.
kolla-mergepwd --old OLD_PASSWDS --new NEW_PASSWDS --final FINAL_PASSWDS
is used to merge passwords from old installation with newly generated
passwords. The workflow is:
Save old passwords from
/etc/kolla/passwords.ymlintopasswords.yml.old.Generate new passwords via
kolla-genpwdaspasswords.yml.new.Merge
passwords.yml.oldandpasswords.yml.newinto/etc/kolla/passwords.yml.
For example:
cp /etc/kolla/passwords.yml passwords.yml.old
cp kolla-ansible/etc/kolla/passwords.yml passwords.yml.new
kolla-genpwd -p passwords.yml.new
kolla-mergepwd --old passwords.yml.old --new passwords.yml.new --final /etc/kolla/passwords.yml
Note
kolla-mergepwd, by default, keeps old, unused passwords intact.
To alter this behavior, and remove such entries, use the --clean
argument when invoking kolla-mergepwd.
Run the command below to pull the new images on target hosts:
kolla-ansible pull
It is also recommended to run prechecks to identify potential configuration issues:
kolla-ansible prechecks
At a convenient time, the upgrade can now be run.
Perform the Upgrade¶
To perform the upgrade:
kolla-ansible upgrade
After this command is complete, the containers will have been recreated from the new images and all database schema upgrades and similar actions performed for you.
Cleanup the Keystone admin port (Zed only)¶
The Keystone admin port is no longer used in Zed. The admin interface points
to the common port. However, during upgrade, the port is preserved for
intermediate compatibility. To clean up the port, it is necessary to run
the deploy action for Keystone. Additionally, the generated
admin-openrc.sh file may need regeneration as it used the admin
port:
kolla-ansible deploy --tags keystone
kolla-ansible post-deploy
After these commands are complete, there are no leftovers of the admin port.
Tips and Tricks¶
Kolla Ansible CLI¶
When running the kolla-ansible CLI, additional arguments may be passed to
ansible-playbook via the EXTRA_OPTS environment variable.
kolla-ansible -i INVENTORY deploy is used to deploy and start all Kolla
containers.
kolla-ansible -i INVENTORY destroy is used to clean up containers and
volumes in the cluster.
kolla-ansible -i INVENTORY mariadb_recovery is used to recover a
completely stopped mariadb cluster.
kolla-ansible -i INVENTORY prechecks is used to check if all requirements
are meet before deploy for each of the OpenStack services.
kolla-ansible -i INVENTORY post-deploy is used to do post deploy on deploy
node to get the admin openrc file.
kolla-ansible -i INVENTORY pull is used to pull all images for containers.
kolla-ansible -i INVENTORY reconfigure is used to reconfigure OpenStack
service.
kolla-ansible -i INVENTORY upgrade is used to upgrades existing OpenStack
Environment.
kolla-ansible -i INVENTORY stop is used to stop running containers.
kolla-ansible -i INVENTORY deploy-containers is used to check and if
necessary update containers, without generating configuration.
kolla-ansible -i INVENTORY prune-images is used to prune orphaned Docker
images on hosts.
kolla-ansible -i INVENTORY1 -i INVENTORY2 ... Multiple inventories can be
specified by passing the --inventory or -i command line option multiple
times. This can be useful to share configuration between multiple environments.
Any common configuration can be set in INVENTORY1 and INVENTORY2 can be
used to set environment specific details.
kolla-ansible -i INVENTORY gather-facts is used to gather Ansible facts,
for example to populate a fact cache.
Using Hashicorp Vault for password storage¶
Hashicorp Vault can be used as an alternative to Ansible Vault for storing passwords generated by Kolla Ansible. To use Hashicorp Vault as the secrets store you will first need to generate the passwords, and then you can save them into an existing KV using the following command:
kolla-writepwd \
--passwords /etc/kolla/passwords.yml \
--vault-addr <VAULT_ADDRESS> \
--vault-token <VAULT_TOKEN>
Note
For a full list of kolla-writepwd arguments, use the --help
argument when invoking kolla-writepwd.
To read passwords from Hashicorp Vault and generate a passwords.yml:
mv kolla-ansible/etc/kolla/passwords.yml /etc/kolla/passwords.yml
kolla-readpwd \
--passwords /etc/kolla/passwords.yml \
--vault-addr <VAULT_ADDRESS> \
--vault-token <VAULT_TOKEN>
Tools¶
Kolla ships with several utilities intended to facilitate ease of operation.
tools/cleanup-containers is used to remove deployed containers from the
system. This can be useful when you want to do a new clean deployment. It will
preserve the registry and the locally built images in the registry, but will
remove all running Kolla containers from the local Docker daemon. It also
removes the named volumes.
tools/cleanup-host is used to remove remnants of network changes
triggered on the Docker host when the neutron-agents containers are launched.
This can be useful when you want to do a new clean deployment, particularly one
changing the network topology.
tools/cleanup-images --all is used to remove all Docker images built by
Kolla from the local Docker cache.
