Major upgrades

Major upgrades

This guide provides information about the upgrade process from Queens to Rocky for OpenStack-Ansible.

Note

You can only upgrade between sequential releases.

Introduction

For upgrades between major versions, the OpenStack-Ansible repository provides playbooks and scripts to upgrade an environment. The run-upgrade.sh script runs each upgrade playbook in the correct order, or playbooks can be run individually if necessary. Alternatively, a deployer can upgrade manually. A major upgrade process performs the following actions:

  • Modifies files residing in the /etc/openstack_deploy directory, to reflect new configuration values.
  • Places flag files that are created by the migration scripts in order to achieve idempotency. These files are placed in the /etc/openstack_deploy.QUEENS directory.
  • Upgrades the infrastructure servers. See setup-infrastructure.yml for details.

For more information about the major upgrade process, see Upgrading by using a script and Upgrading manually.

Warning

The upgrade is always under active development. Test this on a development environment first.

Upgrading by using a script

The Rocky release series of OpenStack-Ansible contains the code for migrating from Queens to Rocky.

Running the upgrade script

To upgrade from Queens to Rocky by using the upgrade script, perform the following steps in the openstack-ansible directory:

  1. Change directory to the repository clone root directory:

    # cd /opt/openstack-ansible
    
  2. Run the following commands:

    # git checkout rocky-eol
    # ./scripts/run-upgrade.sh
    

Upgrading manually

Deployers can run the upgrade steps manually. Manual upgrades are useful for scoping the changes in the upgrade process (for example, in very large deployments with strict SLA requirements), or performing other upgrade automations beyond what is provided by OpenStack-Ansible.

The steps detailed here match those performed by the run-upgrade.sh script. You can safely run these steps multiple times.

Check out the Rocky release

Ensure that your OpenStack-Ansible code is on the latest Rocky tagged release.

# git checkout rocky-eol

Prepare the shell variables

Define these variables to reduce typing when running the remaining upgrade tasks. Because these environments variables are shortcuts, this step is optional. If you prefer, you can reference the files directly during the upgrade.

From the openstack-ansible root directory, run the following commands:

# export MAIN_PATH="$(pwd)"
# export SCRIPTS_PATH="${MAIN_PATH}/scripts"
# export UPGRADE_PLAYBOOKS="${SCRIPTS_PATH}/upgrade-utilities/playbooks"

Deal with existing OpenStack-Ansible artifacts

The deployment node may have previous branch artifacts.

Unset the following environment variables first:

# unset ANSIBLE_INVENTORY

Optionally, take a backup of your environment:

# tar zcf /openstack/previous-ansible_`date +%F_%H%M`.tar.gz /etc/openstack_deploy /etc/ansible/ /usr/local/bin/openstack-ansible.rc

Bootstrap Ansible again

Bootstrap Ansible again to ensure that all OpenStack-Ansible role dependencies are in place before you run playbooks from the Rocky release.

# ${SCRIPTS_PATH}/bootstrap-ansible.sh

Change to the playbooks directory

Change to the playbooks directory so that the OpenStack-Ansible dynamic inventory is found automatically.

# cd playbooks

Preflight checks

Before starting with the upgraded version, perform preflight checks to ensure your environment is stable. If any of those checks fail, the upgrade should stop to let the deployer chose what to do.

Clean up old facts

Some configurations have changed, so purge old facts before the upgrade. For more information, see ansible_fact_cleanup.yml.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/ansible_fact_cleanup.yml"

Update configuration and environment files

The user configuration files in the /etc/openstack_deploy/ directory and the environment layout in the /etc/openstack_deploy/env.d directory have new name values added in Rocky. Update the files as follows. For more information, see deploy-config-changes.yml.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/deploy-config-changes.yml"

Update user secrets file

Rocky introduces new user secrets to the stack. These secrets are populated automatically when you run the following playbook. For more information, see user-secrets-adjustment.yml.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/user-secrets-adjustment.yml"

Clean up the pip.conf file

The presence of the pip.conf file can cause build failures during the upgrade to Rocky. This playbook removes the pip.conf file on all the physical servers and on the repo containers. For more information, see pip-conf-removal.yml.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/pip-conf-removal.yml"

Clean up the ceph-ansible galaxy namespaced roles

The ceph-ansible common roles are no longer namespaced with a galaxy-style ‘.’ (ie. ceph.ceph-common is now cloned as ceph-common), due to a change in the way upstream meta dependencies are handled in the ceph roles. The roles will be cloned according to the new naming, and an upgrade playbook ceph-galaxy-removal.yml has been added to clean up the stale galaxy-named roles.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/ceph-galaxy-removal.yml"

Clean up the molteniron role

The molteniron service is not an official OpenStack project, and has been removed from the integrated build. It can still be optionally added, but due to the fact that it was previously integrated we need to remove it during this major upgrade to ensure that the repo build process does not try to build its wheels.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/molteniron-role-removal.yml"

Upgrade hosts

Before installing the infrastructure and OpenStack, update the host machines.

# openstack-ansible setup-hosts.yml --limit '!galera_all:!rabbitmq_all'

This command is the same setting up hosts on a new installation. The galera_all and rabbitmq_all host groups are excluded to prevent reconfiguration and restarting of any of those containers.

Update the other LXC container configuration

Update the other container configuration independently.

# openstack-ansible lxc-containers-create.yml -e \
'lxc_container_allow_restarts=false' --limit 'galera_all:rabbitmq_all'

This command is a subset of the host setup playbook, limited to the galera_all and rabbitmq_all host groups. The configuration of those containers is updated but a restart forany changes to take effect is deferred to another playbook or later (see the next section).

Perform a controlled rolling restart of the Galera containers

Restart containers one at a time, ensuring that each is started, responding, and synchronized with the other nodes in the cluster before moving on to the next. This step allows the LXC container configuration that you applied earlier to take effect, ensuring that the containers are restarted in a controlled fashion.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/galera-cluster-rolling-restart.yml"

Update repository servers

Update the configuration of the repository servers and build new packages required by the Rocky release.

# openstack-ansible repo-install.yml

Update HAProxy configuration

Install and update any new or changed HAProxy service configurations.

# openstack-ansible haproxy-install.yml

Use the repository servers

Now all containers can be pointed to the repo server’s VIPs.

# openstack-ansible repo-use.yml

Upgrade the MariaDB version

Upgrade MariaDB to the most recent 10.x minor release across the cluster.

# openstack-ansible galera-install.yml -e 'galera_upgrade=true'

Upgrade the infrastructure

The following commands perform all of the steps from the setup-infrastructure playbook, except for repo-install.yml, haproxy-install.yml, and galera-install.yml which you ran earlier. Running these playbook applies the relevant Rocky settings and packages.

For certain versions of Queens, you must upgrade the RabbitMQ service.

For more information, see setup-infrastructure.yml.

# openstack-ansible unbound-install.yml
# openstack-ansible memcached-install.yml
# openstack-ansible rabbitmq-install.yml -e 'rabbitmq_upgrade=true'
# openstack-ansible etcd-install.yml
# openstack-ansible utility-install.yml
# openstack-ansible rsyslog-install.yml

Flush Memcached cache

Flush all of the caches in Memcached. For more information, see memcached-flush.yml.

# openstack-ansible "${UPGRADE_PLAYBOOKS}/memcached-flush.yml"

Upgrade OpenStack

Upgrade the OpenStack components with the same installation playbook, without any additional options.

# openstack-ansible setup-openstack.yml
Creative Commons Attribution 3.0 License

Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.