Updating Content on Overcloud Nodes

The update of overcloud packages and containers to the latest version of the current release is referred to as the ‘minor update’ in TripleO (distinguishing it from the ‘major upgrade’ to the next release). In the Queens cycle the minor update workflow was changed compared to previous cycles. There are thus version specific sections below.

Updating your Overcloud - Queens and beyond

The Queens release brought common CLI and workflow conventions to the main deployment lifecycle operations (minor updates, major upgrades, and fast forward upgrades). This means that the minor update workflow has changed compared to previous releases, and it should now be easier to learn and reason about the lifecycle operations in general.

To update your overcloud to the latest packages / container images of the OpenStack release that you currently operate, perform these steps:

1. Software sources setup

   In case you use pinned repositories (e.g. to some DLRN hash), make sure to update your repository files on overcloud to get the latest RPMs. If you use stable RDO repositories, you don’t need to change anything.

2. Update preparation

   To prepare the overcloud for the update, run:

   openstack overcloud update prepare \
       <OPTIONS> \
       -e containers-prepare-parameter.yaml
   

   In place of the <OPTIONS> token should go all parameters that you used with previous openstack overcloud deploy command.

   The last argument containers-prepare-parameter.yaml differs in content depending on release. In Queens and before, it was a list of individual container image parameters, pointing to images you’ve already uploaded to local registry in previous step. In Rocky and beyond, this file contains the ContainerImagePrepare parameter. The upload of images to local registry is yet to happen, in a separate step after update prepare.

   Note

   The update prepare command performs a Heat stack update, and as such it should be passed all parameters currently used by the Heat stack (most notably environment files, role counts, roles data, and network data). This is crucial in order to keep correct state of the stack.

   Note

   The containers-prepare-parameter.yaml file is intended to replace any previous container parameters file. You should drop the previous container parameter file and pass the new one for any subsequent stack update operations.

   The update prepare command updates the Heat stack outputs with Ansible snippets used in the next steps of the update.

3. Container image upload

   Since Rocky, we will need to upload the container images to the local registry at this point. Run:

   openstack overcloud external-update run --tags container_image_prepare
   

4. Update run

   Run the update procedure on a subset of nodes selected via the --limit parameter:

   openstack overcloud update run --limit overcloud-controller-0
   

   You can specify a role name, e.g. ‘Compute’, to execute the minor update on all nodes of that role in a rolling fashion (serial: 1 is used on the playbooks).

   There is no required node ordering for performing the minor update on the overcloud, but it’s a good practice to keep some consistency in the process. E.g. all controllers first, then all computes, etc.

   Do this for all the overcloud nodes before proceeding to next step.

5. Ceph update (optional)

   If your environment includes Ceph managed by TripleO (i.e. not what TripleO calls “external Ceph”), you’ll want to update Ceph at this point too.

6. Update convergence

   Note

   Update Converge is only required for version less than Wallaby. Converge has been removed for Wallaby and above.

   To finish the update procedure, run:

   openstack overcloud update converge <OPTIONS>
   

   In place of the <OPTIONS> token should go all parameters that you used with previous openstack overcloud update prepare command (including the new -e container-params.yaml).

   Note

   The update converge command performs a Heat stack update, and as such it should be passed all parameters currently used by the Heat stack (most notably environment files, role counts, roles data, and network data). This is crucial in order to keep correct state of the stack.

   The update converge command updates Heat stack outputs with Ansible snippets the same way as overcloud deploy would, and it runs the config management operations to assert that the overcloud state matches the used overcloud templates.

Updating your Overcloud - Pike

Note

The minor update workflow described below is generally not well tested for non containerized Pike environments. The main focus for the TripleO upgrades engineering and QE teams has been on testing the minor update within a containerized Pike environment.

In particular there are currently no pacemaker update_tasks for the non containerized cluster services (i.e., puppet/services/pacemaker) and those will need to be considered and added. You should reach out to the TripleO community if this is an important feature for you and you’d like to contribute to it.

For the Pike cycle the minor update workflow is significantly different to previous cycles. In particular, rather than using a static yum_update.sh we now use service specific ansible update_tasks (similar to the upgrade_tasks used for the major upgrade workflow since Ocata). Furthermore, these are not executed directly via a Heat stack update, but rather, together with the docker/puppet config, collected and written to ansible playbooks. The operator then invokes these to deliver the minor update to particular nodes.

There are essentially two steps: first perform a (relatively short) Heat stack update against the overcloud to generate the “config” ansible playbooks, and then execute these. See bug 1715557 for more information about this mechanism and its implementation.

1. Confirm that your $HOME/containers-prepare-parameter.yaml ContainerImagePrepare parameter includes a tag_from_label value, so that the latest images are discovered on update, otherwise edit the tag value to specify what image versions to update to.

2. Perform a heat stack update to generate the ansible playbooks, specifying the registry file generated from the first step above:

openstack overcloud update --init-minor-update --container-registry-file latest-images.yaml

3. Invoke the minor update on the nodes specified with the --limit parameter:

openstack overcloud update --limit controller-0

Stable Branch

The –limit was introduced in the Stein release, previous versions used –nodes or –roles parameters.

You can specify a role name, e.g. ‘Compute’, to execute the minor update on all nodes of that role in a rolling fashion (serial:1 is used on the playbooks).

Updating your Overcloud - Ocata and earlier

Updating packages on all overcloud nodes involves two steps. The first one makes sure that the overcloud plan is updated (a new tripleo-heat-templates rpm might have brought fixes/changes to the templates):

openstack overcloud deploy --update-plan-only \
--templates \
-e <full environment>

By using the parameter --update-plan-only we make sure we update only the stored overcloud plan and not the overcloud itself. Make sure you pass the exact same environment parameters that were used at deployment time.

The second step consists in updating the packages themselves on all overcloud nodes with a command similar to the following:

openstack overcloud update stack -i overcloud

This command updates the UpdateIdentifier parameter and triggers stack update operation. If this parameter is set, yum update command is executed on each node. Because running update on all nodes in parallel might be unsafe (an update of a package might involve restarting a service), the command above sets breakpoints on each overcloud node so nodes are updated one by one. When the update is finished on a node the command will prompt for removing breakpoint on next one.

Note

Make sure you use the -i parameter, otherwise update runs on background and does not prompt for removing of breakpoints.

Note

Multiple breakpoints can be removed by specifying list of nodes with a regular expression.

Note

If the update command is aborted for some reason you can always continue in the process by re-running same command.

Note

The --templates and --environment-file (-e) are now deprecated. They can still be passed to the command, but they will be silently ignored. This is due to the plan now used for deployment should only be modified via plan modification commands.