Upgrades¶

Before maintenance window¶

• First you should execute required DB schema migrations. To achieve that without interrupting your existing installation, install new Cinder code in new venv or a container and run the DB sync (cinder-manage db sync). These schema change operations should have minimal or no effect on performance, and should not cause any operations to fail.
• At this point, new columns and tables may exist in the database. These DB schema changes are done in a way that both the N and N+1 release can perform operations against the same schema.

During maintenance window¶

1. The first service is cinder-scheduler. It is load-balanced by the message queue, so the only thing you need to worry about is to shut it down gracefully (using SIGTERM signal) to make sure it will finish all the requests being processed before shutting down. Then you should upgrade the code and restart the service.

2. Repeat first step for all of your cinder-scheduler services.

3. Then you proceed to upgrade cinder-volume services. The problem here is that due to Active/Passive character of this service, you’re unable to run multiple instances of cinder-volume managing a single volume backend. This means that there will be a moment when you won’t have any cinder-volume in your deployment and you want that disruption to be as short as possible.

   Note

   The downtime here is non-disruptive as long as it doesn’t exceed the service heartbeat timeout. If you don’t exceed that, then cinder-schedulers will not notice that cinder-volume is gone and the message queue will take care of queuing any RPC messages until cinder-volume is back.

   To make sure it’s achieved, you can either lengthen the timeout by tweaking service_down_time value in cinder.conf, or prepare upgraded cinder-volume on another node and do a very quick switch by shutting down older service and starting the new one just after that.

   Also note that in case of A/P HA configuration you need to make sure both primary and secondary c-vol have the same hostname set (you can override it using host option in cinder.conf), so both will be listening on the same message queue and will accept the same messages.

4. Repeat third step for all cinder-volume services.

5. Now we should proceed with (optional) cinder-backup services. You should upgrade them in the same manner like cinder-scheduler.

   Note

   Backup operations are time consuming, so shutting down a c-bak service without interrupting ongoing requests can take time. It may be useful to disable the service first using cinder service-disable command, so it won’t accept new requests, and wait a reasonable amount of time until all the in-progress jobs are completed. Then you can proceed with the upgrade. To make sure the backup service finished all the ongoing requests, you can check the service logs.

   Note

   Until Liberty cinder-backup was tightly coupled with cinder-volume service and needed to coexist on the same physical node. This is not true starting with Mitaka version. If you’re still keeping that coupling, then your upgrade strategy for cinder-backup should be more similar to how cinder-volume is upgraded.

6. cinder-api services should go last. In HA deployment you’re typically running them behind a load balancer (e.g. HAProxy), so you need to take one service instance out of the balancer, shut it down, upgrade the code and dependencies, and start the service again. Then you can plug it back into the load balancer.

   Note

   You may want to start another instance of older c-api to handle the load while you’re upgrading your original services.

7. Then you should repeat step 6 for all of the cinder-api services.

After maintenance window¶

• Once all services are running the new code, double check in the DB that there are no old orphaned records in services table (Cinder doesn’t remove the records when service is gone or service hostname is changed, so you need to take care of that manually; you should be able to distinguish dead records by looking at when the record was updated). Cinder is basing its RPC version detection on that, so stale records can prevent you from going forward.
• Now all services are upgraded, we need to send the SIGHUP signal, so all the services clear any cached service version data. When a new service starts, it automatically detects which version of the service’s RPC protocol to use, and will downgrade any communication to that version. Be advised that cinder-api service doesn’t handle SIGHUP so it needs to be restarted. It’s best to restart your cinder-api services as last ones, as that way you make sure API will fail fast when user requests new features on a deployment that’s not fully upgraded (new features can fail when RPC messages are backported to lowest common denominator). Order of the rest of the services shouldn’t matter.
• Now all the services are upgraded, the system is able to use the latest version of the RPC protocol and able to access all the features of the new release.
• At this point, you must also ensure you update the configuration, to stop using any deprecated features or options, and perform any required work to transition to alternative features. All the deprecated options should be supported for one cycle, but should be removed before your next upgrade is performed.
• Since Ocata, you also need to run cinder-manage db online_data_migrations command to make sure data migrations are applied. The tool lets you limit the impact of the data migrations by using --max-count option to limit number of migrations executed in one run. If this option is used, the exit status will be 1 if any migrations were successful (even if others generated errors, which could be due to dependencies between migrations). The command should be rerun while the exit status is 1. If no further migrations are possible, the exit status will be 2 if some migrations are still generating errors, which requires intervention to resolve. The command should be considered completed successfully only when the exit status is 0. You need to complete all of the migrations before starting upgrade to the next version (e.g. you need to complete Ocata’s data migrations before proceeding with upgrade to Pike; you won’t be able to execute Pike’s DB schema migrations before completing Ocata’s data migrations).