Upgrades¶

Plan your upgrade¶

• Read and ensure you understand the release notes for the next release.
• You should ensure all required steps from the previous upgrade have been completed, such as data migrations.
• Make a backup of your database. Nova does not support downgrading of the database. Hence, in case of upgrade failure, restoring database from backup is the only choice.
• During upgrade be aware that there will be additional load on nova-conductor. You may find you need to add extra nova-conductor workers to deal with the additional upgrade related load.

Rolling upgrade process¶

To reduce downtime, the compute services can be upgraded in a rolling fashion. It means upgrading a few services at a time. This results in a condition where both old (N) and new (N+1) nova-compute services co-exist for a certain time period. Note that, there is no upgrade of the hypervisor here, this is just upgrading the nova services. If reduced downtime is not a concern (or lower complexity is desired), all services may be taken down and restarted at the same time.

1. Before maintenance window:
   • Start the process with the controller node. Install the code for the next version of Nova, either in a venv or a separate control plane node, including all the python dependencies.
   • Using the newly installed nova code, run the DB sync. (nova-manage api_db sync; nova-manage db sync). These schema change operations should have minimal or no effect on performance, and should not cause any operations to fail.
   • At this point, new columns and tables may exist in the database. These DB schema changes are done in a way that both the N and N+1 release can perform operations against the same schema.
2. During maintenance window:
   • Several nova services rely on the external placement service being at the latest level. Therefore, you must upgrade placement before any nova services. See the placement upgrade notes for more details on upgrading the placement service.
   • For maximum safety (no failed API operations), gracefully shutdown all the services (i.e. SIG_TERM) except nova-compute.
   • Before restarting services with new code, perform the release-specific readiness check with nova-status upgrade check. See the nova-status upgrade check for more details on status check.
   • Start all services on the new code, with [upgrade_levels]compute=auto in nova.conf. It is safest to start nova-conductor first and nova-api last. Note that you may use a static alias name instead of auto, such as [upgrade_levels]compute=<release_name>. Also note that this step is only required if compute services are not upgraded in lock-step with the control services.
   • If desired, gracefully shutdown nova-compute (i.e. SIG_TERM) services in small batches, then start the new version of the code with: [upgrade_levels]compute=auto. If this batch-based approach is used, only a few compute nodes will have any delayed API actions, and to ensure there is enough capacity online to service any boot requests that happen during this time.
3. After maintenance window:
   • Once all services are running the new code, double check in the DB that there are no old orphaned service records using nova service-list.
   • Now that all services are upgraded, we need to send the SIG_HUP signal, so all the services clear any cached service version data. When a new service starts, it automatically detects which version of the compute RPC protocol to use, and it can decide if it is safe to do any online data migrations. Note, if you used a static value for the upgrade_level, such as [upgrade_levels]compute=<release_name>, you must update nova.conf to remove that configuration value and do a full service restart.
   • Now all the services are upgraded and signaled, the system is able to use the latest version of the RPC protocol and can access all of the features in the new release.
   • Once all the services are running the latest version of the code, and all the services are aware they all have been upgraded, it is safe to transform the data in the database into its new format. While some of this work happens on demand when the system reads a database row that needs updating, we must get all the data transformed into the current version before the next upgrade. Additionally, some data may not be transformed automatically so performing the data migration is necessary to avoid performance degradation due to compatibility routines.
   • This process can put significant extra write load on the database. Complete all online data migrations using: nova-manage db online_data_migrations --max-count <number>. Note that you can use the --max-count argument to reduce the load this operation will place on the database, which allows you to run a small chunk of the migrations until all of the work is done. The chunk size you should use depends on your infrastructure and how much additional load you can impose on the database. To reduce load, perform smaller batches with delays between chunks. To reduce time to completion, run larger batches. Each time it is run, the command will show a summary of completed and remaining records. If using the --max-count option, the command should be rerun while it returns exit status 1 (which indicates that some migrations took effect, and more work may remain to be done), even if some migrations produce errors. If all possible migrations have completed and some are still producing errors, exit status 2 will be returned. In this case, the cause of the errors should be investigated and resolved. Migrations should be considered successfully completed only when the command returns exit status 0.
   • At this point, you must also ensure you update the configuration, to stop using any deprecated features or options, and perform any required work to transition to alternative features. All the deprecated options should be supported for one cycle, but should be removed before your next upgrade is performed.

Schema Migrations¶

Schema migrations are defined in nova/db/sqlalchemy/migrate_repo/versions and in nova/db/sqlalchemy/api_migrations/migrate_repo/versions. They are the routines that transform our database structure, which should be additive and able to be applied to a running system before service code has been upgraded.

Data Migrations¶

Online data migrations occur in two places:

1. Inline migrations that occur as part of normal run-time activity as data is read in the old format and written in the new format
2. Background online migrations that are performed using nova-manage to complete transformations that will not occur incidentally due to normal runtime activity.

An example of online data migrations are the flavor migrations done as part of Nova object version 1.18. This included a transient migration of flavor storage from one database location to another.

Migration policy:¶

The following guidelines for schema and data migrations are followed in order to ease upgrades:

• Additive schema migrations - In general, almost all schema migrations should be additive. Put simply, they should only create elements like columns, indices, and tables.
• Subtractive schema migrations - To remove an element like a column or table during the N release cycle:
  1. The element must be deprecated and retained for backward compatibility. (This allows for graceful upgrade from N to N+1.)
  2. Data migration, by the objects layer, must completely migrate data from the old version of the schema to the new version.
     • Data migration example
     • Data migration enforcement example (for sqlalchemy migrate/deprecated scripts):
  3. The column can then be removed with a migration at the start of N+2.
• All schema migrations should be idempotent. (For example, a migration should check if an element exists in the schema before attempting to add it.) This logic comes for free in the autogenerated workflow of the online migrations.
• Constraints - When adding a foreign or unique key constraint, the schema migration code needs to handle possible problems with data before applying the constraint. (Example: A unique constraint must clean up duplicate records before applying said constraint.)
• Data migrations - As mentioned above, data migrations will be done in an online fashion by custom code in the object layer that handles moving data between the old and new portions of the schema. In addition, for each type of data migration performed, there should exist a nova-manage option for an operator to manually request that rows be migrated.
  • See flavor migration spec for an example of data migrations in the object layer.

Future work -

1. Adding plumbing to enforce that relevant data migrations are completed before running contract in the expand/migrate/contract schema migration workflow. A potential solution would be for contract to run a gating test for each specific subtract operation to determine if the operation can be completed.