Online DB schema upgrades¶
https://blueprints.launchpad.net/cinder/+spec/online-schema-upgrades
One of the priorities for Mitaka cycle is making Cinder able to perform rolling upgrades. This effort requires a lot of discipline to block any changes that can break interoperability of different services versions.
Some database migrations can break this interoperability. This spec aims to provide a guideline of how we handle such changes to make sure Cinder is able to do rolling upgrades.
Problem description¶
Non backward compatible database migrations include:
Non-additive migrations:
Column, tables removals
Column, tables renames
Adding constraints
Long-running data migrations (for example from one column to another)
All of these should be run when all the services are down. This may not be a problem for small amount of data, but when we’re talking about thousands of rows, the downtime may be significant.
Use Cases¶
Deployer would like to be able to perform Cinder upgrade without downtime of control plane services.
User would like not to experience downtime of control plane when cloud he’s using is getting upgraded.
Proposed change¶
In general we want to build on Nova’s experiences and adapt their solution for our needs. Detailed description of how Nova is doing DB migrations in a live way is presented in Dan Smith’s blogpost.
At this moment please note that we’re not basing on older Nova’s approach called Online Schema Changes that was automatically doing additive and contracting DB migrations based on current and desired DB model described in models.py. This approach would eventually lead to developers not needing to write migrations manually. We will not adopt this solution as it was considered experimental, was known for destroying data, wasn’t working in late Liberty and was recently removed from Nova.
The main difference between Cinder and Nova that affects online DB schema upgrades solution is the fact that in Cinder all the services are querying the DB. In Nova only nova-conductor has access to the DB. Therefore we need to modify the approach slightly. The changes force us to span the migrations over 3 releases instead of 2 like in Nova.
First of all we should introduce a unit test that will block migrations that are contracting. Basically such test is failing when a migration using DROP or ALTER command is in tree. We will do it in a similar way like Nova did.
After adding such test we should allow only migrations that are expanding or additive, that is - are adding new columns or tables. Good part is that since Juno we’ve only merged migrations that are like that. This means that we will need the special way rarely.
And (a little complicated) special way goes as follows. Let’s use a real-world example to make this a little more useful. We’ll try to model live DB schema change that will change the relationship between consistency groups and volume types to be modelled correctly. This is a n-n relationship that currently is represented by volume_type_id column in consistencygroups table that is storing comma-separated ids of volume types related to the consistency group. Correct way to represent that is to use ConsistencyGroupVolumeTypeMapping table which maintains connections between CG and VT.
We will be working through 3 releases (+ the current one), so let’s identify them using letters:
L - current
M - in development
N
O
M release¶
In L release we have only the old representation of this relationship. In M we should merge a change that adds ConsistencyGroupVolumeTypeMapping model and required columns in ConsistencyGroup and VolumeType models that will form relationships to the table mentioned earlier. This additive migration can be easily executed on database while L services are running.
As on upgrade we will have L and M releases operating in the same moment we should modify M’s CG versioned object to:
Write data in both places.
Read data from old place.
That way we maintain compatibility with M->L (M is writing to old place) and L->M (M is reading from old place).
Once upgrade is completed admin should use a tool that will finish data migrations. This tool will be developed in cinder-manage and will operate on chunks of CG rows (to limit performance impact). It will simply iterate and set the relationship in new place.
In the end everything is running M services, so everythings writes to both places and reads from the old one. Also we should have all the data migrated at this point so for all CGs have relationship defined both in new and old way.
N release¶
Now on upgrade we will have M and N release versions. N’s versioned objects:
Write data in both places.
Read data from new place.
This way we maintain compatibility M->N (N is writing in place M is reading) and N->M (M is writing in place N is reading).
As N is reading from new place, before starting any N’s service we need to make sure that all the CGs are migrated. To enforce that, as a first N’s migration we should merge a dummy one that will block subsequent ones if there are items that weren’t migrated yet.
Please note that we still have old representation in SQLAlchemy model, so we would be unable to drop the column even if we managed to switch N services to write only to the new place.
O release¶
This time we’ll have N and O services cooperating.
In O we finally modify the SQLAlchemy model to remove the old ConsistencyGroup volume_type_id field. This time we’re able to, because it’s guaranteed that all services write and read from new place. We also remove it from O’s versioned object - it will now read and write data only to the new place.
After the upgrade we have everything writing and reading only from the new place. Moreover - SQLAlchemy models in O’s services doesn’t have a notion of volume_type_id column. Therefore we can have a post-upgrade migration script that finally drops the column (or we can simply add it as migration in P release if we want to limit number of steps needed to do upgrades.
Alternatives¶
We may simply block both contracting and data migrations. This however will prevent us from fixing bad decisions made in the past, which may hurt the project, as I don’t believe it is mature enough yet.
It’s possible to implement Online Schema Changes to make DB migrations to have two steps divided automatically. Nova was considering this as experimental. Moreover the code got removed in Mitaka-1. It also doesn’t solve long-running data migrations problem so doesn’t remove all the complications this spec is proposing.
We can also leave things as they are now and assume you need this downtime to upgrade. This probably isn’t what operators want and all the efforts we’ve already did (versioned objects, RPC compatibility) would be wasted.
Data model impact¶
We don’t expect impact to the data model itself, but to the way we’re doing migrations of the data model.
Changes in the model would need to be reviewed carefully to make sure they follow this guideline.
REST API impact¶
None
Security impact¶
None
Notifications impact¶
None
Other end user impact¶
None
Performance Impact¶
We can expect a little performance impact on DB calls because we’ll be queuing and saving additional columns instead of dropping unused ones right away. This shouldn’t be very significant however.
There will be overhead on the DB when executing small chunks of DB migrations while the cloud is running. This is however a tradeoff, as alternative to that is downtime of the whole cloud.
Nova does this in a very similar way and they doesn’t report performance issues.
Other deployer impact¶
It will be needed for deployer to be aware of how to do DB upgrade in a correct way. This will be documented, but knowledge will need to be propagated.
Developer impact¶
There will be huge impact on developers and reviewers as both groups will need to be aware how Cinder is doing DB migrations and write/review the code accordingly to prevent rolling upgrades feature from breaking.
Implementation¶
Assignee(s)¶
- Primary assignee:
michal-dulko-f (dulek)
- Other contributors:
Engagement from whole core team is needed to make sure the code is reviewed also from the perspective of live upgrades.
Work Items¶
Add unit test blocking contracting migrations (needed in Mitaka).
Add online DB data migrations bits to cinder-manage.
Write developer guide on how to write DB migrations properly.
Add DB migrations bits to Cinder’s upgrade documentation.
Dependencies¶
None for itself. Whole rolling upgrades story depends on RPC compatibility (API versioning and versioned objects with compatibility mode).
Testing¶
Partial Grenade tests should be added to make sure upgrading services one-by-one doesn’t break rolling upgrades. This should also cover DB migrations step.
Documentation Impact¶
We need to write detailed developer guide on how to write DB migrations in new model to make sure that we have a clear resource which we can point developers to.
To educate administrators and deployers on how to do rolling upgrades DB migrations parts should be added to general upgrade instruction for Cinder.
