Rolling Upgrades¶
Note
The Rolling Upgrades feature is EXPERIMENTAL and its use in production systems is currently not supported.
Scope of this document¶
This page describes one way to perform a rolling upgrade from Newton to Ocata for a particular configuration of Glance services. There may be other ways to perform a rolling upgrade from Newton to Ocata for other configurations of Glance services, but those are beyond the scope of this document. For the experimental rollout of rolling upgrades, we describe only the following simple case.
Prerequisites¶
- MySQL/MariaDB 5.5 or later
- Glance running Images API v2 only
- Glance not using the Glance Registry
- Multiple Glance nodes
- A load balancer or some other type of redirection device is being used in front of the Glance nodes in such a way that a node can be dropped out of rotation, that is, that Glance node continues running the Glance service but is no longer having requests routed to it
Procedure¶
Following is the process to upgrade Glance with zero downtime:
- Backup the Glance database.
- Choose an arbitrary Glance node or provision a new node to install the new release. If an existing Glance node is chosen, gracefully stop the Glance services. In what follows, this node will be referred to as the NEW NODE.
Note
Gracefully stopping services
Before stopping the Glance processes on a node, one may choose to wait until all the existing connections drain out. This could be achieved by taking the node out of rotation, that is, by ensuring that requests are no longer routed to that node. This way all the requests that are currently being processed will get a chance to finish processing. However, some Glance requests like uploading and downloading images may last a long time. This increases the wait time to drain out all connections and consequently the time to upgrade Glance completely. On the other hand, stopping the Glance services before the connections drain out will present the user with errors. While arguably this is not downtime given that Images API requests are continually being serviced by other nodes, this is nonetheless an unpleasant user experience for the user whose in-flight request has terminated in an error. Hence, an operator must be judicious when stopping the services.
Upgrade the NEW NODE with new release and update the configuration accordingly. DO NOT start the Glance services on the NEW NODE at this time.
Using the NEW NODE, expand the database using the command
glance-manage db expand.Then, also on the NEW NODE, perform the data migrations using the command
glance-manage db migrate.The data migrations must be completed before you proceed to the next step.
Start the Glance processes on the NEW NODE. It is now ready to receive traffic from the load balancer.
Taking one node at a time from the remaining nodes, for each node:
- Stop the Glance processes gracefully as described in Step 2, above. Do not proceed until the “old” Glance services on the node have been completely shut down.
- Upgrade the node to the new release (and corresponding configuration).
- Start the updated Glance processes on the upgraded node.
After ALL of the nodes have been upgraded to run the new Glance services, and there are NO nodes running any old Glance services, contract the database by running the command
glance manage db contractfrom any one of the upgraded nodes.
Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.