Upgrades Guideline¶

Prerequisites¶

• Multiple Heat nodes.
• A load balancer or some other type of redirection device is being used in front of nodes that run heat-api services in such a way that a node can be dropped out of rotation. That node continues running the Heat services (heat-api or heat-engine) but is no longer having requests routed to it.

Procedure¶

These following steps are the process to upgrade Heat with minimal downtime:

1. Install the code for the next version of Heat either in a virtual environment or a separate control plane node, including all the python dependencies.

2. Using the newly installed heat code, run the following command to sync the database up to the most recent version. These schema change operations should have minimal or no effect on performance, and should not cause any operations to fail.

   heat-manage db_sync
   

3. At this point, new columns and tables may exist in the database. These DB schema changes are done in a way that both the N and N+1 release can perform operations against the same schema.

4. Create a new rabbitmq vhost for the new release and change the transport_url configuration in heat.conf file to be:

   transport_url = rabbit://<user>:<password>@<host>:5672/<new_vhost>

   for all upgrade services.

5. Stop heat-engine gracefully, Heat has supported graceful shutdown features (see the spec about rolling upgrades). Then start new heat-engine with new code (and corresponding configuration).

   Note

   Remember to do Step 4, this would ensure that the existing engines would not communicate with the new engine.

6. A heat-api service is then upgraded and started with the new rabbitmq vhost.

   Note

   The second way to do this step is switch heat-api service to use new vhost first (but remember not to shut down heat-api) and upgrade it.

7. The above process can be followed till all heat-api and heat-engine services are upgraded.

   Note

   Make sure that all heat-api services has been upgraded before you start to upgrade the last heat-engine service.

   Warning

   With the convergence architecture, whenever a resource completes the engine will send RPC messages to another (or the same) engine to start work on the next resource(s) to be processed. If the last engine is going to be shut down gracefully, it will finish what it is working on, which may post more messages to queues. It means the graceful shutdown does not wait for queues to drain. The shutdown leaves some messages unprocessed and any IN_PROGRESS stacks would get stuck without any forward progress. The operator must be careful when shutting down the last engine, make sure queues have no unprocessed messages before doing it. The operator can check the queues directly with RabbitMQ’s management plugin.

8. Once all services are upgraded, double check the DB and services