Cells Layout (v2)¶

Cross-cell instance migrations¶

Currently it is not possible to migrate an instance from a host in one cell to a host in another cell. This may be possible in the future, but it is currently unsupported. This impacts cold migration, resizes, live migrations, evacuate, and unshelve operations.

Quota-related quirks¶

Quotas are now calculated live at the point at which an operation would consume more resource, instead of being kept statically in the database. This means that a multi-cell environment may incorrectly calculate the usage of a tenant if one of the cells is unreachable, as those resources cannot be counted. In this case, the tenant may be able to consume more resource from one of the available cells, putting them far over quota when the unreachable cell returns. In the future, placement will provide us with a consistent way to calculate usage independent of the actual cell being reachable.

Performance of listing instances¶

With multiple cells, the instance list operation may not sort and paginate results properly when crossing multiple cell boundaries. Further, the performance of a sorted list operation will be considerably slower than with a single cell.

Notifications¶

With a multi-cell environment with multiple message queues, it is likely that operators will want to configure a separate connection to a unified queue for notifications. This can be done in the configuration file of all nodes. Refer to the oslo.messaging configuration documentation for more details.

Nova Metadata API service¶

Starting from the Stein release, the Nova Metadata API service can be run either globally or per cell using the api.local_metadata_per_cell configuration option.

Global

If you have networks that span cells, you might need to run Nova metadata API globally. When running globally, it should be configured as an API-level service with access to the api_database.connection information. The nova metadata API service must not be run as a standalone service in this case (e.g. must not be run via the nova-api-metadata script).

Local per cell

Running Nova metadata API per cell can have better performance and data isolation in a muli-cell deployment. If your networks are segmented along cell boundaries, then you can run Nova metadata API service per cell. If you choose to run it per cell, you should also configure each Neutron metadata-agent to point to the corresponding nova-metadata-api. The nova metadata API service must be run as a standalone service in this case (e.g. must be run via the nova-api-metadata script).

Consoleauth service and console proxies¶

As of Rocky, the nova-consoleauth service has been deprecated and cell databases are used for storing token authorizations. All new consoles will be supported by the database backend and existing consoles will be reset. Console proxies must be run per cell because the new console token authorizations are stored in cell databases.

Operations Requiring upcalls¶

If you deploy multiple cells with a superconductor as described above, computes and cell-based conductors will not have the ability to speak to the scheduler as they are not connected to the same MQ. This is by design for isolation, but currently the processes are not in place to implement some features without such connectivity. Thus, anything that requires a so-called “upcall” will not function. This impacts the following:

1. Instance reschedules during boot and resize (part 1)

   Note

   This has been resolved in the Queens release [2].

2. Instance affinity reporting from the compute nodes to scheduler

3. The late anti-affinity check during server create and evacuate

4. Querying host aggregates from the cell

   Note

   This has been resolved in the Rocky release [3].

5. Attaching a volume and [cinder]/cross_az_attach=False

6. Instance reschedules during boot and resize (part 2)

The first is simple: if you boot an instance, it gets scheduled to a compute node, fails, it would normally be re-scheduled to another node. That requires scheduler intervention and thus it will not work in Pike with a multi-cell layout. If you do not rely on reschedules for covering up transient compute-node failures, then this will not affect you. To ensure you do not make futile attempts at rescheduling, you should set [scheduler]/max_attempts=1 in nova.conf.

The second two are related. The summary is that some of the facilities that Nova has for ensuring that affinity/anti-affinity is preserved between instances does not function in Pike with a multi-cell layout. If you don’t use affinity operations, then this will not affect you. To make sure you don’t make futile attempts at the affinity check, you should set [workarounds]/disable_group_policy_check_upcall=True and [filter_scheduler]/track_instance_changes=False in nova.conf.

The fourth is currently only a problem when performing live migrations using the XenAPI driver and not specifying --block-migrate. The driver will attempt to figure out if block migration should be performed based on source and destination hosts being in the same aggregate. Since aggregates data has migrated to the API database, the cell conductor will not be able to access the aggregate information and will fail.

The fifth is a problem because when a volume is attached to an instance in the nova-compute service, and [cinder]/cross_az_attach=False in nova.conf, we attempt to look up the availability zone that the instance is in which includes getting any host aggregates that the instance.host is in. Since the aggregates are in the API database and the cell conductor cannot access that information, so this will fail. In the future this check could be moved to the nova-api service such that the availability zone between the instance and the volume is checked before we reach the cell, except in the case of boot from volume where the nova-compute service itself creates the volume and must tell Cinder in which availability zone to create the volume. Long-term, volume creation during boot from volume should be moved to the top-level superconductor which would eliminate this AZ up-call check problem.

The sixth is detailed in bug 1781286 and similar to the first issue. The issue is that servers created without a specific availability zone will have their AZ calculated during a reschedule based on the alternate host selected. Determining the AZ for the alternate host requires an “up call” to the API DB.