Nova introduced the placement API service in the 14.0.0 Newton release. This is a separate REST API stack and data model used to track resource provider inventories and usages, along with different classes of resources. For example, a resource provider can be a compute node, a shared storage pool, or an IP allocation pool. The placement service tracks the inventory and usage of each provider. For example, an instance created on a compute node may be a consumer of resources such as RAM and CPU from a compute node resource provider, disk from an external shared storage pool resource provider and IP addresses from an external IP pool resource provider.
The types of resources consumed are tracked as classes. The service
provides a set of standard resource classes (for example DISK_GB
,
MEMORY_MB
, and VCPU
) and provides the ability to define custom
resource classes as needed.
Each resource provider may also have a set of traits which describe qualitative aspects of the resource provider. Traits describe an aspect of a resource provider that cannot itself be consumed but a workload may wish to specify. For example, available disk may be solid state drives (SSD).
The following specifications represent the stages of design and development of resource providers and the Placement service. Implementation details may have changed or be partially complete at this time.
The placement-api service must be deployed at some point after you have upgraded to the 14.0.0 Newton release but before you can upgrade to the 15.0.0 Ocata release. This is so that the resource tracker in the nova-compute service can populate resource provider (compute node) inventory and allocation information which will be used by the nova-scheduler service in Ocata.
1. Deploy the API service
At this time the placement API code is still in Nova alongside the compute REST
API code (nova-api). So once you have upgraded nova-api to Newton you already
have the placement API code, you just need to install the service. Nova
provides a nova-placement-api
WSGI script for running the service with
Apache, nginx or other WSGI-capable web servers. Depending on what packaging
solution is used to deploy OpenStack, the WSGI script may be in /usr/bin
or /usr/local/bin
.
Note
The placement API service is currently developed within Nova but it is designed to be as separate as possible from the existing code so that it can eventually be split into a separate project.
nova-placement-api
, as a standard WSGI script, provides a module level
application
attribute that most WSGI servers expect to find. This means it
is possible to run it with lots of different servers, providing flexibility in
the face of different deployment scenarios. Common scenarios include:
In all of these scenarios the host, port and mounting path (or prefix) of the
application is controlled in the web server’s configuration, not in the
configuration (nova.conf
) of the placement application.
When placement was first added to DevStack it used the mod_wsgi
style.
Later it was updated to use mod_proxy_uwsgi. Looking at those changes can
be useful for understanding the relevant options.
DevStack is configured to host placement at /placement
on either the
default port for http or for https (80
or 443
) depending on whether TLS
is being used. Using a default port is desirable.
By default, the placement application will get its configuration for settings
such as the database connection URL from /etc/nova/nova.conf
. The directory
the configuration file will be found in can be changed by setting
OS_PLACEMENT_CONFIG_DIR
in the environment of the process that starts the
application.
Note
When using uwsgi with a front end (e.g., apache2 or nginx) something needs to ensure that the uwsgi process is running. In DevStack this is done with systemd. This is one of many different ways to manage uwsgi.
This document refrains from declaring a set of installation instructions for the placement service. This is because a major point of having a WSGI application is to make the deployment as flexible as possible. Because the placement API service is itself stateless (all state is in the database), it is possible to deploy as many servers as desired behind a load balancing solution for robust and simple scaling. If you familiarize yourself with installing generic WSGI applications (using the links in the common scenarios list, above), those techniques will be applicable here.
2. Synchronize the database
In the Newton release the Nova api database is the only deployment
option for the placement API service and the resources it manages. After
upgrading the nova-api service for Newton and running the
nova-manage api_db sync
command the placement tables will be created.
With the Rocky release, it has become possible to use a separate database for
placement. If placement_database.connection
is
configured with a database connect string, that database will be used for
storing placement data. Once the database is created, the
nova-manage api_db sync
command will create and synchronize both the
nova api and placement tables. If [placement_database]/connection
is not
set, the nova api database will be used.
Note
At this time there is no facility for migrating existing placement data from the nova api database to a placement database. There are many ways to do this. Which one is best will depend on the environment.
3. Create accounts and update the service catalog
Create a placement service user with an admin role in Keystone.
The placement API is a separate service and thus should be registered under a placement service type in the service catalog as that is what the resource tracker in the nova-compute node will use to look up the endpoint.
Devstack sets up the placement service on the default HTTP port (80) with a
/placement
prefix instead of using an independent port.
4. Configure and restart nova-compute services
The 14.0.0 Newton nova-compute service code will begin reporting resource provider inventory and usage information as soon as the placement API service is in place and can respond to requests via the endpoint registered in the service catalog.
nova.conf
on the compute nodes must be updated in the [placement]
group to contain credentials for making requests from nova-compute to the
placement-api service.
Note
After upgrading nova-compute code to Newton and restarting the
service, the nova-compute service will attempt to make a connection
to the placement API and if that is not yet available a warning will
be logged. The nova-compute service will keep attempting to connect
to the placement API, warning periodically on error until it is
successful. Keep in mind that Placement is optional in Newton, but
required in Ocata, so the placement service should be enabled before
upgrading to Ocata. nova.conf on the compute nodes will need to be
updated in the [placement]
group for credentials to make requests
from nova-compute to the placement-api service.
The following sub-sections provide notes on upgrading to a given target release.
Note
As a reminder, the nova-status upgrade check tool can be used to help determine the status of your deployment and how ready it is to perform an upgrade.
nova-compute
service will fail to start in Ocata unless the
[placement]
section of nova.conf on the compute is configured. As
mentioned in the deployment steps above, the Placement service should be
deployed by this point so the computes can register and start reporting
inventory and allocation information. If the computes are deployed
and configured before the Placement service, they will continue to try
and reconnect in a loop so that you do not need to restart the nova-compute
process to talk to the Placement service after the compute is properly
configured.nova.scheduler.filter_scheduler.FilterScheduler
in Ocata will
fallback to not using the Placement service as long as there are older
nova-compute
services running in the deployment. This allows for rolling
upgrades of the computes to not affect scheduling for the FilterScheduler.
However, the fallback mechanism will be removed in the 16.0.0 Pike release
such that the scheduler will make decisions based on the Placement service
and the resource providers (compute nodes) registered there. This means if
the computes are not reporting into Placement by Pike, build requests will
fail with NoValidHost errors.nova-scheduler
service to Ocata and restart it, things will still work.
The scheduler will gracefully handle the absence of the Placement service.
However, once all computes are upgraded, the scheduler not being able to make
requests to Placement will result in NoValidHost errors.CoreFilter
, RamFilter
and
DiskFilter
from the list of enabled FilterScheduler filters such that
scheduling decisions are not based on CPU, RAM or disk usage. Once all
computes are reporting into the Placement service, however, and the
FilterScheduler starts to use the Placement service for decisions, those
excluded filters are ignored and the scheduler will make requests based on
VCPU, MEMORY_MB and DISK_GB inventory. If you wish to effectively ignore
that type of resource for placement decisions, you will need to adjust the
corresponding cpu_allocation_ratio
, ram_allocation_ratio
, and/or
disk_allocation_ratio
configuration options to be very high values, e.g.
9999.0.nova-scheduler
process.The nova.scheduler.filter_scheduler.FilterScheduler
in Pike will
no longer fall back to not using the Placement Service, even if older
computes are running in the deployment.
The FilterScheduler now requests allocation candidates from the Placement service during scheduling. The allocation candidates information was introduced in the Placement API 1.10 microversion, so you should upgrade the placement service before the Nova scheduler service so that the scheduler can take advantage of the allocation candidate information.
The scheduler gets the allocation candidates from the placement API and uses those to get the compute nodes, which come from the cell(s). The compute nodes are passed through the enabled scheduler filters and weighers. The scheduler then iterates over this filtered and weighed list of hosts and attempts to claim resources in the placement API for each instance in the request. Claiming resources involves finding an allocation candidate that contains an allocation against the selected host’s UUID and asking the placement API to allocate the requested instance resources. We continue performing this claim request until success or we run out of allocation candidates, resulting in a NoValidHost error.
For a move operation, such as migration, allocations are made in Placement against both the source and destination compute node. Once the move operation is complete, the resource tracker in the nova-compute service will adjust the allocations in Placement appropriately.
For a resize to the same host, allocations are summed on the single compute node. This could pose a problem if the compute node has limited capacity. Since resizing to the same host is disabled by default, and generally only used in testing, this is mentioned for completeness but should not be a concern for production deployments.
1.17
in order to support Request Traits During Scheduling.
This means you must upgrade the placement service before upgrading any
nova-scheduler services to Queens.nova-api
service now requires the [placement]
section to be
configured in nova.conf if you are using a separate config file just for
that service. This is because the nova-api
service now needs to talk
to the placement service in order to (1) delete resource provider allocations
when deleting an instance and the nova-compute
service on which that
instance is running is down (2) delete a nova-compute
service record via
the DELETE /os-services/{service_id}
API and (3) mirror aggregate host
associations to the placement service. This change is idempotent if
[placement]
is not configured in nova-api
but it will result in new
warnings in the logs until configured.connection
setting in
a [placement_database]
group is set in configuration, that group will be
used to describe where and how placement data is stored.The placement API service has its own REST API and data model. One can get a sample of the REST API via the functional test gabbits.
The placement API uses microversions for making incremental changes to the API which client requests must opt into.
It is especially important to keep in mind that nova-compute is a client of the placement REST API and based on how Nova supports rolling upgrades the nova-compute service could be Newton level code making requests to an Ocata placement API, and vice-versa, an Ocata compute service in a cells v2 cell could be making requests to a Newton placement API.
This documents the changes made to the REST API with every microversion change. The description for each version should be a verbose one which has enough information to be suitable for use in user documentation.
New in version Newton.
This is the initial version of the placement REST API that was released in Nova 14.0.0 (Newton). This contains the following routes:
/resource_providers
/resource_providers/allocations
/resource_providers/inventories
/resource_providers/usages
/allocations
New in version Ocata.
The 1.1 version adds support for associating aggregates with resource providers.
The following new operations are added:
GET /resource_providers/{uuid}/aggregates
PUT /resource_providers/{uuid}/aggregates
New in version Ocata.
Placement API version 1.2 adds basic operations allowing an admin to create, list and delete custom resource classes.
The following new routes are added:
GET /resource_classes
POST /resource_classes
PUT /resource_classes/{name}
DELETE /resource_classes/{name}
GET /resource_classes/{name}
Custom resource classes must begin with the prefix CUSTOM_
and contain only
the letters A through Z, the numbers 0 through 9 and the underscore _
character.
New in version Ocata.
Version 1.3 adds support for listing resource providers that are members of any
of the list of aggregates provided using a member_of
query parameter:
?member_of=in:{agg1_uuid},{agg2_uuid},{agg3_uuid}
New in version Ocata.
The 1.4 version adds support for querying resource providers that have the
ability to serve a requested set of resources. A new “resources” query string
parameter is now accepted to the GET /resource_providers
API call. This
parameter indicates the requested amounts of various resources that a provider
must have the capacity to serve. The “resources” query string parameter takes
the form:
?resources=$RESOURCE_CLASS_NAME:$AMOUNT,$RESOURCE_CLASS_NAME:$AMOUNT
For instance, if the user wishes to see resource providers that can service a request for 2 vCPUs, 1024 MB of RAM and 50 GB of disk space, the user can issue a request to:
GET /resource_providers?resources=VCPU:2,MEMORY_MB:1024,DISK_GB:50
If the resource class does not exist, then it will return a HTTP 400.
Note
The resources filtering is also based on the min_unit, max_unit and step_size of the inventory record. For example, if the max_unit is 512 for the DISK_GB inventory for a particular resource provider and a GET request is made for DISK_GB:1024, that resource provider will not be returned. The min_unit is the minimum amount of resource that can be requested for a given inventory and resource provider. The step_size is the increment of resource that can be requested for a given resource on a given provider.
New in version Pike.
Placement API version 1.5 adds DELETE method for deleting all inventory for a resource provider. The following new method is supported:
DELETE /resource_providers/{uuid}/inventories
Delete all inventories for a given resource provider
New in version Pike.
The 1.6 version adds basic operations allowing an admin to create, list, and delete custom traits, also adds basic operations allowing an admin to attach traits to a resource provider.
The following new routes are added:
GET /traits
PUT /traits/{name}
GET /traits/{name}
DELETE /traits/{name}
GET /resource_providers/{uuid}/traits
PUT /resource_providers/{uuid}/traits
DELETE /resource_providers/{uuid}/traits
Custom traits must begin with the prefix CUSTOM_
and contain only the
letters A through Z, the numbers 0 through 9 and the underscore _
character.
New in version Pike.
The 1.7 version changes handling of PUT /resource_classes/{name}
to be a
create or verification of the resource class with {name}
. If the resource
class is a custom resource class and does not already exist it will be created
and a 201
response code returned. If the class already exists the response
code will be 204
. This makes it possible to check or create a resource
class in one request.
New in version Pike.
The 1.8 version adds project_id
and user_id
required request parameters
to PUT /allocations
.
New in version Pike.
The 1.9 version adds usages that can be queried by a project or project/user.
The following new routes are added:
GET /usages?project_id=<project_id>
GET /usages?project_id=<project_id>&user_id=<user_id>
New in version Pike.
The 1.10 version brings a new REST resource endpoint for getting a list of allocation candidates. Allocation candidates are collections of possible allocations against resource providers that can satisfy a particular request for resources.
GET /resource_providers
response¶New in version Queens.
The /resource_providers/{rp_uuid}/allocations
endpoint has been available
since version 1.0, but was not listed in the links
section of the
GET /resource_providers
response. The link is included as of version 1.11.
New in version Queens.
In version 1.12 the request body of a PUT /allocations/{consumer_uuid}
is expected to have an object
for the allocations
property, not as
array
as with earlier microversions. This puts the request body more in
alignment with the structure of the GET /allocations/{consumer_uuid}
response body. Because the PUT
request requires user_id
and
project_id
in the request body, these fields are added to the GET
response. In addition, the response body for GET /allocation_candidates
is updated so the allocations in the alocation_requests
object work
with the new PUT
format.
New in version Queens.
Version 1.13 gives the ability to set or clear allocations for more than
one consumer UUID with a request to POST /allocations
.
New in version Queens.
The 1.14 version introduces the concept of nested resource providers. The resource provider resource now contains two new attributes:
parent_provider_uuid
indicates the provider’s direct parent, or null if
there is no parent. This attribute can be set in the call to POST
/resource_providers
and PUT /resource_providers/{uuid}
if the attribute
has not already been set to a non-NULL value (i.e. we do not support
“reparenting” a provider)root_provider_uuid
indicates the UUID of the root resource provider in
the provider’s tree. This is a read-only attributeA new in_tree=<UUID>
parameter is now available in the GET
/resource-providers
API call. Supplying a UUID value for the in_tree
parameter will cause all resource providers within the “provider tree” of the
provider matching <UUID>
to be returned.
New in version Queens.
Throughout the API, ‘last-modified’ headers have been added to GET responses and those PUT and POST responses that have bodies. The value is either the actual last modified time of the most recently modified associated database entity or the current time if there is no direct mapping to the database. In addition, ‘cache-control: no-cache’ headers are added where the ‘last-modified’ header has been added to prevent inadvertent caching of resources.
New in version Queens.
Add support for a limit
query parameter when making a
GET /allocation_candidates
request. The parameter accepts an integer
value, N
, which limits the maximum number of candidates returned.
New in version Queens.
Add the required
parameter to the GET /allocation_candidates
API. It
accepts a list of traits separated by ,
. The provider summary in the
response will include the attached traits also.
New in version Rocky.
Add support for the required
query parameter to the GET
/resource_providers
API. It accepts a comma-separated list of string trait
names. When specified, the API results will be filtered to include only
resource providers marked with all the specified traits. This is in addition to
(logical AND) any filtering based on other query parameters.
Trait names which are empty, do not exist, or are otherwise invalid will result in a 400 error.
New in version Rocky.
Enhance the payloads for the GET /resource_providers/{uuid}/aggregates
response and the PUT /resource_providers/{uuid}/aggregates
request and
response to be identical, and to include the resource_provider_generation
.
As with other generation-aware APIs, if the resource_provider_generation
specified in the PUT
request does not match the generation known by the
server, a 409 Conflict error is returned.
New in version Rocky.
The POST /resource_providers
API, on success, returns 200 with a payload
representing the newly-created resource provider, in the same format as the
corresponding GET /resource_providers/{uuid}
call. This is to allow the
caller to glean automatically-set fields, such as UUID and generation, without
a subsequent GET.
New in version Rocky.
Add support for the member_of
query parameter to the GET
/allocation_candidates
API. It accepts a comma-separated list of UUIDs for
aggregates. Note that if more than one aggregate UUID is passed, the
comma-separated list must be prefixed with the “in:” operator. If this
parameter is provided, the only resource providers returned will be those in
one of the specified aggregates that meet the other parts of the request.
New in version Rocky.
Add support for expressing traits which are forbidden when filtering
GET /resource_providers
or GET /allocation_candidates
. A forbidden
trait is a properly formatted trait in the existing required
parameter,
prefixed by a !
. For example required=!STORAGE_DISK_SSD
asks that the
results not include any resource providers that provide solid state disk.
New in version Rocky.
JSON formatted error responses gain a new attribute, code
, with a value
that identifies the type of this error. This can be used to distinguish errors
that are different but use the same HTTP status code. Any error response which
does not specifically define a code will have the code
placement.undefined_code
.
New in version Rocky.
Add support for specifying multiple member_of
query parameters to the GET
/resource_providers
API. When multiple member_of
query parameters are
found, they are AND’d together in the final query. For example, issuing a
request for GET /resource_providers?member_of=agg1&member_of=agg2
means get
the resource providers that are associated with BOTH agg1 and agg2. Issuing a
request for GET /resource_providers?member_of=in:agg1,agg2&member_of=agg3
means get the resource providers that are associated with agg3 and are also
associated with any of (agg1, agg2).
GET /allocation_candidates
¶New in version Rocky.
GET /allocation_candidates
is enhanced to accept numbered groupings of
resource, required/forbidden trait, and aggregate association requests. A
resources
query parameter key with a positive integer suffix (e.g.
resources42
) will be logically associated with required
and/or
member_of
query parameter keys with the same suffix (e.g. required42
,
member_of42
). The resources, required/forbidden traits, and aggregate
associations in that group will be satisfied by the same resource provider in
the response. When more than one numbered grouping is supplied, the
group_policy
query parameter is required to indicate how the groups should
interact. With group_policy=none
, separate groupings - numbered or
unnumbered - may or may not be satisfied by the same provider. With
group_policy=isolate
, numbered groups are guaranteed to be satisfied by
different providers - though there may still be overlap with the unnumbered
group. In all cases, each allocation_request
will be satisfied by providers
in a single non-sharing provider tree and/or sharing providers associated via
aggregate with any of the providers in that tree.
The required
and member_of
query parameters for a given group are
optional. That is, you may specify resources42=XXX
without a corresponding
required42=YYY
or member_of42=ZZZ
. However, the reverse (specifying
required42=YYY
or member_of42=ZZZ
without resources42=XXX
) will
result in an error.
The semantic of the (unnumbered) resources
, required
, and member_of
query parameters is unchanged: the resources, traits, and aggregate
associations specified thereby may be satisfied by any provider in the same
non-sharing tree or associated via the specified aggregate(s).
New in version Rocky.
Starting with this version, it is allowed to set the reserved value of the resource provider inventory to be equal to total.
New in version Rocky.
Include all resource class inventories in the provider_summaries
field in
response of the GET /allocation_candidates
API even if the resource class
is not in the requested resources.
New in version Rocky.
A new generation field has been added to the consumer concept. Consumers are
the actors that are allocated resources in the placement API. When an
allocation is created, a consumer UUID is specified. Starting with microversion
1.8, a project and user ID are also required. If using microversions prior to
1.8, these are populated from the incomplete_consumer_project_id
and
incomplete_consumer_user_id
config options from the [placement]
section.
The consumer generation facilitates safe concurrent modification of an allocation.
A consumer generation is now returned from the following URIs:
GET /resource_providers/{uuid}/allocations
The response continues to be a dict with a key of allocations
, which itself
is a dict, keyed by consumer UUID, of allocations against the resource
provider. For each of those dicts, a consumer_generation
field will now be
shown.
GET /allocations/{consumer_uuid}
The response continues to be a dict with a key of allocations
, which
itself is a dict, keyed by resource provider UUID, of allocations being
consumed by the consumer with the {consumer_uuid}
. The top-level dict will
also now contain a consumer_generation
field.
The value of the consumer_generation
field is opaque and should only be
used to send back to subsequent operations on the consumer’s allocations.
The PUT /allocations/{consumer_uuid}
URI has been modified to now require a
consumer_generation
field in the request payload. This field is required to
be null
if the caller expects that there are no allocations already
existing for the consumer. Otherwise, it should contain the generation that the
caller understands the consumer to be at the time of the call.
A 409 Conflict
will be returned from PUT /allocations/{consumer_uuid}
if there was a mismatch between the supplied generation and the consumer’s
generation as known by the server. Similarly, a 409 Conflict
will be
returned if during the course of replacing the consumer’s allocations another
process concurrently changed the consumer’s allocations. This allows the caller
to react to the concurrent write by re-reading the consumer’s allocations and
re-issuing the call to replace allocations as needed.
The PUT /allocations/{consumer_uuid}
URI has also been modified to accept
an empty allocations object, thereby bringing it to parity with the behaviour
of POST /allocations
, which uses an empty allocations object to indicate
that the allocations for a particular consumer should be removed. Passing an
empty allocations object along with a consumer_generation
makes PUT
/allocations/{consumer_uuid}
a safe way to delete allocations for a
consumer. The DELETE /allocations/{consumer_uuid}
URI remains unsafe to
call in deployments where multiple callers may simultaneously be attempting to
modify a consumer’s allocations.
The POST /allocations
URI variant has also been changed to require a
consumer_generation
field in the request payload for each consumer
involved in the request. Similar responses to PUT
/allocations/{consumer_uuid}
are returned when any of the consumers
generations conflict with the server’s view of those consumers or if any of the
consumers involved in the request are modified by another process.
Warning
In all cases, it is absolutely NOT SAFE to create and modify allocations for a consumer using different microversions where one of the microversions is prior to 1.28. The only way to safely modify allocations for a consumer and satisfy expectations you have regarding the prior existence (or lack of existence) of those allocations is to always use microversion 1.28+ when calling allocations API endpoints.
New in version Rocky.
Add support for nested resource providers with the following two features.
1) GET /allocation_candidates
is aware of nested providers. Namely, when
provider trees are present, allocation_requests
in the response of
GET /allocation_candidates
can include allocations on combinations of
multiple resource providers in the same tree.
2) root_provider_uuid
and parent_provider_uuid
are added to
provider_summaries
in the response of GET /allocation_candidates
.
Except where otherwise noted, this document is licensed under Creative Commons Attribution 3.0 License. See all OpenStack Legal Documents.