Clusters¶

Update Cluster’s Profile¶

When profile_id is specified, the request will be interpreted as a wholistic update to all nodes across the cluster. The targeted use case is to do a cluster wide system upgrade. For example, replacing glance images used by the cluster nodes when new kernel patches have been applied or software defects have been fixed.

When receiving such an update request, the Senlin engine will check if the new profile referenced does exist and whether the new profile has the same profile type as that of the existing profile. Exceptions will be thrown if any verification has failed and thus the request is rejected.

After the engine has validated the request, an Action of CLUSTER_UPDATE is created and queued internally for execution. Later on, when a worker thread picks up the action for execution, it will first lock the whole cluster and mark the cluster status as UPDATING. It will then fork NODE_UPDATE actions per node inside the cluster, which are in turn queued for execution. Other worker threads will pick up the node level update action for execution and mark the action as completed/failed. When all these node level updates are completed, the CLUSTER_UPDATE operation continues and marks the cluster as ACTIVE again.

The cluster update operation may take a long time to complete, depending on the response time from the underlying profile operations. Note also, when there is a update policy is attached to the cluster and enabled, the update operation may be split into several batches so that 1) there is a minimum number of nodes remained in service at any time; 2) the pressure on the underlying service is controlled.

Update Cluster Size Properties¶

When either one of the desired_capacity, min_size and max_size property is specified in the CLUSTER_UPDATE request, it may lead to a resize operation on the cluster.

The Senlin API will do a preliminary validation upon the new property values. For example, if both min_size and max_size are specified, they have to be integers and the value for max_size is greater than the value for min_size, unless the value of max_size is -1 which means the upper bound of cluster size is unlimited.

When the request is then received by the Senlin engine, the engine first retrieves the cluster properties from the database and do further cross-verifications between the new property values and the current values. For example, it is treated as an invalid request if a user has specified value for min_size but no value for max_size, however the new min_size is greater than the existing max_size of the cluster. In this case, the user has to provide a valid max_size to override the existing value, or he/she has to lower the min_size value so that the request becomes acceptable.

Once the cross-verification has passed, Senlin engine will calculate the new desired_capacity and adjust the size of the cluster if deemed necessary. For example, when the cluster size is below the new min_size, some nodes will be removed from the cluster; when the cluster size is above the new max_size, new nodes will be created and added to the cluster. If the desired_capacity is set and the property value falls between the new range of cluster size, Senlin tries resize the cluster to the desired_capacity.

When the size of the cluster is adjusted, Senlin engine will check if there are relevant policies attached to the cluster so that the engine will add and/or remove nodes in a predictable way.

Update Other Cluster Properties¶

The update to other cluster properties is relatively straightforward. Senlin engine simply verifies the data types when necessary and override the existing property values in the database.

Note that in the cases where multiple properties are specified in a single CLUSTER_UPDATE request, some will take a longer time to complete than others. Any mixes of update properties are acceptable to the Senlin API and the engine.

Adding Nodes to a Cluster¶

Senlin API provides the add_nodes action for user to add some existing nodes into the specified cluster. The parameter for this action is interpreted as a list in which each item is the UUID, name or short ID of a node.

When receiving an add_nodes action request, the Senlin API only validates if the parameter is a list and if the list is empty. After this validation, the request is forwarded to the Senlin engine for processing.

The Senlin engine will examine nodes in the list one by one and see if any of the following conditions is true. Senlin engine rejects the request if so.

• Any node from the list is not in ACTIVE state?
• Any node from the list is still member of another cluster?
• Any node from the list is not found in the database?
• Number of nodes to add is zero?

When this phase of validation succeeds, the request is translated into a CLUSTER_ADD_NODES builtin action and queued for execution. The engine returns to the user an action UUID for checking.

When the action is picked up by a worker thread for execution, Senlin checks if the profile type of the nodes to be added matches that of the cluster. Finally, a number of NODE_JOIN action is forked and executed from the CLUSTER_ADD_NODES action. When NODE_JOIN actions complete, the CLUSTER_ADD_NODES action returns with success.

In the cases where there are load-balancing policies attached to the cluster, the CLUSTER_ADD_NODES action will save the list of UUIDs of the new nodes into the action’s data field so that those policies could update the associated resources.

Deleting Nodes from a Cluster¶

Senlin API provides the del_nodes action for user to delete some existing nodes from the specified cluster. The parameter for this action is interpreted as a list in which each item is the UUID, name or short ID of a node.

When receiving a del_nodes action request, the Senlin API only validates if the parameter is a list and if the list is empty. After this validation, the request is forwarded to the Senlin engine for processing.

The Senlin engine will examine nodes in the list one by one and see if any of the following conditions is true. Senlin engine rejects the request if so.

• Any node from the list cannot be found from the database?
• Any node from the list is not member of the specified cluster?
• Number of nodes to delete is zero?

When this phase of validation succeeds, the request is translated into a CLUSTER_DEL_NODES builtin action and queued for execution. The engine returns to the user an action UUID for checking.

When the action is picked up by a worker thread for execution, Senlin forks a number of NODE_DELETE actions and execute them asynchronously. When all forked actions complete, the CLUSTER_DEL_NODES returns with a success.

In the cases where there are load-balancing policies attached to the cluster, the CLUSTER_DEL_NODES action will save the list of UUIDs of the deleted nodes into the action’s data field so that those policies could update the associated resources.

Note also that by default Senlin won’t destroy the nodes that are deleted from the cluster. It simply removes the nodes from the cluster so that they become orphan nodes. Senlin also provides a paramater destroy_after_deletion for this action so that a user can request the deleted node(s) to be destroyed right away, instead of becoming orphan nodes.

Replacing Nodes in a Cluster¶

Senlin API provides the replace_nodes action for user to replace some existing nodes in the specified cluster. The parameter for this action is interpreted as a dict in which each item is the node-pair{OLD_NODE:NEW_NODE}. The key OLD_NODE is the UUID, name or short ID of a node to be replaced, and the value NEW_NODE is the UUID, name or short ID of a node as replacement.

When receiving a replace_nodes action request, the Senlin API only validates if the parameter is a dict and if the dict is empty. After this validation, the request is forwarded to the Senlin engine for processing.

The Senlin engine will examine nodes in the dict one by one and see if all of the following conditions is true. Senlin engine accepts the request if so.

• All nodes from the list can be found from the database.
• All replaced nodes from the list are the members of the specified cluster.
• All replacement nodes from the list are not the members of any cluster.
• The profile types of all replacement nodes match that of the specified cluster.
• The statuses of all replacement nodes are ACTIVE.

When this phase of validation succeeds, the request is translated into a CLUSTER_REPLACE_NODES builtin action and queued for execution. The engine returns to the user an action UUID for checking.

When the action is picked up by a worker thread for execution, Senlin forks a number of NODE_LEAVE and related NODE_JOIN actions, and execute them asynchronously. When all forked actions complete, the CLUSTER_REPLACE_NODES returns with a success.

Resizing a Cluster¶

In addition to the cluster_update request, Senlin provides a dedicated API for adjusting the size of a cluster, i.e. cluster_resize. This operation is designed for the auto-scaling and manual-scaling use cases.

Below is a list of API parameters recognizable by the Senlin API when parsing the JSON body of a cluster_resize request:

• adjustment_type: type of adjustment to be performed where the value should be one of the followings:

  • EXACT_CAPACITY: the adjustment is about the targeted size of the cluster;
  • CHANGE_IN_CAPACITY: the adjustment is about the number of nodes to be added or removed from the cluster and this is the default setting;
  • CHANGE_IN_PERCENTAGE: the adjustment is about a relative percentage of the targeted cluster.

  This field is mandatory.

• number: adjustment number whose value will be interpreted base on the value of adjustment_type. This field is mandatory.

• min_size: the new lower bound for the cluster size;

• max_size: the new upper bound for the cluster size;

• min_step: the minimum number of nodes to be added or removed when the adjustment_type is set to CHANGE_IN_PERCENTAGE and the absolute value computed is less than 1;

• strict: a boolean value indicating whether the service should do a best-effort resizing operation even if the request cannot be fully met.

For example, the following request is about increasing the size of the cluster by 20% and Senlin can try a best-effort if the calculated size is greater than the upper limit of the cluster size:

{
  "adj_type": "CHANGE_IN_PERCENTAGE",
  "number": "20",
  "strict": False,
}

When Senlin API receives a cluster_resize request, it first validates the data type of the values and the sanity of the value collection. For example, you cannot specify a min_size greater than the current upper bound (i.e. the max_size property of the cluster) if you are not providing a new max_size that is greater than the min_size.

After the request is forwarded to the Senlin engine, the engine will further validates the parameter values against the targeted cluster. When all validations pass, the request is converted into a CLUSTER_RESIZE action and queued for execution. The API returns the cluster properties and the UUID of the action at this moment.

When executing the action, Senlin will analyze the request parameters and determine the operations to be performed to meet user’s requirement. The corresponding cluster properties are updated before the resize operation is started.

Scaling in/out a Cluster¶

As a convenience method, Senlin provides the scale_out and the scale_in action API for clusters. With these two APIs, a user can request a cluster to be resized by the specified number of nodes.

The scale_out and the scale_in APIs both take a parameter named count which is a positive integer. The integer parameter is optional, and it specifies the number of nodes to be added or removed if provided. When it is omitted from the request JSON body, Senlin engine will check if the cluster has any relevant policies attached that will decide the number of nodes to be added or removed respectively. The Senlin engine will use the outputs from these policies as the number of nodes to create (or delete) if such policies exist. When the request does contain a count parameter and there are policies governing the scaling arguments, the count parameter value may be overridden/ignored.

When a scale_out or a scale_in request is received by the Senlin engine, a CLUSTER_SCALE_OUT or a CLUSTER_SCALE_IN action is then created and queued for execution after some validation of the parameter value.

A worker thread picks up the action and execute it. The worker will check if there are outputs from policy checkings. For CLUSTER_SCALE_OUT actions, the worker checks if the policies checked has left a count key in the dictionary named creation from the action’s runtime data attribute. The worker will use such a count value for node creation. For a CLUSTER_SCALE_OUT action, the worker checks if the policies checked has left a count key in the dictionary named deletion from the action’s runtime data attribute. The worker will use such a count value for node deletion.

Note that both scale_out and scale_in actions will adjust the desired_capacity property of the target cluster.

Attaching a Policy to a Cluster¶

Once a policy is attached (bound) to a cluster, it will be enforced when related actions are performed on that cluster, unless the policy is (temporarily) disabled on the cluster.

When attaching a policy to a cluster, the following properties can be specified:

• enabled: a boolean indicating whether the policy should be enabled on the cluster once attached. Default is True. When specified, it will override the default setting for the policy.

Upon receiving the policy_attach request, the Senlin engine will perform some validations then translate the request into a CLUSTER_ATTACH_POLICY action and queue the action for execution. The action’s UUID is then returned to Senlin API and finally the requestor.

When the engine executes the action, it will try find if the policy is already attached to the cluster. This checking was not done previously because the engine must ensure that the cluster has been locked before this checking, or else there might be race conditions.

The engine calls the policy’s attach method when attaching the policy and record the binding into database if the attach method returns a positive response.

Currently, Senlin does not allow two policies of the same type to be attached to the same cluster. This constraint may be relaxed in future, but for now, it is checked and enforced before a policy gets attached to a cluster.

Policies attached to a cluster are cached at the target cluster as part of its runtime rt data structure. This is an optimization regarding DB queries.

Detaching a Policy from a Cluster¶

Once a policy is attached to a cluster, it can be detached from the cluster at user’s request. The only parameter required for the policy_detach action API is policy_id, which can be the UUID, the name or the short ID of the policy.

Upon receiving a policy_detach request, the Senlin engine will perform some validations then translate the request into a CLUSTER_DETACH_POLICY action and queue the action for execution. The action’s UUID is then returned to Senlin API and finally the requestor.

When the Senlin engine executes the CLUSTER_DETACH_POLICY action, it will try find if the policy is already attached to the cluster. This checking was not done previously because the engine must ensure that the cluster has been locked before this checking, or else there might be race conditions.

The engine calls the policy’s detach method when detaching the policy from the cluster and then removes the binding record from database if the detach method returns a True value.

Policies attached to a cluster are cached at the target cluster as part of its runtime rt data structure. This is an optimization regarding DB queries. The CLUSTER_DETACH_POLICY action will invalidate the cache when detaching a policy from a cluster.

Updating a Policy on a Cluster¶

When a policy is attached to a cluster, there are some properties pertaining to the binding. These properties can be updated as long as the policy is still attached to the cluster. The properties that can be updated include:

• enabled: a boolean value indicating whether the policy should be enabled or disabled. There are cases where some policies have to be temporarily disabled when other manual operations going on.

Upon receiving the policy_update request, Senlin API performs some basic validations on the parameters passed.

Senlin engine translates the policy_update request into an action CLUSTER_UPDATE_POLICY and queue it for execution. The UUID of the action is then returned to Senlin API and eventually the requestor.

During execution of the CLUSTER_UPDATE_POLICY action, Senlin engine simply updates the binding record in the database and returns.