Managing etcd

Kolla Ansible can manage the lifecycle of an etcd cluster and supports the following operations:

  • Bootstrapping a clean multi-node etcd cluster.

  • Adding a new member to the etcd cluster.

  • Optionally, automatically removing a deleted node from the etcd cluster.

It is highly recommended to read the operator documentation for the version of etcd deployed in the cluster.

Note

Once an etcd cluster is bootstrapped, the etcd service takes most of its configuration from the etcd database itself.

This pattern is very different from many other Kolla Ansible services, and is a source of confusion for operators unfamiliar with etcd.

Cluster vs. Node Bootstrapping

Kolla Ansible distinguishes between two forms of bootstrapping in an etcd cluster:

  • Bootstrapping multiple nodes at the same time to bring up a new cluster.

  • Bootstrapping a single node to add it to an existing cluster.

These corresponds to the new and existing parameters for ETCD_INITIAL_CLUSTER_STATE in the upstream documentation. Once an etcd node has completed bootstrap, the bootstrap configuration is ignored, even if it is changed.

Kolla Ansible will decide to perform a new cluster bootstrap if it detects that there is no existing data on the etcd nodes. Otherwise it assumes that there is a healthy etcd cluster and it will add a new node to it.

Forcing Bootstrapping

Kolla Ansible looks for the kolla_etcd volume on the node. If this volume is available, it assumes that the bootstrap process has run on the node and the volume contains the required config.

However, if the process was interrupted (externally, or by an error), this volume might be misconfigured. In order to prevent data loss, manual intervention is required.

Before retriggering bootstrap make sure that there is no valuable data on the volume. This could be because the node was not in service, or that the data is persisted elsewhere.

To retrigger a bootstrap (for either the cluster, or for a single node), remove the volume from all affected nodes by running:

docker volume rm kolla_etcd

Rerunning Kolla Ansible will then trigger the appropriate workflow and either a blank cluster will be bootstrapped, or an empty member will be added to the existing cluster.

Manual Commands

In order to manage etcd manually, the etcdctl command can be used inside the etcd container. This command has been set up with the appropriate environment variables for integrating with automation.

etcdctl is configured with json output by default, you can override that if you are running it yourself:

# list cluster members in a human-readable table
docker exec -it etcd etcdctl -w table member list

Removing Dead Nodes

If globals.yml has the value etcd_remove_deleted_members: "yes" then etcd nodes that are not in the inventory will be removed from the etcd cluster.

Any errors in the inventory can therefore cause unintended removal.

To manually remove a dead node from the etcd cluster, use the following commands:

# list cluster members and identify dead member
docker exec -it etcd etcdctl -w table member list
# remove dead member
docker exec -it etcd etcdctl member remove MEMBER_ID_IN_HEX