Maintenance tasks¶

Verify cluster status¶

Compare the output of the following command with the following output. It should give you information about the status of your cluster.

# ansible galera_container -m shell -a "mysql -h 127.0.0.1 \
-e 'show status like \"%wsrep_cluster_%\";'"
node3_galera_container-3ea2cbd3 | FAILED | rc=1 >>
ERROR 2002 (HY000): Can't connect to local MySQL server
through socket '/var/run/mysqld/mysqld.sock' (2)

node2_galera_container-49a47d25 | FAILED | rc=1 >>
ERROR 2002 (HY000): Can't connect to local MySQL server
through socket '/var/run/mysqld/mysqld.sock' (2)

node4_galera_container-76275635 | success | rc=0 >>
Variable_name             Value
wsrep_cluster_conf_id     7
wsrep_cluster_size        1
wsrep_cluster_state_uuid  338b06b0-2948-11e4-9d06-bef42f6c52f1
wsrep_cluster_status      Primary

In the previous example, all but one node was responding.

Gracefully shutting down the MariaDB service on all but one node allows the remaining operational node to continue processing SQL requests. When gracefully shutting down multiple nodes, perform the actions sequentially to retain operation.

Start a cluster¶

Gracefully shutting down all nodes destroys the cluster. Starting or restarting a cluster from zero nodes requires creating a new cluster on one of the nodes.

1. Start a new cluster on the most advanced node. Change to the playbooks directory and check the seqno value in the grastate.dat file on all of the nodes:

   # ansible galera_container -m shell -a "cat /var/lib/mysql/grastate.dat"
   node2_galera_container-49a47d25 | success | rc=0 >>
   # GALERA saved state version: 2.1
   uuid:    338b06b0-2948-11e4-9d06-bef42f6c52f1
   seqno:   31
   cert_index:
   
   node3_galera_container-3ea2cbd3 | success | rc=0 >>
   # GALERA saved state version: 2.1
   uuid:    338b06b0-2948-11e4-9d06-bef42f6c52f1
   seqno:   31
   cert_index:
   
   node4_galera_container-76275635 | success | rc=0 >>
   # GALERA saved state version: 2.1
   uuid:    338b06b0-2948-11e4-9d06-bef42f6c52f1
   seqno:   31
   cert_index:
   

   In this example, all nodes in the cluster contain the same positive seqno values as they were synchronized just prior to graceful shutdown. If all seqno values are equal, any node can start the new cluster.

   ## for init
   # /etc/init.d/mysql start --wsrep-new-cluster
   ## for systemd
   # systemctl set-environment _WSREP_NEW_CLUSTER='--wsrep-new-cluster'
   # systemctl start mysql
   # systemctl set-environment _WSREP_NEW_CLUSTER=''
   

   Please also have a look at upstream starting a cluster page

   This can also be done with the help of ansible using the shell module:

   # ansible galera_container -m shell -a "/etc/init.d/mysql start --wsrep-new-cluster" --limit galera_container[0]
   

   This command results in a cluster containing a single node. The wsrep_cluster_size value shows the number of nodes in the cluster.

   node2_galera_container-49a47d25 | FAILED | rc=1 >>
   ERROR 2002 (HY000): Can't connect to local MySQL server
   through socket '/var/run/mysqld/mysqld.sock' (111)
   
   node3_galera_container-3ea2cbd3 | FAILED | rc=1 >>
   ERROR 2002 (HY000): Can't connect to local MySQL server
   through socket '/var/run/mysqld/mysqld.sock' (2)
   
   node4_galera_container-76275635 | success | rc=0 >>
   Variable_name             Value
   wsrep_cluster_conf_id     1
   wsrep_cluster_size        1
   wsrep_cluster_state_uuid  338b06b0-2948-11e4-9d06-bef42f6c52f1
   wsrep_cluster_status      Primary
   

2. Restart MariaDB on the other nodes (replace [0] from previous ansible command with [1:]) and verify that they rejoin the cluster.

   node2_galera_container-49a47d25 | success | rc=0 >>
   Variable_name             Value
   wsrep_cluster_conf_id     3
   wsrep_cluster_size        3
   wsrep_cluster_state_uuid  338b06b0-2948-11e4-9d06-bef42f6c52f1
   wsrep_cluster_status      Primary
   
   node3_galera_container-3ea2cbd3 | success | rc=0 >>
   Variable_name             Value
   wsrep_cluster_conf_id     3
   wsrep_cluster_size        3
   wsrep_cluster_state_uuid  338b06b0-2948-11e4-9d06-bef42f6c52f1
   wsrep_cluster_status      Primary
   
   node4_galera_container-76275635 | success | rc=0 >>
   Variable_name             Value
   wsrep_cluster_conf_id     3
   wsrep_cluster_size        3
   wsrep_cluster_state_uuid  338b06b0-2948-11e4-9d06-bef42f6c52f1
   wsrep_cluster_status      Primary
   

Galera cluster recovery¶

Run the galera-install playbook using the galera-bootstrap tag to automatically recover a node or an entire environment.

1. Run the following Ansible command to show the failed nodes:

   # openstack-ansible galera-install.yml --tags galera-bootstrap
   

The cluster comes back online after completion of this command. If this fails, please review restarting the cluster and recovering the primary component in the galera documentation as they’re invaluable for a full cluster recovery.

# ansible galera_container -m shell -a "cat /var/lib/mysql/grastate.dat"
node3_galera_container-3ea2cbd3 | success | rc=0 >>
# GALERA saved state
version: 2.1
uuid:    338b06b0-2948-11e4-9d06-bef42f6c52f1
seqno:   -1
cert_index:

node2_galera_container-49a47d25 | success | rc=0 >>
# GALERA saved state
version: 2.1
uuid:    338b06b0-2948-11e4-9d06-bef42f6c52f1
seqno:   -1
cert_index:

node4_galera_container-76275635 | success | rc=0 >>
# GALERA saved state
version: 2.1
uuid:    338b06b0-2948-11e4-9d06-bef42f6c52f1
seqno:   -1
cert_index:

Create a RabbitMQ cluster¶

RabbitMQ clusters can be formed in two ways:

• Manually with rabbitmqctl
• Declaratively (list of cluster nodes in a config, with rabbitmq-autocluster, or rabbitmq-clusterer plugins)

There are two types of nodes you can configure: disk and RAM nodes. Most commonly, you will use your nodes as disk nodes (preferred). Whereas RAM nodes are more of a special configuration used in performance clusters.

RabbitMQ nodes and the CLI tools use an erlang cookie to determine whether or not they have permission to communicate. The cookie is a string of alphanumeric characters and can be as short or as long as you would like.

The default location of the cookie on *nix environments is /var/lib/rabbitmq/.erlang.cookie or in $HOME/.erlang.cookie.

To form a RabbitMQ Cluster, you start by taking independent RabbitMQ brokers and re-configuring these nodes into a cluster configuration.

Using a 3 node example, you would be telling nodes 2 and 3 to join the cluster of the first node.

1. Login to the 2nd and 3rd node and stop the rabbitmq application.

2. Join the cluster, then restart the application:

   rabbit2$ rabbitmqctl stop_app
   Stopping node rabbit@rabbit2 ...done.
   rabbit2$ rabbitmqctl join_cluster rabbit@rabbit1
   Clustering node rabbit@rabbit2 with [rabbit@rabbit1] ...done.
   rabbit2$ rabbitmqctl start_app
   Starting node rabbit@rabbit2 ...done.
   

Check the RabbitMQ cluster status¶

1. Run rabbitmqctl cluster_status from either node.

You will see rabbit1 and rabbit2 are both running as before.

The difference is that the cluster status section of the output, both nodes are now grouped together:

rabbit1$ rabbitmqctl cluster_status
Cluster status of node rabbit@rabbit1 ...
[{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2]}]},
{running_nodes,[rabbit@rabbit2,rabbit@rabbit1]}]
...done.

To add the third rabbit node to the cluster, repeat the above process by stopping the rabbitmq application on the third node.

1. Join the cluster, and restart the application on the third node.

2. Execute rabbitmq cluster_status to see all 3 nodes:

   rabbit1$ rabbitmqctl cluster_status
   Cluster status of node rabbit@rabbit1 ...
   [{nodes,[{disc,[rabbit@rabbit1,rabbit@rabbit2,rabbit@rabbit3]}]},
    {running_nodes,[rabbit@rabbit3,rabbit@rabbit2,rabbit@rabbit1]}]
   ...done.
   

Stop and restart a RabbitMQ cluster¶

To stop and start the cluster, keep in mind the order in which you shut the nodes down. The last node you stop, needs to be the first node you start. This node is the master.

If you start the nodes out of order, you could run into an issue where it thinks the current master should not be the master and drops the messages to ensure that no new messages are queued while the real master is down.

RabbitMQ and mnesia¶

Mnesia is a distributed database that RabbitMQ uses to store information about users, exchanges, queues, and bindings. Messages, however are not stored in the database.

For more information about Mnesia, see the Mnesia overview.

To view the locations of important Rabbit files, see File Locations.

Repair a partitioned RabbitMQ cluster for a single-node¶

Invariably due to something in your environment, you are likely to lose a node in your cluster. In this scenario, multiple LXC containers on the same host are running Rabbit and are in a single Rabbit cluster.

If the host still shows as part of the cluster, but it is not running, execute:

# rabbitmqctl start_app

However, you may notice some issues with your application as clients may be trying to push messages to the un-responsive node. To remedy this, forget the node from the cluster by executing the following:

1. Ensure rabbit is not running on the node:

   # rabbitmqctl stop_app
   

2. On the Rabbit2 node, execute:

   # rabbitmqctl forget_cluster_node rabbit@rabbit1
   

By doing this, the cluster can continue to run effectively and you can repair the failing node.

rabbit1$ rabbitmqctl start_app
Starting node rabbit@rabbit1 ...

Error: inconsistent_cluster: Node rabbit@rabbit1 thinks it's clustered with node rabbit@rabbit2, but rabbit@rabbit2 disagrees

rabbit1$ rabbitmqctl reset
Resetting node rabbit@rabbit1 ...done.
rabbit1$ rabbitmqctl start_app
Starting node rabbit@mcnulty ...
...done.

Repair a partitioned RabbitMQ cluster for a multi-node cluster¶

The same concepts apply to a multi-node cluster that exist in a single-node cluster. The only difference is that the various nodes will actually be running on different hosts. The key things to keep in mind when dealing with a multi-node cluster are:

• When the entire cluster is brought down, the last node to go down must be the first node to be brought online. If this does not happen, the nodes will wait 30 seconds for the last disc node to come back online, and fail afterwards.

  If the last node to go offline cannot be brought back up, it can be removed from the cluster using the forget_cluster_node command.

• If all cluster nodes stop in a simultaneous and uncontrolled manner, (for example, with a power cut) you can be left with a situation in which all nodes think that some other node stopped after them. In this case you can use the force_boot command on one node to make it bootable again.

Consult the rabbitmqctl manpage for more information.

Running the shell module¶

The two most common modules used are the shell and copy modules. The shell module takes the command name followed by a list of space delimited arguments. It is almost like the command module, but runs the command through a shell (/bin/sh) on the remote node.

For example, you could use the shell module to check the amount of disk space on a set of Compute hosts:

$ ansible compute_hosts -m shell -a 'df -h'

To check on the status of your Galera cluster:

$ ansible galera_container -m shell -a "mysql -h 127.0.0.1\
-e 'show status like \"%wsrep_cluster_%\";'"

When a module is being used as an ad-hoc command, there are a few parameters that are not required. For example, for the chdir command, there is no need to chdir=/home/user ls when running Ansible from the CLI:

$ ansible compute_hosts -m shell -a 'ls -la /home/user'

For more information, see shell - Execute commands in nodes.

Running the copy module¶

The copy module copies a file on a local machine to remote locations. Use the fetch module to copy files from remote locations to the local machine. If you need variable interpolation in copied files, use the template module. For more information, see copy - Copies files to remote locations.

The following example shows how to move a file from your deployment host to the /tmp directory on a set of remote machines:

$ ansible remote_machines -m copy -a 'src=/root/FILE \
dest=/tmp/FILE'

If you want to gather files from remote machines, use the fetch module. The fetch module stores files locally in a file tree, organized by the hostname from remote machines and stores them locally in a file tree, organized by hostname.

The following examples shows the nova-compute.log file being pulled from a single Compute host:

root@libertylab:/opt/rpc-openstack/openstack-ansible/playbooks# ansible compute_hosts -m fetch -a 'src=/var/log/nova/nova-compute.log dest=/tmp'
aio1 | success >> {
    "changed": true,
    "checksum": "865211db6285dca06829eb2215ee6a897416fe02",
    "dest": "/tmp/aio1/var/log/nova/nova-compute.log",
    "md5sum": "dbd52b5fd65ea23cb255d2617e36729c",
    "remote_checksum": "865211db6285dca06829eb2215ee6a897416fe02",
    "remote_md5sum": null
}

root@libertylab:/opt/rpc-openstack/openstack-ansible/playbooks# ls -la /tmp/aio1/var/log/nova/nova-compute.log
-rw-r--r-- 1 root root 2428624 Dec 15 01:23 /tmp/aio1/var/log/nova/nova-compute.log

Using tags¶

Tags are similar to the limit flag for groups except tags are used to only run specific tasks within a playbook. For more information on tags, see Tags and Understanding ansible tags.

Ansible forks¶

The default MaxSessions setting for the OpenSSH Daemon is 10. Each Ansible fork makes use of a session. By default, Ansible sets the number of forks to 5. However, you can increase the number of forks used in order to improve deployment performance in large environments.

Note that more than 10 forks will cause issues for any playbooks which use delegate_to or local_action in the tasks. It is recommended that the number of forks are not raised when executing against the control plane, as this is where delegation is most often used.

The number of forks used may be changed on a permanent basis by including the appropriate change to the ANSIBLE_FORKS in your .bashrc file. Alternatively it can be changed for a particular playbook execution by using the --forks CLI parameter. For example, the following executes the nova playbook against the control plane with 10 forks, then against the compute nodes with 50 forks.

# openstack-ansible --forks 10 os-nova-install.yml --limit compute_containers
# openstack-ansible --forks 50 os-nova-install.yml --limit compute_hosts

For more information about forks, please see the following references:

• OpenStack-Ansible Bug 1479812
• Ansible forks entry for ansible.cfg
• Ansible Performance Tuning

Scale individual services¶

Individual OpenStack services, and other open source project services, run within containers. It is possible to scale out these services by modifying the /etc/openstack_deploy/openstack_user_config.yml file.

1. Navigate into the /etc/openstack_deploy/openstack_user_config.yml file.

2. Access the deployment groups section of the configuration file. Underneath the deployment group name, add an affinity value line to container scales OpenStack services:

   infra_hosts:
     infra1:
       ip: 10.10.236.100
       # Rabbitmq
       affinity:
         galera_container: 1
         rabbit_mq_container: 2
   

   In this example, galera_container has a container value of one. In practice, any containers that do not need adjustment can remain at the default value of one, and should not be adjusted above or below the value of one.

   The affinity value for each container is set at one by default. Adjust the affinity value to zero for situations where the OpenStack services housed within a specific container will not be needed when scaling out other required services.

3. Update the container number listed under the affinity configuration to the desired number. The above example has galera_container set at one and rabbit_mq_container at two, which scales RabbitMQ services, but leaves Galera services fixed.

4. Run the appropriate playbook commands after changing the configuration to create the new containers, and install the appropriate services.

   For example, run the openstack-ansible lxc-containers-create.yml rabbitmq-install.yml commands from the openstack-ansible/playbooks repository to complete the scaling process described in the example above:

   $ cd openstack-ansible/playbooks
   $ openstack-ansible lxc-containers-create.yml rabbitmq-install.yml
   

Destroy and recreate containers¶

Resolving some issues may require destroying a container, and rebuilding that container from the beginning. It is possible to destroy and re-create a container with the lxc-containers-destroy.yml and lxc-containers-create.yml commands. These Ansible scripts reside in the openstack-ansible/playbooks repository.

1. Navigate to the openstack-ansible directory.

2. Run the openstack-ansible lxc-containers-destroy.yml commands, specifying the target containers and the container to be destroyed.

   $ openstack-ansible lxc-containers-destroy.yml --limit "CONTAINER_NAME"
   $ openstack-ansible lxc-containers-create.yml --limit "CONTAINER_NAME"
   

3. Replace ``CONTAINER_NAME`` with the target container.

Finding ports for your external load balancer¶

As explained in the previous section, you can find (in each role documentation) the default variables used for the public interface endpoint ports.

For example, the os_glance documentation lists the variable glance_service_publicuri. This contains the port used for the reaching the service externally. In this example, it is equal to to glance_service_port, whose value is 9292.

As a hint, you could find the whole list of public URI defaults by executing the following:

cd /etc/ansible/roles
grep -R -e publicuri -e port *