5.16. Message Queue High Availability

status

ready

version

1.0

Abstract

This document describes a test plan for analysing high availability of OpenStack message bus. The measurement covers message queue and oslo.messaging library.

5.16.1. Test Plan

5.16.1.1. Test Environment

RabbitMQ is installed on 3 nodes in HA mode. Active monitoring is implemented with help of pacemaker. The test tool is executed on another host.

5.16.1.1.1. Preparation

Setup RabbitMQ cluster, for example by using the setup made by OpenStack Fuel.

5.16.1.1.2. Environment description

The environment description includes hardware specification of servers, network parameters, operation system and OpenStack deployment characteristics.

5.16.1.1.2.1. Hardware

This section contains list of all types of hardware nodes.

Parameter

Value

Comments

model

e.g. Supermicro X9SRD-F

CPU

e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
5.16.1.1.2.2. Network

This section contains list of interfaces and network parameters. For complicated cases this section may include topology diagram and switch parameters.

Parameter

Value

Comments

card model

e.g. Intel

driver

e.g. ixgbe

speed

e.g. 10G or 1G
5.16.1.1.2.3. Software

This section describes installed software.

Parameter

Value

Comments

OS

e.g. Ubuntu 14.04.3

oslo.messaging

e.g. 4.0.0

MQ Server

e.g. RabbitMQ 3.5.6

HA mode

e.g. Cluster

5.16.1.2. Test Case 1: Client and Server connected to Master, Master fails

../../_images/cmsm-km.svg

5.16.1.2.1. Description

In this test case both client and server are connected to RabbitMQ master node. The throughput is measured and at the same time RabbitMQ master process is terminated.

5.16.1.2.2. List of performance metrics

Test case result is time series showing message flow between client and server. It can be shown as chart and/or table. The average throughput and number of errors are calculated.

Priority

Value

Measurement Units

Description

1

Throughput

msg/sec

Number of messages per second

2

Latency

ms

The latency in message processing

5.16.1.2.3. Options

The test case is executed for different types of communication:

5.16.1.3. Test Case 2: Client and Server connected to Slave 1, Slave 1 fails

../../_images/cs1ss1-ks1.svg

5.16.1.3.1. Description

In this test case both client and server are connected to the same RabbitMQ slave node. The throughput is measured and at the same time RabbitMQ slave process is terminated.

5.16.1.3.2. List of performance metrics

Test case result is time series showing message flow between client and server. It can be shown as chart and/or table. The average throughput and number of errors are calculated.

Priority

Value

Measurement Units

Description

1

Throughput

msg/sec

Number of messages per second

2

Latency

ms

The latency in message processing

5.16.1.3.3. Options

The test case is executed for different types of communication:

5.16.1.4. Test Case 3: Client and Server on different slaves. Client Slave fails

../../_images/cs1ss2-ks2.svg

5.16.1.4.1. Description

In this test case client and server are connected to different RabbitMQ slave node. The throughput is measured and at the same time RabbitMQ slave process is terminated.

5.16.1.4.2. List of performance metrics

Test case result is time series showing message flow between client and server. It can be shown as chart and/or table. The average throughput and number of errors are calculated.

Priority

Value

Measurement Units

Description

1

Throughput

msg/sec

Number of messages per second

2

Latency

ms

The latency in message processing

5.16.1.4.3. Options

The test case is executed for different types of communication:

5.16.1.5. Test Case 4: Client on Master and Server on Slave, Master fails

../../_images/cmss2-km.svg

5.16.1.5.1. Description

In this test case client and server are connected to different RabbitMQ slave node: the client to master and server to slave. The throughput is measured and at the same time RabbitMQ slave process is terminated.

5.16.1.5.2. List of performance metrics

Test case result is time series showing message flow between client and server. It can be shown as chart and/or table. The average throughput and number of errors are calculated.

Priority

Value

Measurement Units

Description

1

Throughput

msg/sec

Number of messages per second

2

Latency

ms

The latency in message processing

5.16.1.5.3. Options

The test case is executed for different types of communication:

5.16.2. Tools

This section contains tools that can be used to perform the test plan.

5.16.2.1. Performa omsimulator

This section describes how to perform Message Queue High Availability with Performa tool.

5.16.2.1.1. Test environment preparation

To perform the test plan you will need to install performa toolkit. The toolkit uses MongoDB for stats processing and reporting.

$ pip install performa
The execution requires the following parameters to be set:

  • mongo-address - The host where MongoDB is installed

  • remote-user - The user that can connect to remote host in OpenStack cloud

  • tester-hosts - List of hosts were omsimulator will be executed

  • rabbit-url - RabbitMQ address, it has the form of rabbit://{<user>:<password>@<host>:<port>[,]}/

  • report - folder where to store the report

5.16.2.1.2. Execution

RPC CALL measurements:

$ performa --mongo-url <mongo-address> --mongo-db performa
  --scenario mq/omsimulator-ha-call --remote-user <remote-user>
  --vars "{tester_hosts: [<tester-hosts>], rabbit_url: \"<rabbit-url>\"}"
  --book <report>
example:

$ performa --mongo-url 172.20.9.20 --mongo-db performa --scenario mq/omsimulator-ha-call --remote-user root --vars "{rabbit_hosts: [node-123, node-111, node-58], tester_hosts: [node-144], rabbit_url: \"rabbit://nova:tGAPNtjHh8yvvkR69MooN1eD@node-58:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-111:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-123:5673/\"}" --book books/omsimulator-ha-call-cmsm-km

RPC CAST measurements:

$ performa --mongo-url <mongo-address> --mongo-db performa
  --scenario mq/omsimulator-ha-cast --remote-user <remote-user>
  --vars "{tester_hosts: [<tester-hosts>], rabbit_url: \"<rabbit-url>\"}"
  --book <report>
example:

$ performa --mongo-url 172.20.9.20 --mongo-db performa --scenario mq/omsimulator-ha-cast --remote-user root --vars "{rabbit_hosts: [node-123, node-111, node-58], tester_hosts: [node-144], rabbit_url: \"rabbit://nova:tGAPNtjHh8yvvkR69MooN1eD@node-58:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-111:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-123:5673/\"}" --book books/omsimulator-ha-cast-cmsm-km

NOTIFY measurements:

$ performa --mongo-url <mongo-address> --mongo-db performa
  --scenario mq/omsimulator-ha-notify --remote-user <remote-user>
  --vars "{tester_hosts: [<tester-hosts>], rabbit_url: \"<rabbit-url>\"}"
  --book <report>
example:

$ performa --mongo-url 172.20.9.20 --mongo-db performa --scenario mq/omsimulator-ha-notify --remote-user root --vars "{rabbit_hosts: [node-123, node-111, node-58], tester_hosts: [node-144], rabbit_url: \"rabbit://nova:tGAPNtjHh8yvvkR69MooN1eD@node-58:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-111:5673,nova:tGAPNtjHh8yvvkR69MooN1eD@node-123:5673/\"}" --book books/omsimulator-ha-notify-cmsm-km

5.16.3. Reports

Test plan execution reports: