OpenStack testing tools

With the rapid expansion of OpenStack, a variety of tools have emerged for testing different aspects of OpenStack. However there is currently no unified solution that can test availability, reliability, performance, and scalability of OpenStack infrastructure.

The aim of this SIG is to create self-healing infrastructure, and in order to move in to that direction there is a need for an automated destructive/extreme testing framework for OpenStack infrastructure. As there have already been several projects tackling various parts of the challenge, one goal is to avoid duplication of effort and see how we can take advantage of these already developed tools and come up with a unified solution.

Here is list of tools and the testing features that are provided by those (as of August 2019):

Tools/Features	Control plane testing	Data plane testing	Storage plane testing	Performance testing	Scalability testing	Concurrency testing	Disruptions/ Failure injection	OpenStack Deployment	Report generation	Active community 1
Rally (Uses Tempest)	X			X	X	X	X	X	X	Medium
Yardstick	X	X	X	X	X	X		X	X	NA since May 2019
Tempest	X			X						High
Shaker		X		X				X	X	Low
OS-Faults							X			NA since July 2019
KloudBuster		X	X	X	X				X	NA since June 2019
HA-Failure-Test (Uses Tempest)	X						X			NA since November 2016
Cloud99	X	X					X		X	NA since May 2017
Browbeat (Uses Rally and Shaker)	X	X		X	X			X		High
Harbinger		X	X							Low
Enos (Uses Rally and Shaker)	X	X		X				X	X	NA since May 2019
Chaostoolkit-openstack							X		X	NA since April 2019
Jepsen (not specific to OpenStack)							X		X	High
OPNFV-Doctor							X			NA since April 2019
Stepler (Uses OS-faults)							X			NA since February 2019
NFVbench		X	X	X	X				X	Medium
VMTP		X		X					X	Low
Eris (In development)	X	X		X	X	X	X		X	NA since November 2017

1

Commits per month: Low<5, Medium<15, High=>15, NA=No Activity

Rally

• Deployment of OpenStack

• Verification of deployment

• Benchmarking: Simulate real user load

• Profiler for scaling and performance issues

• Report generation: Processing and aggregation of results

De-facto control plane performance testing, allows to create different kind of tests including functional, concurrency, regression, load, scale capacity and chaos testing.

Yardstick

• Deployment

• Run compute, storage, network test scenarios using a runner

• Report generation with the following metrics:

  • Performance/Speed

  • Capacity/Scale

  • Availability/Reliability

Inspired by Rally, was developed mainly from the prespective of VNF(Virtual Network Function), but is a generic framework for testing scenarios in OpenStack for performance, scalability and availability/reliability.

Tempest

• Integration testing for live OpenStack

• Test OpenStack publicly available API and scenarios that demonstrate a working cloud

• Load testing possible as well

Mainly used for integration testing but non-functional testing possible as well.

Shaker

• Deploy instances and network in different topologies or network configurations using Heat

• Run network related test scenarios

• Report generation

Network load generation, testing and benchmarking. Based on ‘iperf(3)’ and ‘netperf’ -> network testing topologies.

OS-Faults

• Library for initiating or terminating service/node/container/network as a way for introducing fault injections.

• Allows for both hardware and software level faults

Used by Rally to introduce faults.

KloudBuster

• Benchmarking of data plane and storage services in OpenStack

A fully integrated framework which is easy to use with basic knowledge of OpenStack.

HA-Failure-Test

• High Availability testing of OpenStack components

• Set of Tempest scripts to test HA of components in HA cluster.

• Stop service(s), perform sanity and verify

Cloud99

• Monintoring: Nodes and services

• Disruptions: in services, node, containers can be parallel, sequential or round robin

• Runner: Perform actual tests with disruptions

• All 3 runs as seperate threads and parallel

Makes use of Rally runner.

Browbeat

• Performance and scalability testing

• Create Rally work loads

• Run Shaker for network configuration, benchmarking and testing

• CI and deployment: Can be used for CI or deploying data analysis tools.

Harbinger

• Manager for data plane testing frameworks in OpenStack

• Can similtaneously execute multiple testing frameworks (Shaker, Yardstick)

Enos

• Deployment of OpenStack on VMs or bare metal

• Setup with different configurations

• Benchmarking:

  • Rally scenario for control plane

  • Shaker scenario of data plane

• Report generation: Get metrics/reports in Grafana

chaostoolkit-openstack

• Driver to perform set of actions and probes.

A minimal chaostoolkit driver for OpenStack compute resources.

Jepsen

• Introduction of disruptions/faults

• Testing

• Report generation

• Checkers for analyzing history

Not specific to OpenStack but general library for distributed systems, written in Clojure.

OPNFV Doctor

• Fault management and maintenance in VIM(Virtualized Infrastructure Manager) e.g OpenStack

• Fault monitoring (Hardware, Hypervisor, Host OS faults)

• Fault notification

• Failure recovery (Switch to STBY config.)

Stepler

• Destructive testing

• Test the availability of OpenStack cluster incase of node/service failure(s)

Make use of os-faults.

NFVbench

• Measure network performance in OpenStack

• Data plane performance testing

• Benchmarking NFVi platforms and packet paths that are critical for NFV workloads

Adjusted more for NFV workloads.

VMTP

• Benchmarking VM to VM data path performance measurement

• Latency measurement

• TCP/UDP throughput measurement

Similar to NFVbench but more adapted towards enterprise workloads.

Eris

• Control and data plane performance testing

• High Availability testing

• Scalability Testing

• Concurrency Testing

• Benchmarking and results generation