5.3. OpenStack control plane performance test plan¶

status: ready
version: 1.0
Abstract: This test plan aims to provide set of tests to identify Control Plane performance against given OpenStack cloud using simple minimalistic set of Rally tests.

5.3.1. Test Plan¶

This test plan describes several test cases that can cover almost all most important in terms of performance basic cloud operations e.g. VMs creation, work with the security groups, authentication and other operations.

5.3.1.1. Test Environment¶

5.3.1.1.1. Preparation¶

This test plan is performed either against existing OpenStack cloud with pre-installed Rally framework or can be executed via Rally from very beginning including deployment of the OpenStack cloud. As an option verification (Tempest testing) can be run prior the benchmarking (scenarios to be described in this document).

5.3.1.1.2. Environment description¶

The environment description includes hardware specification of servers, network parameters, operation system and OpenStack deployment characteristics.

5.3.1.1.2.1. Hardware¶

This section contains list of all types of hardware nodes.

Parameter	Value	Comments
model		e.g. Supermicro X9SRD-F
CPU		e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
role		e.g. compute or network

5.3.1.1.2.2. Network¶

This section contains list of interfaces and network parameters. For complicated cases this section may include topology diagram and switch parameters.

Parameter	Value	Comments
network role		e.g. provider or public
card model		e.g. Intel
driver		e.g. ixgbe
speed		e.g. 10G or 1G
MTU		e.g. 9000
offloading modes		e.g. default

5.3.1.1.2.3. Software¶

This section describes installed software.

Parameter	Value	Comments
OS		e.g. Ubuntu 14.04.3
OpenStack		e.g. Mitaka
Hypervisor		e.g. KVM
Neutron plugin		e.g. ML2 + OVS
L2 segmentation		e.g. VLAN / VxLAN / GRE
virtual routers		e.g. HA / DVR

5.3.1.2. Test tool¶

Rally is a benchmarking tool that was designed specifically for OpenStack API testing. To make this possible, Rally automates and unifies multi-node OpenStack deployment, cloud verification, benchmarking & profiling. This is a simple way to check cloud workability and performance of control plane operations running on it.

5.3.1.3. Test Case 1: Boot, attach, migrate and delete server with security groups¶

5.3.1.3.1. Description¶

The most user-facing control plane operation is new virtual machine creation. At the same time security groups management is very time consuming operation in case of lots VMs attached to the same security group, therefore it’s vital to understand these operations performance. Special Rally plugin can be written for this purpose.

5.3.1.3.2. Parameters¶

Name	Description
IMAGE	Image from which boot server
FLAVOR	Flavor type from which boot server
SEC_GROUP_COUNT	Count of security groups to be created in one iteration
RULES_PER_SECURITY_GROUP	Count of rules to be added to each security group
VOLUME_SIZE size	Size of volume to be created in Cinder
CONCURRENCY	Amount of parallel executors
ITERATIONS	Total amount of iterations processed by all executors

5.3.1.3.3. List of performance metrics¶

Priority	Value	Measurement Units	Description
1		sec	Time of atomic operations

5.3.1.3.4. Measuring performance values¶

Successively create SEC_GROUP_COUNT security groups through Nova API. Duration of this step represent time that control plane process create_SEC_GROUP_COUNT_security_groups atomic operation.
Successively create RULES_PER_SECURITY_GROUP rules for security groups through Nova API.
Create server with FLAVOR flavor from IMAGE image through Nova API
Create VOLUME_SIZE GB size volume through Cinder API.
Attach created volume to server.
Migrate server to pre-chosen host through Nova API.
List all security groups through Nova API.
Detach volume from server through Nova API.
Delete server through Nova API.
Delete volume through Cinder API.
Successively delete SEC_GROUP_COUNT security group through Nova API.

This 11 steps executed successively in CONCURRENCY parallel executors. One cycle of this 11 steps is called as iteration. ITERATIONS is a total amount of iterations which was processed by executors.

At the end of this test case you should calculate average, 90% percentile, 50% percentile, minimum and maximum for each step. You need to fill the following tables with calculated values:

5.3.1.4. Cinder¶

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_volume
delete_volume

5.3.1.5. Neutron¶

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_N_security_groups
delete_N_security_groups
create_M_rules
delete_M_rules

Note

Change operation name to appropriate regarding SEC_GROUP_COUNT and RULES_PER_SECURITY_GROUP values.

5.3.1.6. Nova¶

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_server
attach_volume
live_migrate
detach_volume
delete_server

5.3.1.6.1. Example of Rally scenario configuration¶

{% set flavor_name = flavor_name or "m1.tiny" %}
{% set image_name = image_name or "^(cirros.*uec|TestVM)$" %}
{
   "NovaPerformancePlugin.boot_attach_and_delete_server_with_secgroups": [
        {
            "args": {
                "flavor": {
                    "name": "{{flavor_name}}"
                },
                "image": {
                    "name": "{{image_name}}"
                },
                "security_group_count": 2,
                "rules_per_security_group": 10,
                "boot_server_kwargs": { "auto_assign_nic" : true },
                "create_volume_kwargs": {},
                "volume_size": 1,
                "do_delete": true
            },
            "runner": {
                "type": "constant",
                "times": {{ 2 * compute }},
                "concurrency": {{concurrency}}
            },
            "context": {
                "users": {
                    "tenants": 1,
                    "users_per_tenant": 10
                },
                "quotas": {
                    "neutron": {
                        "network": -1,
                        "security_group": -1,
                        "security_group_rule": -1,
                        "port": -1,
                        "subnet": -1,
                        "router": -1
                    },
                    "nova": {
                        "instances": -1,
                        "cores": -1,
                        "ram": -1,
                        "floating_ips": -1,
                        "security_groups": -1,
                        "security_group_rules": -1
                    },
                    "cinder": {
                        "volumes": -1,
                        "gigabytes": -1,
                        "snapshots": -1
                    }
                },
                "network": {
                    "start_cidr": "{{ "100.1.0.0/25" if gre_enabled else "1.0.0.0/25" }}",
                    "networks_per_tenant": 10
                }
            },
            "sla": {
                "failure_rate": { "max": 0 }
            }
        }
    ]
}

This scenario use custom Rally plugin: nova_performance.py

5.3.1.7. Test Case 2: Create and delete image¶

5.3.1.7.1. Description¶

To cover Glance control plane operations simple crete and delete image scenario can be used.

5.3.1.7.2. Parameters¶

Name	Description
IMAGE	Image to upload to glance
CONTAINER_FORMAT	Container format to create
DISK_FORMAT	Disk format to create
CONCURRENCY	Amount of parallel executors
ITERATIONS	Total amount of iterations processed by all executors

5.3.1.7.3. List of performance metrics¶

Priority	Value	Measurement Units	Description
1		sec	Time of atomic operations

5.3.1.7.4. Measuring performance values¶

Create image from IMAGE with CONTAINER_FORMAT container format and DISK_FORMAT disk format through Glance API.
Delete image from Glance through Glance API.

This 2 steps executed successively in CONCURRENCY parallel executors. One cycle of this 2 steps is called as iteration. ITERATIONS is a total amount of iterations which was processed by executors.

At the end of this test case you should calculate average, 90% percentile, 50% percentile, minimum and maximum for each step. You need to fill the following tables with calculated values:

5.3.1.8. Glance¶

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_image
delete_image

5.3.1.8.1. Example of Rally scenario configuration¶

{% set http_server_with_glance_images = http_server_with_glance_images or "127.0.0.1" %}
{
    "GlanceImages.create_and_delete_image": [
        {
            "args": {
                "image_location": "http://{{http_server_with_glance_images}}/images/glance_1g_image.raw",
                "container_format": "bare",
                "disk_format": "raw"
            },
            "runner": {
                "type": "constant",
                "times": 400,
                "concurrency": 50
            },
            "context": {
                "users": {
                    "tenants": 2,
                    "users_per_tenant": 3
                }
            },
            "sla": {
                "failure_rate": { "max": 0 }
            }
        }
    ]
}

5.3.1.9. Test case 3: Keystone authentication¶

5.3.1.9.1. Description¶

To cover Keystone control plane operations simple authenticate scenario can be used.

5.3.1.9.2. Parameters¶

Name	Description
RPS	Generated load
ITERATIONS	Total amount of iterations processed by all executors

5.3.1.9.3. Measuring performance values¶

Authenticate in Keystone through Keystone API.

This step executed in parallel on multiple executors to generate RPS load. Execution of this step is called as iteration. ITERATIONS is a total amount of iterations which was processed by executors.

At the end of this test case you should calculate average, 90% percentile, 50% percentile, minimum and maximum for each step. You need to fill the following tables with calculated values:

5.3.1.10. Keystone¶

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
authenticate

5.3.1.10.1. List of performance metrics¶

Priority	Value	Measurement Units	Description
1		sec	Time of atomic operations

5.3.1.10.2. Example of Rally scenario configuration¶

{
    "Authenticate.keystone": [
        {
            "runner": {
                "type": "rps",
                "times": 120000,
                "rps": 180
            },
            "context": {
                "users": {
                    "tenants": 10,
                    "users_per_tenant": 10
                }
            },
            "sla": {
                "failure_rate": { "max": 0 }
            }
        }
    ]
}

5.3.2. Reports¶

Test plan execution reports:

5.3. OpenStack control plane performance test plan

5.3. OpenStack control plane performance test plan¶

5.3.1. Test Plan¶

5.3.1.1. Test Environment¶

5.3.1.1.1. Preparation¶

5.3.1.1.2. Environment description¶

5.3.1.1.2.1. Hardware¶

5.3.1.1.2.2. Network¶

5.3.1.1.2.3. Software¶

5.3.1.2. Test tool¶

5.3.1.3. Test Case 1: Boot, attach, migrate and delete server with security groups¶

5.3.1.3.1. Description¶

5.3.1.3.2. Parameters¶

5.3.1.3.3. List of performance metrics¶

5.3.1.3.4. Measuring performance values¶

5.3.1.4. Cinder¶

5.3.1.5. Neutron¶

5.3.1.6. Nova¶

5.3.1.6.1. Example of Rally scenario configuration¶

5.3.1.7. Test Case 2: Create and delete image¶

5.3.1.7.1. Description¶

5.3.1.7.2. Parameters¶

5.3.1.7.3. List of performance metrics¶

5.3.1.7.4. Measuring performance values¶

5.3.1.8. Glance¶

5.3.1.8.1. Example of Rally scenario configuration¶

5.3.1.9. Test case 3: Keystone authentication¶

5.3.1.9.1. Description¶

5.3.1.9.2. Parameters¶

5.3.1.9.3. Measuring performance values¶

5.3.1.10. Keystone¶

5.3.1.10.1. List of performance metrics¶

5.3.1.10.2. Example of Rally scenario configuration¶

5.3.2. Reports¶

performance_docs 0.1

Page Contents