5.6. Fuel Containerized Control Plane performance testing

status

ready

version

1.0

Abstract

This test plan aims to provide set of tests to identify OpenStack performance against given containerized OpenStack cloud (installed on the top of pre-deployed Kubernetes cluster) using simple minimalistic set of Rally tests.

5.6.1. Test Plan

This document is inspired by OpenStack control plane performance test plan, and aims to cover baseline cloud operations and extend this test suite to verify containerized deployment approach. As OpenStack control plane performance test plan this test plan covers basic cloud operations e.g. VMs creation, work with the security groups, authentication and more, as well as long-running test suites to verify cloud stability.

5.6.1.1. Test Environment

5.6.1.1.1. Preparation

This test plan is performed against existing OpenStack cloud installed on top of pre-deployed Kubernetes cluster with fuel-ccp tool with pre-installed Rally framework.

5.6.1.1.2. Environment description

The environment description includes hardware specification of servers, network parameters, operation system and OpenStack deployment characteristics.

5.6.1.1.2.1. Hardware

This section contains list of all types of hardware nodes (table below is an example).

Parameter	Value	Comments
model		e.g. Supermicro X9SRD-F
CPU		e.g. 6 x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz
role		e.g. compute or network

5.6.1.1.2.2. Network

This section contains list of interfaces and network parameters. For complicated cases this section may include topology diagram and switch parameters (table below is an example).

Parameter	Value	Comments
network role		e.g. provider or public
card model		e.g. Intel
driver		e.g. ixgbe
speed		e.g. 10G or 1G
MTU		e.g. 9000
offloading modes		e.g. default

5.6.1.1.2.3. Software

This section describes installed software (table below is an example).

Parameter	Value	Comments
OS		e.g. Ubuntu 14.04.3
OpenStack		e.g. Mitaka
Hypervisor		e.g. KVM
Neutron plugin		e.g. ML2 + OVS
L2 segmentation		e.g. VLAN / VxLAN / GRE
virtual routers		e.g. HA / DVR

5.6.1.2. Test tool

Rally is a benchmarking tool that was designed specifically for OpenStack API testing. To make this possible, Rally automates and unifies multi-node OpenStack deployment, cloud verification, benchmarking & profiling. This is a simple way to check cloud workability and performance of control plane operations running on it.

5.6.1.3. Test Case 1: Boot and delete server

5.6.1.3.1. Description

The most user-facing control plane operation is new virtual machine creation. This scenario covers the most basic OpenStack server creation to present the baseline numbers for Nova (OpenStack Compute) control plane.

5.6.1.3.2. Parameters

Name	Description
IMAGE	Image from which boot server
FLAVOR	Flavor type from which boot server
ASSIGN_NIC	Bool, whether or not to auto assign NICs in Rally scenario
CONCURRENCY	Amount of parallel executors
ITERATIONS	Total amount of iterations processed by all executors

5.6.1.3.3. List of performance metrics

Priority	Value	Measurement Units	Description
1		sec	Time of atomic operations

5.6.1.3.4. Measuring performance values

1. Create server with FLAVOR flavor from IMAGE image through Nova API

2. Delete server through Nova API.

These 2 steps executed successively in CONCURRENCY parallel executors. ASSIGN_NIC parameter reflects Rally scenario configuration whether to assign NIC to the booted server in automatic fashion.

One cycle of these 2 steps is called an iteration. ITERATIONS is a total amount of iterations which was processed by executors.

At the end of this test case you should calculate average, 90% percentile, 50% percentile, minimum and maximum for each step. You need to fill the following tables with calculated values:

5.6.1.4. Nova

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_server
delete_server

5.6.1.4.1. Example of Rally scenario configuration

---
  NovaServers.boot_and_delete_server:
    -
      args:
        flavor:
            name: "^scaletest$"
        image:
            name: "^cirros$"
        force_delete: false
      runner:
        type: "constant"
        times: {{ 5 * computes_count }}
        concurrency: 5
      context:
        users:
          tenants: 10
          users_per_tenant: 10
        quotas:
          nova:
            instances: -1
            cores: -1
            ram: -1
            floating_ips: -1
            security_groups: -1
            security_group_rules: -1
          neutron:
            network: -1
            security_group: -1
            security_group_rule: -1
            port: -1
            subnet: -1
            router: -1
      sla:
        failure_rate:
          max: 0

5.6.1.5. Test Case 2: Boot and delete server with security groups

5.6.1.5.1. Description

The most user-facing control plane operation is new virtual machine creation. At the same time security groups management is very time consuming operation in case of lots VMs attached to the same security group, therefore it’s vital to understand these operations performance. Standard Rally scanario is used for this purpose (comparing with OpenStack control plane performance test plan fuel-ccp does not support yet Cinder installation or appropriate Nova configuration to support live migrations, so there is no need to write separated plugin to cover these operations).

5.6.1.5.2. Parameters

Name	Description
IMAGE	Image from which boot server
FLAVOR	Flavor type from which boot server
SEC_GROUP_COUNT	Count of security groups to be created in one iteration
RULES_PER_SECURITY_GROUP	Count of rules to be added to each security group
CONCURRENCY	Amount of parallel executors
ITERATIONS	Total amount of iterations processed by all executors

5.6.1.5.3. List of performance metrics

Priority	Value	Measurement Units	Description
1		sec	Time of atomic operations

5.6.1.5.4. Measuring performance values

1. Successively create SEC_GROUP_COUNT security groups through Nova API. Duration of this step represent time that control plane process create_SEC_GROUP_COUNT_security_groups atomic operation.

2. Successively create RULES_PER_SECURITY_GROUP rules for security groups through Nova API.

3. Create server with FLAVOR flavor from IMAGE image through Nova API

4. List all security groups through Nova API.

5. Delete server through Nova API.

6. Successively delete SEC_GROUP_COUNT security group through Nova API.

These 6 steps executed successively in CONCURRENCY parallel executors. One cycle of these 6 steps is called an iteration.

ITERATIONS is a total amount of iterations which was processed by executors.

At the end of this test case you should calculate average, 90% percentile, 50% percentile, minimum and maximum for each step. You need to fill the following tables with calculated values:

5.6.1.6. Neutron

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_N_security_groups
delete_N_security_groups
create_M_rules
delete_M_rules

Note

Change operation name to appropriate regarding SEC_GROUP_COUNT and RULES_PER_SECURITY_GROUP values.

5.6.1.7. Nova

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_server
delete_server

5.6.1.7.1. Example of Rally scenario configuration

---
  NovaSecGroup.boot_and_delete_server_with_secgroups:
    -
      args:
        flavor:
            name: "^scaletest$"
        image:
            name: "^cirros$"
        security_group_count: 2
        rules_per_security_group: 10
      runner:
        type: "constant"
        times: {{ 5 * computes_count }}
        concurrency: 5
      context:
        users:
          # it's important to have only 1 tenant to have single network to load
          tenants: 1
          users_per_tenant: 10
        network:
          start_cidr: "{{ network_cidr }}"
          networks_per_tenant: 1
        quotas:
          nova:
            instances: -1
            cores: -1
            ram: -1
            floating_ips: -1
            security_groups: -1
            security_group_rules: -1
          neutron:
            network: -1
            security_group: -1
            security_group_rule: -1
            port: -1
            subnet: -1
            router: -1
      sla:
        failure_rate:
          max: 0

5.6.1.8. Test Case 3: Boot and list servers

5.6.1.8.1. Description

This scenario covers density aspect of server creation control plane operation and checks how many virtual machines can be booted on top of containerized OpenStack.

5.6.1.8.2. Parameters

Name	Description
IMAGE	Image from which boot server
FLAVOR	Flavor type from which boot server
CONCURRENCY	Amount of parallel executors
ITERATIONS	Total amount of iterations processed by all executors

5.6.1.8.3. List of performance metrics

Priority	Value	Measurement Units	Description
1		sec	Time of atomic operations

5.6.1.8.4. Measuring performance values

1. Create server with FLAVOR flavor from IMAGE image through Nova API

2. List all existing servers through Nova API.

These 2 steps executed successively in CONCURRENCY parallel executors.

One cycle of these 2 steps is called an iteration. ITERATIONS is a total amount of iterations which was processed by executors.

At the end of this test case you should calculate average, 90% percentile, 50% percentile, minimum and maximum for each step. You need to fill the following tables with calculated values:

5.6.1.9. Nova

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_server
list_servers

5.6.1.9.1. Example of Rally scenario configuration

---
  NovaServers.boot_and_list_server:
    -
      args:
        flavor:
            name: "^scaletest$"
        image:
            name: "^cirros$"
        detailed: True
      runner:
        type: "constant"
        times: {{ 5 * computes_count }}
        concurrency: 5
      context:
        users:
          tenants: 10
          users_per_tenant: 10
        quotas:
          nova:
            instances: -1
            cores: -1
            ram: -1
            floating_ips: -1
            security_groups: -1
            security_group_rules: -1
          neutron:
            network: -1
            security_group: -1
            security_group_rule: -1
            port: -1
            subnet: -1
            router: -1
      sla:
        failure_rate:
          max: 0

5.6.1.10. Test Case 4: Create and delete image

5.6.1.10.1. Description

To cover Glance control plane operations simple create and delete image scenario can be used.

5.6.1.10.2. Parameters

Name	Description
IMAGE	Image to upload to glance
CONTAINER_FORMAT	Container format to create
DISK_FORMAT	Disk format to create
CONCURRENCY	Amount of parallel executors
ITERATIONS	Total amount of iterations processed by all executors

5.6.1.10.3. List of performance metrics

Priority	Value	Measurement Units	Description
1		sec	Time of atomic operations

5.6.1.10.4. Measuring performance values

1. Create image from IMAGE with CONTAINER_FORMAT container format and DISK_FORMAT disk format through Glance API.

2. Delete image from Glance through Glance API.

These 2 steps executed successively in CONCURRENCY parallel executors. One cycle of thee 2 steps is called an iteration.

ITERATIONS is a total amount of iterations which was processed by executors.

At the end of this test case you should calculate average, 90% percentile, 50% percentile, minimum and maximum for each step. You need to fill the following tables with calculated values:

5.6.1.11. Glance

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_image
delete_image

5.6.1.11.1. Example of Rally scenario configuration

---
  GlanceImages.create_and_delete_image:
    -
      args:
        image_location: "{{ image_server_url }}/1g.img"
        container_format: "bare"
        disk_format: "qcow2"
      runner:
        type: "constant"
        times: 400
        concurrency: 50
      context:
        users:
          tenants: 2
          users_per_tenant: 3
      sla:
        failure_rate:
          max: 0

5.6.1.12. Test Case 5: Create and list images

5.6.1.12.1. Description

To cover Glance control plane operations simple create and list images scenario can be used.

5.6.1.12.2. Parameters

Name	Description
IMAGE	Image to upload to glance
CONTAINER_FORMAT	Container format to create
DISK_FORMAT	Disk format to create
CONCURRENCY	Amount of parallel executors
ITERATIONS	Total amount of iterations processed by all executors

5.6.1.12.3. List of performance metrics

Priority	Value	Measurement Units	Description
1		sec	Time of atomic operations

5.6.1.12.4. Measuring performance values

1. Create image from IMAGE with CONTAINER_FORMAT container format and DISK_FORMAT disk format through Glance API.

2. List existing images through Glance API.

These 2 steps executed successively in CONCURRENCY parallel executors. One cycle of thee 2 steps is called an iteration.

ITERATIONS is a total amount of iterations which was processed by executors.

At the end of this test case you should calculate average, 90% percentile, 50% percentile, minimum and maximum for each step. You need to fill the following tables with calculated values:

5.6.1.13. Glance

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
create_image
list_images

5.6.1.13.1. Example of Rally scenario configuration

---
  GlanceImages.create_and_list_image:
    -
      args:
        image_location: "{{ image_server_url }}/cirros.img"
        container_format: "bare"
        disk_format: "qcow2"
      runner:
        type: "constant"
        times: 200
        concurrency: 50
      context:
        users:
          tenants: 2
          users_per_tenant: 3
      sla:
        failure_rate:
          max: 0

5.6.1.14. Test case 6: Keystone authentication

5.6.1.14.1. Description

To cover Keystone control plane operations simple authenticate scenario can be used.

5.6.1.14.2. Parameters

Name	Description
RPS	Generated load
ITERATIONS	Total amount of iterations processed by all executors

5.6.1.14.3. Measuring performance values

1. Authenticate in Keystone through Keystone API.

This step is executed in parallel on multiple executors to generate RPS load.

Execution of this step is called an iteration.

ITERATIONS is a total amount of iterations which was processed by executors.

At the end of this test case you should calculate average, 90% percentile, 50% percentile, minimum and maximum for each step. You need to fill the following tables with calculated values:

5.6.1.15. Keystone

Operation	Mean (sec)	90%ile (sec)	50%ile (sec)	Max (sec)	Min (sec)
authenticate

5.6.1.15.1. List of performance metrics

Priority	Value	Measurement Units	Description
1		sec	Time of atomic operations

5.6.1.15.2. Example of Rally scenario configuration

---
  Authenticate.keystone:
    -
      runner:
        type: "rps"
        times: 120000
        rps: 90
      context:
        users:
          tenants: 10
          users_per_tenant: 10
      sla:
        failure_rate:
          max: 0

5.6.2. Reports

Test plan execution reports:

• Fuel Containerized Control Plane performance report