6.9. Fuel Containerized Control Plane performance report during upgrade

Abstract:This document includes Fuel CCP control plane performance test results for various environments (from 50 to 350 nodes). All tests have been performed regarding Fuel Containerized Control Plane upgradability performance testing

6.9.1. Environment description

For Kubernetes pre-deployment Kargo tool was used. More information about fuel-ccp and how it can be installed can be found in official documentation.

6.9.1.1. Hardware configuration of each server

All servers have configuration describing in table below

Description of servers hardware
server vendor,model HP,DL380 Gen9
CPU vendor,model Intel,E5-2680 v3
processor_count 2
core_count 12
frequency_MHz 2500
RAM vendor,model HP,752369-081
amount_MB 262144
NETWORK interface_name p1p1
vendor,model Intel,X710 Dual Port
bandwidth 10G
STORAGE dev_name /dev/sda
vendor,model
raid10 - HP P840
12 disks EH0600JEDHE
SSD/HDD HDD
size 3,6TB

6.9.1.2. Network configuration of each server

All servers have same network configuration:

Network Scheme of the environment

6.9.1.3. Fuel CCP configuration

CCP main configuration ccp.yaml

CCP Cluster configuration configs.yaml

CCP Cluster topology topology.yaml

CCP repositories configuration repos.yaml

CCP versions configuration for Mitaka versions-mitaka.yaml

CCP versions configuration for upgrade from Mitaka to Newton versions-newton.yaml

6.9.1.4. Rally configuration and scenario

Rally deployment rally_deployment.json

Rally scenario create_and_delete.yaml

6.9.1.5. Upgrade and tests procedure

  • Install fuel-ccp
  • Rename ccp.yaml to /root/.ccp.ymal
  • Copy all yaml files to /root/
  • Rename versions-mitaka.yaml to versions.yaml
  • Execute ccp deploy and wait ok status for all services
  • Create rally deployment with rally_deployment.json
  • Create role for rally openstack --insecure role create member
  • Start rally tests rally task start --task-args 'gre_enabled: true' ./create_and_delete.yaml
  • Rename versions-newton.yaml to versions.yaml
  • Execute ccp deploy and wait ok status for all services
  • Wait tests results from rally

6.9.2. Test results

6.9.2.1. Test Case 1: Boot and delete server

The following set of results is dedicated to investigate how Nova installed against Kubernetes cluster via fuel-ccp tool is behaving during update from Mitaka to Newton. Upgrade started on 100 iteration and finished on the 156 iteration.

6.9.2.1.1. 149 nodes OpenStack cluster, concurrency 5, 1530 iterations

NovaServers.boot_and_delete_server scenario in upgrade_newton_to_mitaka.html

Boot and delete servers Rally scenario (200 nodes)
Operation Median (sec) 90%ile (sec) 95%ile (sec) Max (sec) Min (sec)
boot_server 11.879 16.753 18.708 48.84 7.563
delete_server 2.593 4.754 4.898 6.829 2.313

The control plane downtime of keystone upgrades will be avoided in the feature versions of fuel-ccp