2. Measuring performance of container cluster systems

status:ready version:1.0 Abstract:A complete microservices architecture has the ability to manage a large number of small applications which together form a set of services. Container technology provides a way to package and run applications independent of the underlying operating system. Microservice and container technologies are very complimentary and usually used in conjunction. There are several container management systems, such as Apache Mesos, Kubernetes, and Docker Swarm, which provide various ways to manage groups of containers.

2.1. Test Plan

2.1.1. Test Environment

2.1.1.1. Preparation

Typical container frameworks consist of several components:

• Configuration and Synchronization (Zookeeper)
• Scheduler (Mesos)
• Application manager (Marathon)

2.1.1.2. Environment description

In this document only performance and scaling aspects of the different frameworks will be covered. It is expected that existing solutions will be used and that they have had sufficient functional testing. Performance aspects should cover elapsed times for operations used in the framework.

2.1.2. Test Case #1: A new application creation time

2.1.2.1. Description

This test is aimed at measuring the total elapsed time of the create application operation. This metric includes the time of application submission processing, time of scheduling, and time of actual all containers spawning on the slaves hosts.

2.1.2.2. List of performance metrics

list of test metrics to be collected during this test

Parameter	Description
APPLICATION_OPERATION_MAX	Maximum execution time of one operation
APPLICATION_OPERATION_MIN	Minimum execution time of one operation
APPLICATION_OPERATION_MEAN	Mean execution time of all operations
APPLICATION_OPERATION_MEDIAN	Median execution time of all operations

list of test metrics to be persistent during this test:

Parameter	Value
NODES_NUMBER_PER_APPLICATION	50, 100, 500
CONCURRENCY	1, 2, 4, 8, 16

2.1.2.3. Measuring values

1. Start to create simultaneously the number of applications equal CONCURRENCY with number of instances equal NODES_NUMBER_PER_APPLICATION in the each application through Marathon API. Every instances is docker container with nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
2. Wait util all application go to state Running
3. As a result of the previous step you should be able to provide the amount of graphs and tables with the dependencies on an concurrency and instances number of a operation time. One table for all results and graphs for every concurrency and every nodes number. On this step you need to calculate minimum, maxima, average and median of operation time. You need to fill the following table with calculated values:

results

CONCURRENCY	NODES_NUMBER_PER_APPLICATION	APPLICATION_OPERATION (sec)
		minimum	maximum	average	median
1	50
1	100
1	500
2	50
2	100
2	500
4	50
4	100
4	500
8	50
8	100
8	500
16	50
16	100
16	500

2.1.3. Test Case #2: A application CPU update time

2.1.3.1. Description

This test is aimed at measuring the total elapsed time of the update CPU application operation. This metric includes the time of application submission processing, time of scheduling, and time of actual all containers updated on the slaves hosts.

2.1.3.2. List of performance metrics

list of test metrics to be collected during this test

Parameter	Description
APPLICATION_OPERATION_MAX	Maximum execution time of one operation
APPLICATION_OPERATION_MIN	Minimum execution time of one operation
APPLICATION_OPERATION_MEAN	Mean execution time of all operations
APPLICATION_OPERATION_MEDIAN	Median execution time of all operations

list of test metrics to be persistent during this test:

Parameter	Value
NODES_NUMBER_PER_APPLICATION	50, 100, 500
CONCURRENCY	1, 2, 4, 8, 16

2.1.3.3. Measuring values

1. Start to create simultaneously the number of applications equal CONCURRENCY with number of instances equal NODES_NUMBER_PER_APPLICATION in the each application through Marathon API. Every instances is docker container with nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
2. Wait util all application go to state Running
3. Update applications CPU up to 2.
4. Wait util all application go to state Running
5. As a result of the previous step you should be able to provide the amount of graphs and tables with the dependencies on an concurrency and instances number of a operation time. One table for all results and graphs for every concurrency and every nodes number. On this step you need to calculate minimum, maxima, average and median of operation time. You need to fill the following table with calculated values:

results

CONCURRENCY	NODES_NUMBER_PER_APPLICATION	APPLICATION_OPERATION (sec)
		minimum	maximum	average	median
1	50
1	100
1	500
2	50
2	100
2	500
4	50
4	100
4	500
8	50
8	100
8	500
16	50
16	100
16	500

2.1.4. Test Case #3: A application memory update time

2.1.4.1. Description

This test is aimed at measuring the total elapsed time of the update memory application operation. This metric includes the time of application submission processing, time of scheduling, and time of actual all containers updated on the slaves hosts.

2.1.4.2. List of performance metrics

list of test metrics to be collected during this test

Parameter	Description
APPLICATION_OPERATION_MAX	Maximum execution time of one operation
APPLICATION_OPERATION_MIN	Minimum execution time of one operation
APPLICATION_OPERATION_MEAN	Mean execution time of all operations
APPLICATION_OPERATION_MEDIAN	Median execution time of all operations

list of test metrics to be persistent during this test:

Parameter	Value
NODES_NUMBER_PER_APPLICATION	50, 100, 500
CONCURRENCY	1, 2, 4, 8, 16

2.1.4.3. Measuring values

1. Start to create simultaneously the number of applications equal CONCURRENCY with number of instances equal NODES_NUMBER_PER_APPLICATION in the each application through Marathon API. Every instances is docker container with nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
2. Wait util all application go to state Running
3. Update applications memory up to 512M.
4. Wait util all application go to state Running
5. As a result of the previous step you should be able to provide the amount of graphs and tables with the dependencies on an concurrency and instances number of a operation time. One table for all results and graphs for every concurrency and every nodes number. On this step you need to calculate minimum, maxima, average and median of operation time. You need to fill the following table with calculated values:

results

CONCURRENCY	NODES_NUMBER_PER_APPLICATION	APPLICATION_OPERATION (sec)
		minimum	maximum	average	median
1	50
1	100
1	500
2	50
2	100
2	500
4	50
4	100
4	500
8	50
8	100
8	500
16	50
16	100
16	500

2.1.5. Test Case #4: A application disk update time

2.1.5.1. Description

This test is aimed at measuring the total elapsed time of the update disk application operation. This metric includes the time of application submission processing, time of scheduling, and time of actual all containers updated on the slaves hosts.

2.1.5.2. List of performance metrics

list of test metrics to be collected during this test

Parameter	Description
APPLICATION_OPERATION_MAX	Maximum execution time of one operation
APPLICATION_OPERATION_MIN	Minimum execution time of one operation
APPLICATION_OPERATION_MEAN	Mean execution time of all operations
APPLICATION_OPERATION_MEDIAN	Median execution time of all operations

list of test metrics to be persistent during this test:

Parameter	Value
NODES_NUMBER_PER_APPLICATION	50, 100, 500
CONCURRENCY	1, 2, 4, 8, 16

2.1.5.3. Measuring values

1. Start to create simultaneously the number of applications equal CONCURRENCY with number of instances equal NODES_NUMBER_PER_APPLICATION in the each application through Marathon API. Every instances is docker container with nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
2. Wait util all application go to state Running
3. Update applications disk up to 100G.
4. Wait util all application go to state Running
5. As a result of the previous step you should be able to provide the amount of graphs and tables with the dependencies on an concurrency and instances number of a operation time. One table for all results and graphs for every concurrency and every nodes number. On this step you need to calculate minimum, maxima, average and median of operation time. You need to fill the following table with calculated values:

results

CONCURRENCY	NODES_NUMBER_PER_APPLICATION	APPLICATION_OPERATION (sec)
		minimum	maximum	average	median
1	50
1	100
1	500
2	50
2	100
2	500
4	50
4	100
4	500
8	50
8	100
8	500
16	50
16	100
16	500

2.1.6. Test Case #5: A application instances update time

2.1.6.1. Description

This test is aimed at measuring the total elapsed time of the update number of instances application operation. This metric includes the time of application submission processing, time of scheduling, and time of actual all containers updated on the slaves hosts.

2.1.6.2. List of performance metrics

list of test metrics to be collected during this test

Parameter	Description
APPLICATION_OPERATION_MAX	Maximum execution time of one operation
APPLICATION_OPERATION_MIN	Minimum execution time of one operation
APPLICATION_OPERATION_MEAN	Mean execution time of all operations
APPLICATION_OPERATION_MEDIAN	Median execution time of all operations

list of test metrics to be persistent during this test:

Parameter	Value
NODES_NUMBER_PER_APPLICATION	50, 100, 500
CONCURRENCY	1, 2, 4, 8, 16

2.1.6.3. Measuring values

1. Start to create simultaneously the number of applications equal CONCURRENCY with number of instances equal NODES_NUMBER_PER_APPLICATION in the each application through Marathon API. Every instances is docker container with nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
2. Wait util all application go to state Running
3. Update instances up to twice in the each application.
4. Wait util all application go to state Running
5. As a result of the previous step you should be able to provide the amount of graphs and tables with the dependencies on an concurrency and instances number of a operation time. One table for all results and graphs for every concurrency and every nodes number. On this step you need to calculate minimum, maxima, average and median of operation time. You need to fill the following table with calculated values:

results

CONCURRENCY	NODES_NUMBER_PER_APPLICATION	APPLICATION_OPERATION (sec)
		minimum	maximum	average	median
1	50
1	100
1	500
2	50
2	100
2	500
4	50
4	100
4	500
8	50
8	100
8	500
16	50
16	100
16	500

2.1.7. Test Case #6: A application restart time

2.1.7.1. Description

This test is aimed at measuring the total elapsed time of the restart application operation. This metric includes the time of application submission processing, time of scheduling, and time of actual all containers restarted on the slaves hosts.

2.1.7.2. List of performance metrics

list of test metrics to be collected during this test

Parameter	Description
APPLICATION_OPERATION_MAX	Maximum execution time of one operation
APPLICATION_OPERATION_MIN	Minimum execution time of one operation
APPLICATION_OPERATION_MEAN	Mean execution time of all operations
APPLICATION_OPERATION_MEDIAN	Median execution time of all operations

list of test metrics to be persistent during this test:

Parameter	Value
NODES_NUMBER_PER_APPLICATION	50, 100, 500
CONCURRENCY	1, 2, 4, 8, 16

2.1.7.3. Measuring values

1. Start to create simultaneously the number of applications equal CONCURRENCY with number of instances equal NODES_NUMBER_PER_APPLICATION in the each application through Marathon API. Every instances is docker container with nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
2. Wait util all application go to state Running
3. Restart all applications.
4. Wait util all application go to state Running
5. As a result of the previous step you should be able to provide the amount of graphs and tables with the dependencies on an concurrency and instances number of a operation time. One table for all results and graphs for every concurrency and every nodes number. On this step you need to calculate minimum, maxima, average and median of operation time. You need to fill the following table with calculated values:

results

CONCURRENCY	NODES_NUMBER_PER_APPLICATION	APPLICATION_OPERATION (sec)
		minimum	maximum	average	median
1	50
1	100
1	500
2	50
2	100
2	500
4	50
4	100
4	500
8	50
8	100
8	500
16	50
16	100
16	500

2.1.8. Test Case #7: A application delete time

2.1.8.1. Description

This test is aimed at measuring the total elapsed time of the delete application operation. This metric includes the time of application submission processing, time of scheduling, and time of actual all containers deleted on the slaves hosts.

2.1.8.2. List of performance metrics

list of test metrics to be collected during this test

Parameter	Description
APPLICATION_OPERATION_MAX	Maximum execution time of one operation
APPLICATION_OPERATION_MIN	Minimum execution time of one operation
APPLICATION_OPERATION_MEAN	Mean execution time of all operations
APPLICATION_OPERATION_MEDIAN	Median execution time of all operations

list of test metrics to be persistent during this test:

Parameter	Value
NODES_NUMBER_PER_APPLICATION	50, 100, 500
CONCURRENCY	1, 2, 4, 8, 16

2.1.8.3. Measuring values

1. Start to create simultaneously the number of applications equal CONCURRENCY with number of instances equal NODES_NUMBER_PER_APPLICATION in the each application through Marathon API. Every instances is docker container with nginx from dockerhub. Every instances has 1 CPU, 256M RAM and 50G disk.
2. Wait util all application go to state Running
3. Delete all applications.
4. Wait util list of application return empty list.
5. As a result of the previous step you should be able to provide the amount of graphs and tables with the dependencies on an concurrency and instances number of a operation time. One table for all results and graphs for every concurrency and every nodes number. On this step you need to calculate minimum, maxima, average and median of operation time. You need to fill the following table with calculated values:

results

CONCURRENCY	NODES_NUMBER_PER_APPLICATION	APPLICATION_OPERATION (sec)
		minimum	maximum	average	median
1	50
1	100
1	500
2	50
2	100
2	500
4	50
4	100
4	500
8	50
8	100
8	500
16	50
16	100
16	500

2.1.9. Test Case #8: Health check Performance

2.1.9.1. Description

This test is aimed at validating the response time of a health check when many health checks are configured.

2.1.9.2. List of performance metrics

list of test metrics to be collected during this test

Parameter	Description
HEALTH_CHECK_INTERVAL_DEVIATION(CONTAINERS_COUNT)	Difference between real health check interval and configured value

list of test metrics to be persistent during this test:

Parameter	Value
HEALTH_CHECK_TEST_DURATION	20
HEALTH_CHECK_INTERVAL	30

2.1.9.3. Measuring REAL_INTERVAL(TASK_COUNT) values

1. Deploy Marathon cluster from scratch.
2. Run CONTAINERS_COUNT containers in container cluster system. Setup health check for those containers in container cluster system with health check interval is set to HEALTH_CHECK_INTERVAL seconds.
3. Gather HEALTH_CHECK_INTERVAL_DEVIATION from containers during HEALTH_CHECK_TEST_DURATION minutes.
4. You need to perform step 2 and step 3 per each CONTAINERS_COUNT value from the following list:
   • CONTAINERS_COUNT=2500
   • CONTAINERS_COUNT=5000
   • CONTAINERS_COUNT=7500
   • CONTAINERS_COUNT=10000
   • CONTAINERS_COUNT=12500
   • CONTAINERS_COUNT=15000
   • CONTAINERS_COUNT=17500
   • CONTAINERS_COUNT=20000

3. As a result of the previous step you should be able to provide table with the dependences on an health check interval deviation of a containers count. You need to calculate minimuml, maximal, average and 95% percental of HEALTH_CHECK_INTERVAL_DEVIATION parameter per each CONTAINERS_COUNT value. You need to fill the following table with calculated values:

   PUSH_TIME(CONCURRENCY)

   CONTAINERS_COUNT	HEALTH_CHECK_INTERVAL_DEVIATION
   	minimum	maximal	average	95%

2.2. Applications

2.2.1. list of container platforms

Name of container platform	Version
Apache Mesos
Marathon
Docker

2.3. Reports

Test plan execution reports:

• Results of measuring performance of Mesos Marathon