5.12. Kubernetes POD recovery test plan

status:ready
version:1.0
Abstract:This test plan covers scenarios for Kuberneter POD recovery testing.

5.12.1. Test Plan

In Kubernetes “classic” HTTP-based services can be represented with help of replication controller and service. Replication controller is responsible for keeping number of PODs constant and in restarts POD in case of failure. If the POD is deleted, then replication controller creates a new one. Depending on scheduler settings a new instance of POD can be scheduled to a different node.

Under service performance we mean the amount of work produced by a service, for HTTP service this can be measured as number of requests per second.

5.12.1.1. Test Environment

5.12.1.1.1. Preparation

The test plan is executed against Kubernetes deployed on bare-metal nodes.

5.12.1.1.2. Environment description

The environment description includes hardware specification of servers, network parameters, operation system and OpenStack deployment characteristics.

5.12.1.2. Test Case #1: POD restart

5.12.1.2.1. Description

In this test case we investigate how POD rescheduling affects service performance.

Steps:
  • Build Docker image and push it into local registry: docker build -t localhost:31500/qa/svc code/ ``docker push localhost:31500/qa/svc
  • Create Kubernetes Service and Deployments based on files code/svc-dpl.yaml and code/svc-svc.yaml
  • Measure service performance during 20 seconds: wrk -c 20 -d 10 -t 20 -H "Connection: close" --latency --timeout 10s http://svc.ccp:8000/
  • While measurement is running delete the pod via kubectl delete pod -l app=svc

5.12.1.2.2. List of performance metrics

list of test metrics to be collected during this test
Parameter Description
RPS_BASE Base RPS value
RPS_WHILE_RESCHEDULING RPS measured during rescheduling

5.12.2. Reports

Test plan execution reports: