6.18.1. Neutron L3 HA test results Liberty

This report is generated for OpenStack Neutron L3 HA Test Plan test plan.

6.18.1.1. Environment description

6.18.1.1.1. Cluster description

  • 3 controllers

  • 46 compute nodes

6.18.1.1.2. Software versions

MOS 8.0

6.18.1.1.3. Hardware configuration of each server

Description of servers hardware

Compute Vendor:

1x SUPERMICRO SUPERSERVER 5037MR-H8TRF MICRO-CLOUD http://www.supermicro.com/products/system/3u/5037/sys-5037mr-h8trf.cfm

CPU

1x INTEL XEON Ivy Bridge 6C E5-2620 V2 2.1G 15M 7.2GT/s QPI 80w SOCKET 2011R 1600 http://ark.intel.com/products/75789/Intel-Xeon-Processor-E5-2620-v2-15M-Cache-2_10-GHz

RAM:

4x Samsung DDRIII 8GB DDR3-1866 1Rx4 ECC REG RoHS M393B1G70QH0-CMA

NIC

1x AOC-STGN-i2S - 2-port 10 Gigabit Ethernet SFP+

6.18.1.2. Rally test results

L3 HA has a restriction of 255 routers per HA network per tenant. At this moment we do not have the ability to create new HA network per tenant if the number of VIPs exceed this limit. Based on this, for some tests, the number of tenants was increased (NeutronNetworks.create_and_list_router).

The most important results are provided by test_create_delete_routers test, as it allows to catch possible race conditions during creation/deletion of HA routers, HA networks and HA interfaces. There are already several known bugs related to this which have been fixed in upstream. To find out more possible issues test_create_delete_routers has been run multiple times with different concurrency.

Results of test_create_delete_routers

Times

Concurrency

Number of errors

Link for rally report

92

20

0

rally report

92

40

0

rally report

150

50

1

rally report

150

50

0

rally report

200

60

1

rally report

200

60

1

rally report

200

70

2

rally report

200

70

0

rally report

200

75

1

rally report

200

75

1

rally report

300

100

1

rally report

300

100

0

rally report

400

100

1

rally report

400

100

0

rally report

Multiple scenarios:

Test

Number of tenants

Times

Concurrency

Number of errors

Link for rally report

create_and_delete_routers

1

92

10

0

rally report

create_and_list_routers

2

368

10

272

create_and_update_routers

1

92

10

0

create_and_delete_routers

1

92

10

0

rally report

create_and_list_routers

2

100

10

6

create_and_update_routers

1

92

10

0

create_and_delete_routers

1

92

10

0

rally report

create_and_list_routers

10

368

10

0

create_and_update_routers

1

92

10

0

create_and_delete_routers

1

300

50

1

rally report

create_and_list_routers

10

368

50

0

create_and_update_routers

1

300

50

0

create_and_delete_routers

1

300

50

1

rally report

create_and_list_routers

10

368

50

0

create_and_update_routers

1

300

50

0

The errors discovered have been classified as the following bugs:

Bugs

Short description

Trace

Upstream bug

Status

IpAddressGenerationFailure No more IP addresses available on network

trace

bug/1562887

Open (Affects Neutron without L3 HA enabled, probably Rally bug)

Device “tap-<id>” does not exist.

trace

bug/1562887

Open

Session rollback

trace

bug/1550886

In progress

SubnetInUse: Unable to complete operation on subnet

trace

bug/1562878

Open

MessagingTimeout: Timed out waiting for a reply to message

trace

bug/1555670

Open

DBDeadlock: ipallocationpools

trace

bug/1562876

Open

Not all HA networks deleted

not a trace

bug/1562892

Open

6.18.1.2.1. Summary:

  1. The number of failed tests is less than 1% (exception test_create_list_routers, but with increased number of tenants the problem was fixed; automatic creation of new HA network after the previous one ran out of virtual ips is more like a feature request).

  2. All bugs found are Medium or Low priority.

6.18.1.3. Shaker test results

L3 HA

L3 HA during L3 agents restart

Router rescheduling (Non L3 HA) during L3 agent restart

Lost

Errors

Link for report

Lost

Errors

Link for report

Lost

Errors

Link for report

OpenStack L3 East-West

0

0

report

0

0

report

50

5

report

OpenStack L3 East-West Performance

1

0

report

0

0

report

0

1 (all)

report

OpenStack L3 North-South

0

0

report

8

0

report

95

3

report

OpenStack L3 North-South UDP

10

1

report

14

0

report

OpenStack L3 North-South Performance

(concurrency 2)

0

0

report

0

0

report

OpenStack L3 North-South Performance

(concurrency 5)

0

0

report

1

0

report

OpenStack L3 North-South Dense

0

0

report

41

0

report

81

1

report

Shaker provides statistics about maximum, minimum and mean values of different connection measurements. For each test was found the maximum among all maximum values, minimum among all minimum values and counts the mean value from all mean values. In the table below, these values are presented.

type

L3 HA

L3 HA during l3 agents restart

Router rescheduling (Non L3 HA) during l3 agent restart

min

mean

max

min

mean

max

min

mean

max

OpenStack L3 East-West

ping_icmp,

ms

0.05

2.45

12.39

0.07

7.39

18.03

0.41

32.84

2583.93

tcp_download

Mbits/s

0.02

874.04

5820.88

0.11

957.66

5883.96

77.41

896.96

3703.83

tcp_upload

Mbits/s

0.02

884.25

5649.94

0.13

897.11

5963.02

64.11

1268.74

5111.02

OpenStack L3 East-West Performance

ping_icmp ms

0.64

0.81

1.45

0.57

0.82

1.79

No statistic

Bandwidth Mbit/s

839.84

1876.83

3880.01

630.0

1497.19

3020.0

Packets pps

101680.0

129664.2

136880.0

89660.0

129515.33

367930.0

retransmits

0.0

0.67

25.0

0.0

2.5

72.0

OpenStack L3 North-South

ping_icmp,

ms

0.08

9.83

27.61

0.06

7.11

25.73

0.33

0.62

2.45

tcp_download

Mbits/s

65.28

902.35

4454.43

72.7

769.61

4494.97

741.95

1647.07

2776.53

tcp_upload

Mbits/s

0.13

815.02

4345.86

0.13

867.68

4289.98

No statistic

OpenStack L3 North-South UDP

Packets pps

31218.0

123452.06

476254.0

39196.0

122214.76

431108.0

OpenStack L3 North-South Performance

(concurrency 2)

ping_icmp ms

0.9

1.22

2.36

0.67

0.93

2.34

Bandwidth Mbit/s

439.91

449.94

525.5

0.0

2000.8

3400.5

Packets pps

126360.0

129349.33

135150.0

131700.0

135319.33

140550.0

retransmits

0.0

1.0

83.0

0.0

3.0

205.0

OpenStack L3 North-South Performance

(concurrency 5)

ping_icmp ms

0.74

0.97

1.72

0.2

1.02

3.01

Bandwidth Mbit/s

41.99

181.01

386.43

0.0

1720.71

3519.77

Packets pps

122140.0

131601.17

138220.0

103510.0

129021.6

138860.0

retransmits

0.0

1.0

49.0

0.0

3.17

231.0

OpenStack L3 North-South Dense

ping_icmp,

ms

0.56

18.18

96.42

0.38

4.07

56.35

0.45

9.79

106.52

tcp_download

Mbits/s

1.72

210.2

862.02

322.24

1634.48

4656.44

11.61

407.69

2235.84

tcp_upload

Mbits/s

18.88

209.49

781.86

49.96

1590.83

4667.82

18.77

1955.41

4333.32

These results show that there is no significant difference between results during multiple l3 agent restarts and normal test execution.

Average value of difference between these values without and with restart presented in the next table:

ping_icmp,

ms

tcp_download

Mbits/s

tcp_upload

Mbits/s

Bandwidth Mbit/s

Packets pps

retransmits

min

0.17

-103.34

-10.39

230.58

4333

0

mean

2.02

-458.39

-482.39

-903.64

-501.07

-2

max

5.78

-1299.35

-1381.05

-1717.11

-47986

-117

6.18.1.3.1. Summary:

  1. Results of comparison between L3 HA and standard router rescheduling show that L3 HA allows to perform testing uninterrupted without huge loss of statistics during L3 agent restarts.

  2. Comparing L3 HA results with and without restart show that bandwidth and speed do not decrease during agent restart.

6.18.1.4. Manual tests execution

During manual testing, the following scenarios were tested:

  • Ping to external network from VM during reset of primary(non-primary)

    controller

  • Ping from one VM to another VM in different network during ban L3

    agent

  • Iperf UPD testing between VMs in different networks during ban L3

    agent

All tests were performed with large number of routers.

6.18.1.4.1. Ping to external network from VM during reset of primary(non-primary) controller

../../../_images/ping_external1.png

Iteration

Number of routers

Command

Number of loss packages

1

1

3

2

25

3

3

50

3

4

100

3

5

150

3

6

170

ping 8.8.8.8

3

7

175

89

8

175

116

9

175

52

10

200

51

11

200

3

Current result looks unstable and not directly dependent on the number of routers. The huge loss of packages on iterations 7-10 happened because agent from recovered controller became “active” (master) while there was already another active L3 agent. After some time it became the only “active” L3 agent for router.

This issue needs special attention and will be investigated as bug/1563298.

6.18.1.4.2. Ping from one VM to another VM in different network during ban L3 agent

../../../_images/ping1.png

Iteration

Number of routers

Command

Number of loss packages

1

100

4

2

4

3

3

4

200

3

5

3

6

ping 10.0.1.6

103

7

26

8

3

9

250

3

10

4

The loss of packages on iterations 6-7 happened for the similar reason as for previous manual scenario. L3 agent status flapped during loss.

With 250 routers l3 agents started to fail with unmanaged state.

6.18.1.4.3. Iperf UPD testing between VMs in different networks ban L3 agent

../../../_images/iperf_addresses1.png

Number of routers

Command

Loss (%)

10

0.14

4.9

1.3

5.3

24

1.3

iperf -c 10.0.3.4 -p 5001 -t 60 -i 10 –bandwidth 30M –len 64 -u

8.9

6.1

2.4

50

1.7

10

40

18

6.18.1.4.4. Summary:

  1. For unstable behaviour of L3 HA, bug was filed.

  2. With number of routers less than 170, the network can be classified as stable for failures.

  3. With number of routers more than 240, agent’s recovery leads to falling into unmanaged state.