2020-01-29 - Albert Braden¶
Here are the scaling issues I’ve encountered recently at Synopsys, in reverse chronological order:
Thursday 12/19/2019: openstack server list –all-projects does not return all VMs.¶
In /etc/nova/nova.conf we have default: # max_limit = 1000
The recordset cleanup script depends on correct output from “openstack server list –all-projects”
Fix: Increased max_limit to 2000
The recordset cleanup script will run “openstack server list –all-projects|wc –l” and compare the output to max_limit, and refuse to run if max_limit is too low. If this happens, increase max_limit so that it is greater than the number of VMs in the cluster.
As time permits we need to look into paging results: https://docs.openstack.org/api-guide/compute/paginated_collections.html
Friday 12/13/2019: Arp table got full on pod2 controllers¶
https://www.cyberciti.biz/faq/centos-redhat-debian-linux-neighbor-table-overflow/
Fix: Increase sysctl values:
--- a/roles/openstack/controller/neutron/tasks/main.yml
+++ b/roles/openstack/controller/neutron/tasks/main.yml
@@ -243,6 +243,9 @@
with_items:
- { name: 'net.bridge.bridge-nf-call-iptables', value: '1' }
- { name: 'net.bridge.bridge-nf-call-ip6tables', value: '1' }
+ - { name: 'net.ipv4.neigh.default.gc_thresh3', value: '4096' }
+ - { name: 'net.ipv4.neigh.default.gc_thresh2', value: '2048' }
+ - { name: 'net.ipv4.neigh.default.gc_thresh1', value: '1024' }
12/10/2019: RPC workers were overloaded¶
http://lists.openstack.org/pipermail/openstack-discuss/2019-December/011465.html
Fix: increase number of RPC workers. modify /etc/neutron/neutron.conf on controllers:
148c148
< #rpc_workers = 1
---
> rpc_workers = 8
October 2019: Rootwrap¶
Neutron was timing out because rootwrap was taking too long to spawn.
Fix: Run rootwrap daemon:
Add line to /etc/neutron/neutron.conf on the controllers:
root_helper_daemon = “sudo /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf”
Add line to /etc/sudoers.d/neutron_sudoers on the controllers:
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf