[ English | English (United Kingdom) | Deutsch | русский | español | Indonesia ]
HAProxy and Keepalived in LXC containers¶
There can be a usecase where you might want to run HAProxy and Keepalived inside LXC containers. For instance, running these services on bare metal assumes that a default route for hosts should be set towards a public network. This scenario might be un-preferable for some deployments, especially in cases where you do not have standalone Load-Balancing hosts, but they’re co-located with other infra services instead.
Inventory overrides¶
In order to tell dynamic_inventory to generate a set of containers
for haproxy, you need to create a file
/etc/openstack_deploy/env.d/haproxy.yml
with the following content:
container_skel:
haproxy_container:
properties:
is_metal: false
Defining host networking¶
In order to make a public network available, you need to ensure having a corresponsive bridge on your hosts to which HAProxy containers will be plugged in with one side of a veth pair. The bridge should also contain a VLAN interface providing «public» connectivity.
You can create a bridge manually or leverage our systemd_networkd role which is capable of configuring required networking on hosts.
For the example below, let’s name our bridge br-public-api
and public vlan
with ID 40
. In your user_variables.yml
define the following variables:
_systemd_networkd_generic_devices:
- NetDev:
Name: bond0
Kind: bond
Bond:
Mode: 802.3ad
TransmitHashPolicy: layer3+4
LACPTransmitRate: fast
MIIMonitorSec: 100
filename: 05-generic-bond0
_systemd_networkd_public_api_devices:
- NetDev:
Name: vlan-public-api
Kind: vlan
VLAN:
Id: 40
filename: 10-openstack-vlan-public-api
- NetDev:
Name: br-public-api
Kind: bridge
Bridge:
ForwardDelaySec: 0
HelloTimeSec: 2
MaxAgeSec: 12
STP: off
filename: 11-openstack-br-public-api
openstack_hosts_systemd_networkd_devices: |-
{% set devices = [] %}
{% if is_metal %}
{% set _ = devices.extend(_systemd_networkd_generic_devices) %}
{% if inventory_hostname in groups['haproxy_hosts'] %}
{% set _ = devices.extend(_systemd_networkd_public_api_devices) %}
{% endif %}
{% endif %}
{{ devices }}
_systemd_networkd_bonded_networks:
- interface: ens3
filename: 05-generic-ens3
bond: bond0
link_config_overrides:
Match:
MACAddress: df:25:83:e1:77:c8
- interface: ens6
filename: 05-generic-ens6
bond: bond0
link_config_overrides:
Match:
MACAddress: df:25:83:e1:77:c9
- interface: bond0
filename: 05-general-bond0
vlan:
- vlan-public-api
_systemd_networkd_public_api_networks:
- interface: "vlan-public-api"
bridge: "br-public-api"
filename: 10-openstack-vlan-public-api
- interface: "br-pub-api"
filename: "11-openstack-br-public-api"
openstack_hosts_systemd_networkd_networks: |-
{% set networks = [] %}
{% if is_metal %}
{% set _ = networks.extend(_systemd_networkd_bonded_networks) %}
{% if inventory_hostname in groups['haproxy_hosts'] %}
{% set _ = networks.extend(_systemd_networkd_public_api_networks) %}
{% endif %}
{% endif %}
{{ networks }}
Defining container networking¶
In case of deploying HAProxy inside LXC you need to ensure connectivity
with a public network and that haproxy_bind_external_lb_vip_address
will be present inside the container as well as external_lb_vip_address
is reachable.
For that we need to do the following series of changes in the
openstack_user_config.yml
file.
In
cidr_networks
add a network which should be used as «public» network for accessing APIs. For example we will be using 203.0.113.128/28:
cidr_networks: ... public_api: 203.0.113.128/28
In
used_ips
you need to reserve IP address for your gateway andhaproxy_keepalived_external_vip_cidr
/external_lb_vip_address
used_ips: ... - "203.0.113.129" - "203.0.113.140-203.0.113.142"
In
provider_networks
you need to define a new container network and assign it to HAproxy group.global_overrides: ... provider_networks: ... - network: group_binds: - haproxy type: "raw" container_bridge: "br-public-api" container_interface: "eth20" container_type: "veth" ip_from_q: public_api static_routes: - cidr: 0.0.0.0/0 gateway: 203.0.113.129
While these are all changes, that need to be done in
openstack_user_config.yml
, there is one more override that needs to be
applied.
As you might have spotted, we are defining a default route for the container through eth20. However, by default all containers have their default route through eth0, which is a local LXC bridge where address is recieved through DHCP. In order to avoid a conflict, you need to ensure that the default route will not be set for eth0 inside the container. For that, create a file /etc/openstack_deploy/group_vars/haproxy with the following content:
lxc_container_networks:
lxcbr0_address:
bridge: "{{ lxc_net_bridge | default('lxcbr0') }}"
bridge_type: "{{ lxc_net_bridge_type | default('linuxbridge') }}"
interface: eth0
type: veth
dhcp_use_routes: False
Configuring HAProxy binding inside containers¶
As IP provisioning is quite random inside containers, it may not always be
handy to bind HAProxy to a specific IP address. If that’s the case, you can
bind HAProxy to an interface instead, since we always know the interface names
inside containers. With that keepalived public/internal VIPs are supposed to
be added in used_ips
, so you still can define them freely.
Example bellow shows a possible content in user_variables.yml
:
haproxy_bind_external_lb_vip_interface: eth20
haproxy_bind_internal_lb_vip_interface: eth1
haproxy_bind_external_lb_vip_address: "*"
haproxy_bind_internal_lb_vip_address: "*"
haproxy_keepalived_external_vip_cidr: 203.0.113.140/32
haproxy_keepalived_internal_vip_cidr: 172.29.236.9/32
haproxy_keepalived_external_interface: "{{ haproxy_bind_external_lb_vip_interface }}"
haproxy_keepalived_internal_interface: "{{ haproxy_bind_internal_lb_vip_interface }}"
Alternatively, you can detect IPs used inside your containers to configure
haproxy binds. This can be done by reffering to container_networks
mapping:
haproxy_bind_external_lb_vip_address: "{{ container_networks['public_api_address']['address'] }}"
haproxy_bind_internal_lb_vip_address: "{{ container_networks['management_address']['address'] }}"
Creating containers¶
Once all steps above are accomplished, it’s time to create our new haproxy containers. For that run the following command:
# openstack-ansible playbooks/lxc-containers-create.yml --limit haproxy,lxc_hosts