Active-Active, N+1 Amphorae Setup
https://blueprints.launchpad.net/octavia/+spec/active-active-topology
Problem description
This blueprint describes how Octavia implements an active-active
loadbalancer (LB) solution that is highly-available through redundant
Amphorae. It presents the high-level service topology and suggests
high-level code changes to the current code base to realize this scenario.
In a nutshell, an Amphora Cluster of two or more active Amphorae
collectively provide the loadbalancing service.
The Amphora Cluster shall be managed by an Amphora Cluster Manager (ACM).
The ACM shall provide an abstraction that allows different types of
active-active features (e.g., failure recovery, elasticity, etc.). The
initial implementation shall not rely on external services, but the
abstraction shall allow for interaction with external ACMs (to be developed
later).
This blueprint uses terminology defined in Octavia glossary when available,
and defines new terms to describe new components and features as necessary.
Note: Items marked with [P2] refer to lower priority features to be
designed / implemented only after initial release.
Proposed change
A tenant should be able to start a highly-available, loadbalancer for the
tenant’s backend services as follows:
- The operator should be able to configure an active-active topology
through an Octavia configuration file or [P2] through a Neutron flavor,
which the loadbalancer shall support. Octavia shall support active-active
topologies in addition to the topologies that it currently supports.
- In an active-active topology, a cluster of two or more amphorae shall
host a replicated configuration of the load-balancing services. Octavia
will manage this Amphora Cluster as a highly-available service using a
pool of active resources.
- The Amphora Cluster shall provide the load-balancing services and support
the configurations that are supported by a single Amphora topology,
including L7 load-balancing, SSL termination, etc.
- The active-active topology shall support various Amphora types and
implementations; including, virtual machines, [P2] containers, and
bare-metal servers.
- The operator should be able to configure the high-availability
requirements for the active-active load-balancing services. The operator
shall be able to specify the number of healthy Amphorae that must exist
in the load-balancing Amphora Cluster. If the number of healthy Amphorae
drops under the desired number, Octavia shall automatically and
seamlessly create and configure a new Amphora and add it to the Amphora
Cluster. [P2] The operator should be further able to define that the
Amphora Cluster shall be allocated on separate physical resources.
- An Amphora Cluster will collectively act to serve as a single logical
loadbalancer as defined in the Octavia glossary. Octavia will seamlessly
distribute incoming external traffic among the Amphorae in the Amphora
Cluster. To that end, Octavia will employ a Distributor component that
will forward external traffic towards the managed amphora instances.
Conceptually, the Distributor provides an extra level of load-balancing
for an active-active Octavia application, albeit a simplified one.
Octavia should be able to support several Distributor implementations
(e.g., software-based and hardware-based) and different affinity models
(at minimum, flow-affinity should be supported to allow TCP connectivity
between clients and Amphorae).
- The detailed design of the Distributor component will be described in a
separate document (see “Distributor for Active-Active, N+1 Amphorae
Setup”, active-active-distributor.rst).
High-level Topology Description
Single Tenant
- The following diagram illustrates the active-active topology:
Front-End Back-End
Internet Network Network
(world) (tenant) (tenant)
║ ║ ║
┌─╨────┐ floating IP ║ ║ ┌────────┐
│Router│ to LB VIP ║ ┌────┬─────────┬────┐ ║ │ Tenant │
│ GW ├──────────────►╫◄─┤ IP │ Amphora │ IP ├─►╫◄─┤Service │
└──────┘ ║ └┬───┤ (1) │back│ ║ │ (1) │
║ │VIP├─┬──────┬┴────┘ ║ └────────┘
║ └───┘ │ MGMT │ ║ ┌────────┐
╓◄───────────────────║─────────┤ IP │ ║ │ Tenant │
║ ┌─────────┬────┐ ║ └──────┘ ╟◄─┤Service │
║ │ Distri- │ IP├►╢ ║ │ (2) │
║ │ butor ├───┬┘ ║ ┌────┬─────────┬────┐ ║ └────────┘
║ └─┬──────┬┤VIP│ ╟◄─┤ IP │ Amphora │ IP ├─►╢ ┌────────┐
║ │ MGMT │└─┬─┘ ║ └┬───┤ (2) │back│ ║ │ Tenant │
╟◄────┤ IP │ └arp►╢ │VIP├─┬──────┬┴────┘ ╟◄─┤Service │
║ └──────┘ ║ └───┘ │ MGMT │ ║ │ (3) │
╟◄───────────────────║─────────┤ IP │ ║ └────────┘
║ ┌───────────────┐ ║ └──────┘ ║
║ │ Octavia LBaaS │ ║ ••• ║ •
╟◄─┤ Controller │ ║ ┌────┬─────────┬────┐ ║ •
║ └┬─────────────┬┘ ╙◄─┤ IP │ Amphora │ IP ├─►╢
║ │ Amphora │ └┬───┤ (k) │back│ ║ ┌────────┐
║ │ Cluster Mgr.│ │VIP├─┬──────┬┴────┘ ║ │ Tenant │
║ └─────────────┘ └───┘ │ MGMT │ ╙◄─┤Service │
╟◄─────────────────────────────┤ IP │ │ (m) │
║ └──────┘ └────────┘
║
Management Amphora Cluster Back-end Pool
Network 1..k 1..m
- An example of high-level data-flow:
- Internet clients access a tenant service through an externally visible
floating-IP (IPv4 or IPv6).
- If IPv4, a gateway router maps the floating IP into a loadbalancer’s
internal VIP on the tenant’s front-end network.
- The (multi-tenant) Distributor receives incoming requests to the
loadbalancer’s VIP. It acts as a one-legged direct return LB,
answering arp requests for the loadbalancer’s VIP (see Distributor
spec.).
- The Distributor distributes incoming connections over the tenant’s
Amphora Cluster, by forwarding each new connection opened with a
loadbalancer’s VIP to a front-end MAC address of an Amphora in the
Amphora Cluster (layer-2 forwarding). Note: the Distributor may
implement other forwarding schemes to support more complex routing
mechanisms, such as DVR (see Distributor spec.).
- An Amphora receives the connection and accepts traffic addressed to
the loadbalancer’s VIP. The front-end IPs of the Amphorae are
allocated on the tenant’s front-end network. Each Amphora accepts VIP
traffic, but does not answer arp request for the VIP address.
- The Amphora load-balances the incoming connections to the back-end
pool of tenant servers, by forwarding each external request to a
member on the tenant network. The Amphora also performs SSL
termination if configured.
- Outgoing traffic traverses from the back-end pool members, through
the Amphora and directly to the gateway (i.e., not through the
Distributor).
Multi-tenant Support
- The following diagram illustrates the active-active topology with
multiple tenants:
Front-End Back-End
Internet Networks Networks
(world) (tenant) (tenant)
║ B A A
║ floating IP ║ ║ ║ ┌────────┐
┌─╨────┐ to LB VIP A ║ ║ ┌────┬─────────┬────┐ ║ │Tenant A│
│Router├───────────────║─►╫◄─┤A IP│ Amphora │A IP├─►╫◄─┤Service │
│ GW ├──────────────►╢ ║ └┬───┤ (1) │back│ ║ │ (1) │
└──────┘ floating IP ║ ║ │VIP├─┬──────┬┴────┘ ║ └────────┘
to LB VIP B ║ ║ └───┘ │ MGMT │ ║ ┌────────┐
╓◄───────────────────║──║─────────┤ IP │ ║ │Tenant A│
║ ║ ║ └──────┘ ╟◄─┤Service │
M B A ┌────┬─────────┬────┐ ║ │ (2) │
║ ║ ╟◄─┤A IP│ Amphora │A IP├─►╢ └────────┘
║ ║ ║ └┬───┤ (2) │back│ ║ ┌────────┐
║ ║ ║ │VIP├─┬──────┬┴────┘ ║ │Tenant A│
║ ║ ║ └───┘ │ MGMT │ ╟◄─┤Service │
╟◄───────────────────║──║─────────┤ IP │ ║ │ (3) │
║ ║ ║ └──────┘ ║ └────────┘
║ B A ••• B •
║ ┌─────────┬────┐ ║ ║ ┌────┬─────────┬────┐ ║ •
║ │ │IP A├─╢─►╫◄─┤A IP│ Amphora │A IP├─►╢ ┌────────┐
║ │ ├───┬┘ ║ ║ └┬───┤ (k) │back│ ║ │Tenant A│
║ │ Distri- │VIP├─arp►╜ │VIP├─┬──────┬┴────┘ ╙◄─┤Service │
║ │ butor ├───┘ ║ └───┘ │ MGMT │ │ (m) │
╟◄─ │ │ ─────║────────────┤ IP │ └────────┘
║ │ ├────┐ ║ └──────┘
║ │ │IP B├►╢ tenant A
║ │ ├───┬┘ ║ = = = = = = = = = = = = = = = = = = = = =
║ │ │VIP│ ║ ┌────┬─────────┬────┐ B tenant B
║ └─┬──────┬┴─┬─┘ ╟◄────┤B IP│ Amphora │B IP├─►╢ ┌────────┐
║ │ MGMT │ └arp►╢ └┬───┤ (1) │back│ ║ │Tenant B│
╟◄────┤ IP │ ║ │VIP├─┬──────┬┴────┘ ╟◄─┤Service │
║ └──────┘ ║ └───┘ │ MGMT │ ║ │ (1) │
╟◄───────────────────║────────────┤ IP │ ║ └────────┘
║ ┌───────────────┐ ║ └──────┘ ║
M │ Octavia LBaaS │ B ••• B •
╟◄─┤ Controller │ ║ ┌────┬─────────┬────┐ ║ •
║ └┬─────────────┬┘ ╙◄────┤B IP│ Amphora │B IP├─►╢
║ │ Amphora │ └┬───┤ (q) │back│ ║ ┌────────┐
║ │ Cluster Mgr.│ │VIP├─┬──────┬┴────┘ ║ │Tenant B│
║ └─────────────┘ └───┘ │ MGMT │ ╙◄─┤Service │
╟◄────────────────────────────────┤ IP │ │ (r) │
║ └──────┘ └────────┘
║
Management Amphora Clusters Back-end Pool
Network A(1..k), B(1..q) A(1..m),B(1..r)
- Both tenants A and B share the Distributor, but each has a different
front-end network. The Distributor listens on both loadbalancers’ VIPs
and forwards to either A’s or B’s Amphorae.
- The Amphorae and the back-end (tenant) networks are not shared between
tenants.
Problem Details
- Octavia should support different Distributor implementations, similar
to its support for different Amphora types. The operator should be able
to configure different types of algorithms for the Distributor. All
algorithms should provide flow-affinity to allow TLS termination at the
amphora. See Distributor for Active-Active, N+1 Amphorae Setup for details.
- Octavia controller shall seamlessly configure any newly created Amphora
([P2] including peer state synchronization, such as sticky-tables, if
needed) and shall reconfigure the other solution components (e.g.,
Neutron) as needed. The controller shall further manage all Amphora
life-cycle events.
- Since it is impractical at scale for peer state synchronization to occur
between all Amphorae part of a single load balancer, Amphorae that are all
part of a single load balancer configuration need to be divided into smaller
peer groups (consisting of 2 or 3 Amphorae) with which they should
synchronize state information.
Required changes
The active-active loadbalancers require the following high-level changes:
Amphora Cluster Manager driver for the active-active topology (new)
- Add an active-active topology to the topology types.
- Add a new driver to support creation/deletion of an Amphora Cluster via
an ACM. This will re-use existing controller-worker flows as much as
possible. The reference ACM will call the existing drivers to create
compute nodes for the Amphorae and configure them.
- The ACM shall orchestrate creation and deletion of Amphora instances to
meet the availability requirements. Amphora failover will utilize the
existing health monitor flows, with hooks to notify the ACM when
ACTIVE-ACTIVE topology is used. [P2] ACM shall handle graceful amphora
removal via draining (delay actual removal until existing connections are
terminated or some timeout has passed).
- Change the flow of LB creation. The ACM driver shall create an Amphora
Cluster instance for each new loadbalancer. It should maintain the
desired number of Amphorae in the Cluster and meet the
high-availability configuration given by the operator. Note: a base
functionality is already supported by the Health Manager; it may be
enough to support a fixed or dynamic cluster size. In any case, existing
flows to manage Amphora life cycle will be re-used in the reference ACM
driver.
- The ACM shall be responsible for providing health, performance, and
life-cycle management at the Cluster-level rather than at Amphora-level.
Maintaining the loadbalancer status (as described above) by some function
of the collective status of all Amphorae in the Cluster is one example.
Other examples include tracking configuration changes, providing Cluster
statistics, monitoring and maintaining compute nodes for the Cluster,
etc. The ACM abstraction would also support pluggable ACM implementations
that may provide more advance capabilities (e.g., elasticity, AZ aware
availability, etc.). The reference ACM driver will re-use existing
components and/or code which currently handle health, life-cycle, etc.
management for other load balancer topologies.
- New data model for an Amphora Cluster which has a one-to-one mapping with
the loadbalancer. This defines the common properties of the Amphora
Cluster (e.g., id, min. size, desired size, etc.) and additional
properties for the specific implementation.
- Add configuration file options to support configuration of an
active-active Amphora Cluster. Add default configuration. [P2] Add
Operator API.
- Add or update documentation for new components added and new or changed
functionality.
- Communication between the ACM and Distributors should be secured using
two-way SSL certificate authentication much the same way this is accomplished
between other Octavia controller components and Amphorae today.
Network driver changes
- Support the creation, connection, and configuration of the various
networks and interfaces as described in ‘high-level topology’ diagram.
- Adding a new loadbalancer requires attaching the Distributor to the
loadbalancer’s front-end network, adding a VIP port to the Distributor,
and configuring the Distributor to answer arp requests for the VIP.
The Distributor shall have a separate interface for each loadbalancer and
shall not allow any routing between different ports; in particular,
Amphorae of different tenants must not be able to communicate with each
other. In the reference implementation, this will be accomplished by using
separate OVS bridges per load balancer.
- Adding a new Amphora requires attaching it to the front-end and back-end
networks (similar to current implementation), adding the VIP (but with
arp disabled), and registering the Amphora with the Distributor. The
tenant’s front-end and back-end networks must allow attachment of
dynamically created Amphorae by involving the ACM (e.g., when the health
monitor replaces a failed Amphora). ([P2] extend the LBaaS API to allow
specifying an address range for new Amphorae usage, e.g., a subnet pool).
Amphora health-monitoring support
- Modify Health Manager to manage the health for an Amphora Cluster through
the ACM; namely, forward Amphora health change events to the ACM, so it
can decide when the Amphora Cluster is considered to be in healthy state.
This should be done in addition to managing the health of each Amphora.
[P2] Monitor the Amphorae also on their front-end network (i.e., from
the Distributor).
Distributor support
- Note: as mentioned above, the detailed design of the Distributor
component is described in a separate document). Some design
considerations are highlighted below.
- The Distributor should be supported similarly to an Amphora; namely, have
its own abstract driver.
- For a reference implementation, add support for a Distributor image.
- Define a REST API for Distributor configuration (no SSH API). The API
shall support:
- Add and remove a VIP (loadbalancer) and specify distribution parameters
(e.g., affinity, algorithm, etc.).
- Registration and de-registration of Amphorae.
- Status
- [P2] Macro-level stats
- Spawn Distributors (if using on demand Distributor compute nodes) and/or
attach to existing ones as needed. Manage health and life-cycle of the
Distributor(s). Create, connect, and configure Distributor networks as
necessary.
- Create data model for the Distributor.
- Add Distributor driver and flows to (re-)configure the Distributor on
creation/destruction of a new loadbalancer (add/remove loadbalancer VIP)
and [P2] configure the distribution algorithm for the loadbalancer’s
Amphora Cluster.
- Add flows to Octavia to (re-)configure the Distributor on adding/removing
Amphorae from the Amphora Cluster.
Packaging
- Extend Octavia installation scripts to create an image for the Distributor.
Alternatives
- Use external services to manage the cluster directly.
This utilizes functionality that already exists in OpenStack (e.g.,
like Heat and Ceilometer) rather than replicating it. This approach
would also benefit from future extensions to these services. On the
other hand, this adds undesirable dependencies on other projects (and
their corresponding teams), complicates handling of failures, and
require defensive coding around service calls. Furthermore, these
services cannot handle the LB-specific control configuration.
- Implement a nested Octavia
Use another layer of Octavia to distribute traffic across the Amphora
Cluster (i.e., the Amphorae in the Cluster are back-end members of
another Octavia instance). This approach has the potential to provide
greater flexibility (e.g., provide NAT and/or more complex distribution
algorithms). It also potentially reuses existing code. However, we do
not want the Distributor to proxy connections so HA-Proxy cannot be
used. Furthermore, this approach might significantly increase the
overhead of the solution.
Data model impact
- loadbalancer table
- cluster_id: associated Amphora Cluster (no changes to table, 1-1
relationship from Cluster data-model)
- lb_topology table
- amphora_role table
- Distributor table (new): Distributor information, similar to Amphora.
See Distributor for Active-Active, N+1 Amphorae Setup
- Cluster table (new): an extension to loadbalancer (i.e., one-to-one
mapping to load-balancer)
- id (primary key)
- cluster_name: identifier of Cluster instance for Amphora Cluster
Manager
- desired_size: required number of Amphorae in Cluster. Octavia will
create this many active-active Amphorae in the Amphora Cluster.
- min_size: number of ACTIVE Amphorae in Cluster must be above this
number for Amphora Cluster status to be ACTIVE
- cooldown: cooldown period between successive add/remove Amphora
operations (to avoid thrashing)
- load_balancer_id: 1:1 relationship to loadbalancer
- distributor_id: N:1 relationship to Distributor. Support multiple
Distributors
- provisioning_status
- operating_status
- enabled
- cluster_type: type of Amphora Cluster implementation
REST API impact
- Distributor REST API – This is a new internal API that will be secured
via two-way SSL certificate authentication. See
Distributor for Active-Active, N+1 Amphorae Setup
- Amphora REST API – support configuration of disabling arp on VIP.
- [P2] LBaaS API – support configuration of desired availability, perhaps
by selecting a flavor (e.g., gold is a minimum of 4 Amphorae, platinum is
a minimum of 10 Amphora).
- Operator API –
- Topology to use
- Cluster type
- Default availability parameters for the Amphora Cluster
Notifications impact
None.
Other end user impact
None.
Other deployer impact
The reference ACM becomes a new process that is part of the Octavia control
components (like the controller worker, health monitor and housekeeper). If
the reference implementation is used, a new Distributor image will need to be
created and stored in glance much the same way the Amphora image is created
and stored today.
Testing
- Unit tests with tox.
- Function tests with tox.
- Scenario tests.
Documentation Impact
Need to document all new APIs and API changes, new ACTIVE-ACTIVE topology
design and features, and new instructions for operators seeking to deploy
Octavia with ACTIVE-ACTIVE topology.
References