Please refer to the following diagram of the Octavia v0.5 components:
This milestone release of Octavia concentrates on making the service delivery scalable (though individual listeners are not horizontally scalable at this stage), getting API and other interfaces between major components correct, without worrying about making the command and control layer scalable.
Note that this design is not yet “operator grade” but is a good first step to achieving operator grade (which will occur with version 1 of Octavia).
The entities in this section describe components that are part of the Neutron LBaaS project, with which Octavia interfaces to deliver load balancing services.
This is the front-end that users (and user GUIs or what have you) talk to to manipulate load balancing services.
Notes:
The entities in this section are “glue” components which allow Octavia to interface with other services in the OpenStack environment. The idea here is that we want Octavia to be as loosely-coupled as possible with those services with which it must interact in order to keep these interfaces as clean as possible.
Initially, all the components in this section will be entirely under the purview of the Octavia project. Over time some of these components might be eliminated entirely, or reduced in scope as these third-party services evolve and increase in cleanly-consumable functionality.
This is the part of the load balancing service that actually interfaces between the (sanitized) user and operator configuration and the back-end load balancing appliances or other “service providing entity.”
Notes:
In order to keep Octavia’s design more clean as a pure consumer of network services, yet still be able to develop Octavia at a time when it is impossible to provide the kind of load balancing services we need to provide without “going around” the existing Neutron API, we have decided to write a “network driver” component which does those dirty back-end configuration commands via an API we write, until these can become a standard part of Neutron. This component should be as loosely coupled with Octavia as Octavia will be with Neutron and present a standard interface to Octavia for accomplishing network configuration tasks (some of which will simply be a direct correlation with existing Neutron API commands).
Notes:
Everything from here down are entities that have to do with the Octavia driver and load balancing system. Other vendor drivers are unlikely to have the same components and internal structure. It is planned that Octavia will become the new reference implementation for LBaaS, though it of course doesn’t need to be the only one. (In fact, a given operator should be able to use multiple vendors with potentially multiple drivers and multiple driver configurations through the Neutron Flavor framework.)
This is exactly like the USER API HANDLER in function, except that implementation details are exposed to the operator, and certain admin-level features are exposed (ex. listing a given tenant’s loadbalancers, & etc.)
It’s also anticipated that the Operator API needs will vary enough from implementation to implementation that no single Operator API will be sufficient for the needs of all vendor implementations. (And operators will definitely have implementation-specific concerns.) Also, we anticipate that most vendors will already have an operator API or other interface which is controlled and configured outsite the purview of OpenStack in general. As such it makes sense for Octavia to have its own operator API / interface.
Notes:
This is the component providing all the command and control for the amphorae. On the front end, it takes its commands and controls from the LBaaS driver.
It should be noted that in later releases of Octavia, the controller functions will be split across several components. At this stage we are less concerned with how this internal communcation will happen, and are most concerned with ensuring communication with amphorae, the amphora LB driver, and the Network driver are all made as perfect as possible.
Among the controller’s responsibilities are:
Notes:
This is the abstraction layer that the controller talks to for communicating with the amphorae. Since we want to keep Octavia flexible enough so that certain components (like the amphora) can be replaced by third party products if the operator so desires, it’s important to keep many of the implementation-specific details contained within driver layers. An amphora LB driver also gives the operator the ability to have different open-source amphorae with potentially different capabilities (accessed via different flavors) which can be handy for, for example, field-testing a new amphora image.
The reference implementation for the amphora LB driver will be for the amphora described below.
Responsibilities of the amphora LB driver include:
This is the subnet that controllers will use to communicate with amphorae. This means that controllers must have connectivity (either layer 2 or routed) to this subnet in order to function, and visa versa. Since amphorae will be communicating on it, this means the network is not part of the “undercloud.”
Notes:
This is a Nova VM which actually provides the load balancing services as configured by the user. Responsibilities of these entities include:
Notes:
There are actually a few of these, all of which need to be driven by some daemon(s) which periodically check that heartbeats from monitored entities are both current and showing “good” status, if applicable. Specifically:
The idea here is that internal health monitors will monitor a periodic heartbeat coming from the amphorae, and take appropriate action (assuming these are down) if they fail to check in with a heartbeat frequently enough. This means that internal health monitors need to take the form of a daemon which is constantly checking for and processing heartbeat requests (and updating controller or amphorae statuses, and triggering other events as appropriate).
In order to keep things as scalable as possible, the thought was that short, periodic and arguably less vital messages being emitted by the amphora and associated controller would be done via HMAC-signed UDP, and more vital, more sensitive, and potentially longer transactional messages would be handled via a RESTful API on the controller, accessed via bi-directionally authenticated HTTPS.
Specifically, we should expect the following to happen over UDP: * heartbeats from the amphora VM to the controller
And the following would happen over TCP: * haproxy / tls certificate configuration changes
Initially, I propose we support two topologies with version 0.5 of Octavia:
We considered also supporting “Single node” topology, but this turns out to be the same thing as option 1 above with a spares pool size of zero.
This is actually where things get tricky, as far as amphora plumbing is concerned. And it only grows trickier when we consider that front-end connectivity (ie. to the ‘loadbalancer’ vip_address) and back-end connectivity (ie. to members of a loadbalancing pool) can be handled in different ways. Having said this, we can break things down into LB network, front-end and back-end topology to discuss the various possible permutations here.
Each amphora needs to have a connection to a LB network. And each controller needs to have access to this management network (this could be layer-2 or routed connectivity). Command and control will happen via the amphorae’s LB network IP.
There are generally two ways to handle the amphorae’s connection to the front-end IP address (this is the vip_address of the loadbalancer object):
Option 1: Layer-2 connectivity
The amphora can have layer-2 connectivity to the neutron network which is host to the subnet on which the loadbalancer vip_address resides. In this scenario, the amphora would need to send ARP responses to requests for the vip_address, and therefore amphorae need to have interfaces plumbed on said vip_address subnets which participate in ARP.
Note that this is somewhat problematic for active / standby virtual appliance topologies because the vip_address for a given load balancer effectively becomes a highly-available IP address (a true floating VIP), which means on service failover from active to standby, the active amphora needs to relenquish all the vip_addresses it has, and the standby needs to take them over and start up haproxy services. This is OK if a given amphora only has a few load balancers, but can lead to several minutes’ down-time during a graceful failover if there are a dozen or more load balancers on the active/standby amphora pair. It’s also more risky: The standby node might not be able to start up all the haproxy services during such a failover. What’s more, most types of VRRP-like services which handle floating IPs require amphorae to have an additional IP address on the subnet housing the floating vip_address in order for the standby amphora to monitor the active amphora.
Also note that in this topology, amphorae need an additional virtual network interface plumbed when new front-end loadbalancer vip_addresses are assigned to them which exist on subnets to which they don’t already have access.
Option 2: Routed (layer-3) connectivity
In this layout, static routes are injected into the routing infrastructure (Neutron) which essentially allow traffic destined for any given loadbalancer vip_address to be routed to an IP address which lives on the amphora. (I would recommend this be something other than the LB network IP.) In this topology, it’s actually important that the loadbalancer vip_address does not exist in any subnet with potential front-end clients because in order for traffic to reach the loadbalancer, it must pass through the routing infrastructure (and in this case, front-end clients would attempt layer-2 connectivity to the vip_address).
This topology also works much better for active/standby configurations, because both the active and standby amphorae can bind to the vip_addresses of all their assigned loadbalancer objects on a dummy, non-ARPing interface, both can be running all haproxy services at the same time, and keep the standby server processes from interfering with active loadbalancer traffic through the use of fencing scripts on the amphorae. Static routing is accomplished to a highly available floating “routing IP” (using some VRRP-like service for just this IP) which becomes the trigger for the fencing scripts on the amphora. In this scenario, fail-overs are both much more reliable, and can be accomplished in usually < 5 seconds.
Further, in this topology, amphorae do not need any additional virtual interfaces plumbed when new front-end loadbalancer vip_addresses are assigned to them.
There are also two ways that amphorae can potentially talk to back-end member IP addresses. Unlike the front-end topologies (where option 1 and option 2 are basically mutually exclusive, if not practically exclusive) both of these types of connectivity can be used on a single amphora, and indeed, within a single loadbalancer configuration.
Option 1: Layer-2 connectivity
This is layer-2 connectivity to back-end members, and is implied when a member object has a subnet_id assigned to it. In this case, the existence of the subnet_id implies amphorae need to have layer-2 connectivity to that subnet, which means they need to have a virtual interface plumbed to it, as well as an IP address on the subnet. This type of connectivity is useful for “secure” back-end subnets that exist behind a NATing firewall where PAT is not in use on the firewall. (In this way it effectively bypasses the firewall.) We anticipate this will be the most common form of back-end connectivity in use by most OpenStack users.
Option 2: Routed (layer-3) connectivity
This is routed connectivity to back-end members. This is implied when a member object does not have a subnet_id specified. In this topology, it is assumed that member ip_addresses are reachable through standard neutron routing, and therefore connections to them can be initiated from the amphora’s default gateway. No new virtual interfaces need to be plumbed for this type of connectivity to members.