Kuryr Kubernetes Integration Design¶

Watcher¶

Watcher is a common software component used by both the Controller and the CNI driver. Watcher connects to Kubernetes API. Watcher’s responsibility is to observe the registered (either on startup or dynamically during its runtime) endpoints and invoke registered callback handler (pipeline) to pass all events from registered endpoints.

Event Handler¶

EventHandler is an interface class for the Kubernetes event handling. There are several ‘wrapper’ event handlers that can be composed to implement Controller handling pipeline.

Retry Event Handler is used for handling specified failures during event processing. It can be used to ‘wrap’ another EventHandler and in case of specified error will retry the wrapped event handler invocation within specified timeout. In case of persistent failure, Retry will raise the wrapped EventHandler exception.

Async Event Handler is used to execute event handling asynchronously. Events are grouped based on the specified ‘thread_groups’. Events of the same group are processed in order of arrival. Thread group maps to an unique K8s resource (each Pod, Service, etc.). Async can be used to ‘wrap’ another EventHandler. Queues per thread group are added dynamically once relevant events arrive and removed once queue is empty.

LogExceptions Event Handler suppresses exceptions and sends them to log facility.

Dispatcher is an Event Handler that distributes events to registered handlers based on event content and handler predicate provided during event handler registration.

ControllerPipeline¶

ControllerPipeline serves as an event dispatcher of the Watcher for Kuryr-K8s controller Service. Currently watched endpoints are ‘pods’, ‘services’ and ‘endpoints’. Kubernetes resource event handlers (Event Consumers) are registered into the Controller Pipeline. There is a special EventConsumer, ResourceEventHandler, that provides API for Kubernetes event handling. When a watched event arrives, it is processed by all Resource Event Handlers registered for specific Kubernetes object kind. Pipeline retries on resource event handler invocation in case of the ResourceNotReady exception till it succeeds or the number of retries (time-based) is reached. Any unrecovered failure is logged without affecting other Handlers (of the current and other events). Events of the same group (same Kubernetes object) are handled sequentially in the order arrival. Events of different Kubernetes objects are handled concurrently.

ResourceEventHandler¶

ResourceEventHandler is a convenience base class for the Kubernetes event processing. The specific Handler associates itself with specific Kubernetes object kind (through setting OBJECT_KIND) and is expected to implement at least one of the methods of the base class to handle at least one of the ADDED/MODIFIED/DELETED events of the Kubernetes object. For details, see k8s-api. Since both ADDED and MODIFIED event types trigger very similar sequence of actions, Handler has ‘on_present’ method that is invoked for both event types. The specific Handler implementation should strive to put all the common ADDED and MODIFIED event handling logic in this method to avoid code duplication.

Pluggable Handlers¶

Starting with the Rocky release, Kuryr-Kubernetes includes a pluggable interface for the Kuryr controller handlers. The pluggable handlers framework allows :

• Using externally provided handlers.
• Controlling which handlers should be active.

To control which Kuryr Controller handlers should be active, the selected handlers need to be included in kuryr.conf at the ‘kubernetes’ section. If not specified, Kuryr Controller will run the default handlers, which currently includes the following:

For example, to enable only the ‘vif’ controller handler we should set the following at kuryr.conf:

[kubernetes]
enabled_handlers=vif

Providers¶

Provider (Drivers) are used by ResourceEventHandlers to manage specific aspects of the Kubernetes resource in the OpenStack domain. For example, creating a Kubernetes Pod will require a neutron port to be created on a specific network with the proper security groups applied to it. There will be dedicated Drivers for Project, Subnet, Port and Security Groups settings in neutron. For instance, the Handler that processes pod events, will use PodVIFDriver, PodProjectDriver, PodSubnetsDriver and PodSecurityGroupsDriver. The Drivers model is introduced in order to allow flexibility in the Kubernetes model mapping to the OpenStack. There can be different drivers that do Neutron resources management, i.e. create on demand or grab one from the precreated pool. There can be different drivers for the Project management, i.e. single Tenant or multiple. Same goes for the other drivers. There are drivers that handle the Pod based on the project, subnet and security groups specified via configuration settings during cluster deployment phase.

NeutronPodVifDriver¶

PodVifDriver subclass should implement request_vif, release_vif and activate_vif methods. In case request_vif returns Vif object in down state, Controller will invoke activate_vif. Vif ‘active’ state is required by the CNI driver to complete pod handling. The NeutronPodVifDriver is the default driver that creates neutron port upon Pod addition and deletes port upon Pod removal.

Communication¶

CNI Daemon Server is starting an HTTP server on a local network socket (127.0.0.1:50036 by default). Currently server is listening for 2 API calls. Both calls load the CNIParameters from the body of the call (it is expected to be JSON).

For reference see updated pod creation flow diagram: