Deployment Configuration Terminology¶
Before diving into the details of the individual deployment configurations, this section introduces key terminology that will be used throughout the StarlingX Deployment Configurations.
Node Types¶
Controller Node / Function¶
Runs cloud control functions for managing cloud resources.
Hosts all Kubernetes control functions (e.g., managing images, pods, services).
Typically, part of a two-node HA cluster operating in Active/Standby mode.
In small standard configurations, the system can also support a local Kubernetes persistent volume storage solution by running a small-scale Rook Ceph cluster (i.e., the Rook ‘Controller’ Deployment model).
Worker Node / Function¶
Hosts containerized applications.
For large standard configurations, worker nodes may also participate in a local Kubernetes persistent volume storage solution by running assigned services for a Rook Ceph cluster.
In the Rook ‘Dedicated’ deployment model, you can assign storage devices only to worker hosts. You can also schedule additional application workloads on these workers, but it is recommended to minimize this because Rook Ceph storage processes consume significant resources.
In the Rook ‘Open’ deployment model, any host can be assigned a Rook Ceph monitor or storage device. No restrictions are placed on where Ceph resources are assigned. As the model does not enforce assignment restrictions, care should be taken when making these assignments to ensure they result in a healthy Ceph cluster. As with the ‘Dedicated’ Deployment model, additional application workloads can be scheduled to these hosts. Proper allocation of resources (CPU/memory) should be considered for optimal performance of any application and the Rook Ceph cluster.
Storage Node / Function¶
This node type/function provides local storage and does not run Kubernetes. This node is dedicated to local storage, not running Kubernetes, and participates in a multi-node, highly available Ceph cluster (bare metal Ceph). It supports up to 4x pairs of nodes configured with a Ceph replication of 2 or 3x pairs of nodes configured with a Ceph replication of 3.
While Rook Ceph storage is recommended for new deployments, the storage node/function continues to be supported for standard configurations.
All-In-One (AIO) Node¶
A single physical node that combines controller, worker, and storage roles. The workers may provide local Kubernetes persistent volume storage by running services for a Rook Ceph cluster (the Rook ‘Controller’ Deployment model) using one or more disks (SATA, SAS, SSD, NVMe) as Ceph OSDs. However, deploying Rook Ceph is optional, and you can use an external solution (such as NetApp CSI or Dell CSI) instead of local persistent storage.
Networking¶
OAM Network¶
Controller nodes use this network to expose all external StarlingX platform interfaces, including REST APIs (Keystone, StarlingX, Kubernetes), the Horizon web UI, SSH access, and SNMP services.
It is typically provisioned as a 1 Gigabit Ethernet (1GE) network and is dedicated to external platform access and administrative operations.
Management (MGMT) Network¶
A private network used for internal StarlingX platform operations, including system monitoring, control, and container access to the storage cluster.
This network is typically provisioned as a high-bandwidth 10 Gigabit Ethernet (10GE) connection.
Admin Network¶
The admin network is an optional Layer 3 network used in Distributed Cloud environments to facilitate monitoring and control between the System Controller (central cloud) and subclouds within the StarlingX.
In the absence of an admin network, this functionality is handled by the management network. However, for new Distributed Cloud deployments, using a dedicated admin network is recommended to separate the system controller to subcloud traffic from the management network. The admin network offers greater flexibility for reconfiguration, particularly when adapting to changes in subnetting or IP addressing.
Cluster Host Network¶
The cluster host network facilitates Kubernetes control and management functions, as well as private container-to-container communication. It is powered by the Calico CNI service, which enables secure, tunneled networking between containers.
By default, this network is internal and shares the same physical interface as the management network. However, it can be configured on a dedicated interface or on a separate network during initial setup, if needed.
Depending on deployment requirements, the cluster host network can serve as either an internal-only network or be extended to provide external connectivity. Rook-deployed containerized Ceph services are configured to use this network by default.
Network Connectivity with EXTERNAL Cluster Host Network¶
Calico provides private tunneled networking between hosted containers on the externally exposed cluster host network.
Containers can be exposed externally through:
NodePort services on all controller and worker node interfaces.
Ingress controller services.
BGP within the Calico CNI service. The Calico BGP configuration can be modified to advertise selected application container services or the ingress controller service to a BGP peer, specifying the available next hop worker nodes’ cluster host IP addresses.
HA achieved through:
External HA load balancer across multiple worker nodes.
Multiple DNS records pointing to different worker node IPs.
Network Connectivity with INTERNAL Cluster Host Network¶
When the cluster host network is INTERNAL, external network access is provided through:
The OAM port, or
Additional configured ports on both controller and worker nodes.
Containers exposed through:
NodePort services on OAM or additional interfaces.
Ingress controller services.
HA achieved through:
An external HA load balancer across multiple worker nodes, or
Multiple DNS records pointing to different worker node IPs.
Calico BGP is not supported for advertising container endpoints in this configuration.
Additional External/Data Networks¶
Used by ingress controllers and hosted application containers to expose Kubernetes services (e.g., through NodePort).
Node interfaces connected to these networks are configured as platform class interfaces.
May also refer to data networks attached to node interfaces configured as PCI-SR-IOV class interfaces.
Supports direct connectivity between hosted application containers and the host’s network interface through:
PCI passthrough, or
SR-IOV
IPMI Network¶
An optional Layer 3 network used to connect the IPMI interfaces of all nodes.
This network must be IP-reachable from the controllers’ OAM interfaces to allow platform services running on the controllers to manage and monitor other hosts (workers, storage) in the cluster through IPMI. For example, out-of-band IPMI resets, collecting IPMI sensor data, etc.
PxeBoot Network (All Nodes)¶
An IPv4 network over which nodes netboot from controllers.
The node’s interface to the PXE Boot Network MUST be untagged and on the same physical interface as the node’s management network interface. If the node’s Management Network interface is configured untagged on a physical interface, then the node’s interface to the PXE Boot Network does NOT need to be explicitly configured.
However, if the node’s Management Network interface is configured on a tagged VLAN interface, then the node’s interface to the PXE Boot Network must be EXPLICITLY configured.
The node’s Management Network interface is recommended to be configured on a tagged VLAN in the following cases:
If the Management Network MUST be IPv6, it is recommended to configure the node’s interface to the Management Network on a tagged VLAN. This avoids potential complications with multiplexing an IPv4 PXE Boot Network and an IPv6 Management Network on the same L2 network.
If the Management Network needs to be dual-stack (IPv4 and IPv6), IPv6 is selected as the primary address pool family (see IPv4/IPv6 Dual-Stack Network). Again this avoids potential complications with multiplexing an IPv4 PXE Boot Network and a dual-stack Management Network on the same L2 network.
Node Interfaces¶
Node interfaces define how network connectivity is physically and logically structured across the platform.
Node network interfaces can be configured as:
Untagged single port
Untagged two-port LAG, optionally split across redundant L2 switches
VLAN on a single port or two-port LAG
