Example LXC based Ironic deployment¶
This section describes a specific deployment of Ironic using Openstack-Ansible. A number of design choices are made which illustrate how to configure the Ironic service for a specific set of requirements.
Deployment design decisions:
LXC containers are used in the openstack control plane
A single
bmaas
network is used for Ironic provisioning, cleaning and inspectionThe
bmaas
network is not routable to any other networks nor to the internal VIPMultitenancy is used with Ironic servers attached to project networks
networking-generic-switch will be used to control network devices to achieve multitenancy
Cisco NXOS switches
The deployment uses VXLAN project networks
A number of these design decisions are opinionated and could be
changed, e.g. allowing the bmaas
network to be routed to
other networks including the internal VIP would result in some
simplification.
This example is illustrative of a specific set of deployment requirements but is not intended to be followed rigidly. It demonstrates some of the capabilities of Ironic and Openstack-Ansible and how to approach most parts of a practical deployment of Ironic at small to modest scales.
Warning
Consideration should be given to the security of IPMI or other out-of-band interfaces which are notoriously buggy and often have vendor specific in-band tools which allow the BMC and its firmware to be manipulated from userspace. Understand the risks of allowing IPMI/BMC interfaces to share a physical interface with onboard ethernet ports, this feature will allow full access to the management interface of a deployed Ironic node. For high security environments or where the users are untrusted consult your hardware vendor for appropriate hardening steps.
BMAAS network address plan¶
In this example the subnet for the bmaas
network is 10.88.104.0/24.
The size of the subnet determines the maximum number of nodes
in the Ironic deployment.
Address |
Purpose |
---|---|
10.88.104.0 to .10 |
Reserve for potential physical routers / SVI |
10.88.104.11 to .29 |
Control plane Ironic container interfaces |
10.88.104.64 to .95 |
Neutron DHCP allocation pool in Ironic_Network |
10.88.104.96 to .127 |
Ironic Inspector DHCP pool range |
10.88.104.128 to .254 |
Static IPs manually assigned to IPMI/iDRAC |
In this deployment the bmaas
network combines several functions
for simplicity. It would be possible to use separate networks for
inspection, cleaning and provisioning but that is beyond the scope
of this example.
The subnet is divided into several address ranges, a reservation for
future interfaces on physical routers, or a gateway address if the
subnet is to be made routable in the future. A small number of addresses
are then reserved for the bmaas
network to connect to the Ironic
containers in the control plane, and the remainder of the addresses
are shared between a neutron network for provisioning, a range for
Ironic Inspector to allocate with DHCP and finally a block of
addresses for the IPMI (or other protocol) management ports
of the Ironic nodes.
Note
This example will use VLAN ID 3003 for the bmaas
network but
any available VLAN ID could be used.
Warning
This example collapses the Ironic IPMI, provisioning, cleaning and inspection networks into the same subnet. It is possible to make these be separate from each other by creating a network for each different function and updating the container networks and Ironic service configuration suitably. In particular it should be understood that the nodes are “untrusted” during the cleaning phase and will be in an arbitrary state as left by the previous user.
IPMI Interfaces¶
When configuring IPMI interfaces for Ironic nodes, the following steps are recommended:
Use static IP allocations for the IPMI interfaces, unless there is already a very reliable means of allocating addresses with DHCP. The Ironic team do not consider the Neutron DHCP agent to be suitable for assigning addresses to the IPMI interfaces.
Use dedicated IPMI ports on Ironic nodes especially if multitenancy is required. A node with a shared onboard ethernet/IPMI port will have that port moved into the tenant network when deployment is complete and the Ironic control plane will no longer be able to communicate with the management interface of the node.
Maximum size of the deployment¶
The maximum size of this Ironic deployment is limited by the
address alloction in the bmaas
network. In this example there can
be a maximum of 127 server BMC interfaces in the range 10.88.104.128/25.
The maxiumum number of servers that can be simultanously provisioned is determined by the address allocation to the Neutron DHCP pool.
The maximum number of servers that can be simultanously inspected is determined by the address allocation to the Ironic Inspector DHCP pool.
To increase the size of the deployment, the size of the bmaas
subnet should be increased and the addresses allocated to meet the
number of required nodes and maximum number of simultaneous nodes
being provisioned and inspected.
Openstack-Ansible configuration¶
Once the address plan has been decided, the Openstack-Ansible configuration can be updated to match.
The existing cidr_networks
and used_ips
sections in
/etc/openstack_deploy/openstack_user_config.yml
must have extra
entries to describe the network range available for the Ansible inventory
to assign to Ironic control plane containers, in this example all addresses
in the bmaas
network are marked as used except the range 10.88.104.11
to 10.88.104.29.
An additional network is defined in the provider_networks
list which
represents the connection between the bridge br-bmaas
on the
controller and eth15
inside the ironic service containers.
The bmaas
network must be extended from the control plane hosts
to the switch ports connected to the Ironic node IPMI interfaces,
and also to switch ports connected to the interfaces on the Ironic
nodes that will be used for PXEboot. This will typically be a VLAN
allocated specifically for Ironic provisioning.
The hosts for the Ironic control plane containers are assigned.
Note
It is the responsibility of the deployer to create br-bmaas
on the controller nodes and ensure that it is connected to
the correct VLAN ID for the bmaas
network. Configuration of
host networking is outside the scope of Openstack-Ansible.
Note
The range
key in the provider network definition is not used
but its useful as an reminder in the config file of the VLAN ID.
cidr_networks:
<existing entries>
bmaas: 10.88.104.0/24 # for containers on the bmaas network
used_ips:
<existing entries>
# bmaas ips
- "10.88.104.0,10.88.104.10" # reserve for routers or other infrastructure
- "10.88.104.30,10.88.104.255" # reserve for ironic IPMI and provisioning
provider_networks:
<existing entries>
# Network definition to connect Ironic LXC containers to the bmaas network
# on the infra hosts
- network:
net_name: physnet_neutron
container_type: "veth"
container_bridge: "br-bmaas"
container_interface: "eth15"
ip_from_q: bmaas
type: "vlan"
range: "3003:3003"
group_binds:
- ironic_api_container
- ironic_compute_container
- ironic_inspector_container
# ironic API and conductor
ironic-infra_hosts:
infra1: *_infra1_
infra2: *_infra2_
infra3: *_infra3_
# nova used by ironic for machine state management
ironic-compute_hosts:
infra1: *_infra1_
infra2: *_infra2_
infra3: *_infra3_
# Ironic-inspector can only support a single instance at the moment
# High availability for ironic-inspector is not yet implemented
ironic-inspector_hosts:
infra1: *_infra1_
Note
This example uses YAML Anchors to simplify openstack_user_config.yml
allowing the IP addresses of the infra nodes to be defined only once.
See https://yaml.org/spec/1.2.2/#alias-nodes.
Create the Neutron configuration¶
Enable the Neutron baremetal
and genericswitch
mechanism drivers
by updating /etc/openstack_deploy/group_vars/neutron_server.yml
---
neutron_plugin_types:
- ml2.genericswitch
- ml2.baremetal
# keep the ml2 drivers in this order
# see https://storyboard.openstack.org/#!/story/2008686
neutron_ml2_mechanism_drivers: "genericswitch,baremetal"
Configure neutron networking-generic-switch to know about the switches
that the Ironic nodes are connected to in /etc/openstack_deploy/user_variables.yml
.
These switches are programmed by neutron to switch the Ironic nodes between the
provisioning and project networks once deployment is complete. This is enabling
multitenancy for Ironic.
This example is for a Cisco NXOS based switch, which uses the same command
set as a Cisco IOS based switch for the functions needed by
networking-generic-switch. There is no specific device_type
for NXOS.
Note
A MAC address for the switch must be specified in the neutron config, but Cisco
and some other switch vendors present a unique MAC address per port so the MAC
address as seen from the client cannot be used to identify the switch. For IOS/NXOS
networking-generic-switch uses the field switch_info
from the Ironic node
local_link_connection
information rather than match a MAC address when
choosing which switch to configure for a particular node.
neutron_neutron_conf_overrides:
genericswitch:my-switch-name: # This should match the hostname configured on the switch
device_type: netmiko_cisco_ios # It is really NXOS but the commands are the same
ngs_mac_address: "cc:46:d6:64:4b:41" # Doesn't seem to matter but is required - this is taken from an SVI on the mgmt network
ip: "10.80.240.3" # An IP on the switch which has ssh access from the br-mgmt network, loopback, SVI or mgmt0 as needed
username: "neutron" # The user that Neutron will SSH to the switch as
password: "supersecret" # The password that Neutron will use to SSH to the switch
## key_file: <ssh key file> # An SSH key may be used instead of a password
ngs_manage_vlans: "False" # VLANs are already provisioned on the switch so tell neutron not to create/delete VLANs
Note
The configuration for networking-generic-switch is added to neturon.conf rather than ml2_conf_genericswitch.ini as the config needs to be read by both neutron-rpc-server and neutron-server. neutron-server is a uwsgi service in openstack-ansible so is only passed one config file, see https://bugs.launchpad.net/openstack-ansible/+bug/1987405
Note
If there is already an override in place for this variable then extend the existing override rather than making a second one.
Configure switch to allow ssh from Neutron¶
To achieve multitenancy, Neutron will connect to the specified switch and
configure the port for the Ironic node being provisioned to be in the
correct project VLAN once the deployment is complete. During deployment
Neutron will ensure that the node is in the bmaas
provisioning
network as specified in the Ironic config.
A suitable user and credential must exist on the switch. The SSH
connection will originate from the Neutron processes running on the
OpenStack control plane, on the mgmt
network. There must be an IP
route from the mgmt
network to an interface on the switch which
permits SSH login. That interface could be a physical management port
(mgmt0
on NXOS), a loopback interface, an SVI or another interface
with an IP address. SSH communication with the switch can happen either
in-band or out-of-band depending on the requirements of the deployment.
This example config is for a neutron
user using password authentication
on an NXOS switch as seen by show run
. The config applied on the
switch gives the neutron
user access to a minimal set of commands
for configuring VLAN membership on specific ports.
To control the commands that the neutron
user is allowed to issue
on the Cisco Nexus switch create a role:
role name neutron-role
rule 3 permit command configure t
rule 2 permit read-write feature interface
rule 1 permit read
vlan policy deny
permit vlan 3003-3003
permit vlan 3100-3200
interface policy deny
permit interface Ethernet1/1
permit interface Ethernet1/2
permit interface Ethernet1/3
permit interface Ethernet1/4
permit interface Ethernet1/5
permit interface Ethernet1/6
permit interface Ethernet1/7
permit interface Ethernet1/8
This role allows the neutron
user assign a port to VLAN 3003
which is the bmaas
network and is used during node provisioning.
Any project VLANS that nodes should be able to be moved into after
deployment should also be permitted, range 3100-3200 here.
The interfaces which the neutron
user is permitted to modify are
listed, in this case individually but consult the switch documentation
for other options such as a regular expression.
A similar config can be made on an Arista switch, where a much more explicit list of allowed CLI commands must be defined using regular expressions.
role neutron-role
10 permit mode exec command configure
20 permit mode exec command terminal width 511
30 permit mode exec command terminal length 0
40 permit mode exec command enable
50 permit mode exec command copy running-config startup-config
60 permit mode config command interface
70 permit mode if-Et([1-9]|27|29)\/1 command switchport mode access
80 permit mode if-Et([1-9]|27|29)\/1 command (no )*switchport access vlan (3003|3966)
90 permit mode if-Et([1-9]|27|29)\/1 command no switchport mode trunk
100 permit mode if-Et([1-9]|27|29)\/1 command switchport trunk allowed vlan none
110 permit mode config command copy running-config startup-config
Create the user and password, which must match those in the
neutron.conf / genericswitch
config file options:
username neutron password 5 <ENCRYPTED-PASSWORD-HERE> role neutron-role
Allow SSH to the switch from the expected
IP addresses, for example a pair out of band management hosts
192.168.0.100/31 and the OpenStack mgmt
network 10.80.240.0/24.
ip access-list ACL_ALLOW_SSH_VTY
10 permit tcp 192.168.0.100/31 any eq 22
20 permit tcp 10.80.240.0/22 any eq 22
line vty
session-limit 5
exec-timeout 10
access-class ACL_ALLOW_SSH_VTY in
Create the Neutron network for Ironic provisioning, cleaning and inspection¶
openstack network create \
--internal \
--provider-network-type vlan \
--provider-physical-network physnet_neutron \
--provider-segment 3003 \
Ironic_Network
openstack subnet create \
--allocation-pool 10.88.104.64-10.88.104.95 \
--dhcp \
--subnet-range 10.88.104.0/24
--gateway none
Ironic_Subnet
Create the Ironic configuration¶
In /etc/openstack_deploy/user_variables_ironic.yml
## IRONIC ##
ironic_ipxe_enabled: yes # use HTTP image download from the ironic conductor container
ironic_enable_web_server_for_images: yes # use same web server to cache user images
# Ensure values used during PXEboot refer directly to the correct interface on Ironic API container
# instead of the internal VIP
ironic_http_url: "{{ ironic_ipxe_proto }}://{{ container_networks['bmaas_address']['address'] }}:{{ ironic_ipxe_port }}"
ironic_tftp_server_address: "{{ container_networks['bmaas_address']['address'] }}"
# Enable ironic drivers
ironic_drivers_enabled: # Use PXE boot and IPMItool
- agent_ipmitool
- pxe_ipmitool
- pxe_drac # enables drivers for Dell iDrac interface
# Configure Ironic to use Neutron networking
ironic_enabled_network_interfaces_list: "noop,neutron"
ironic_default_network_interface: neutron
# Enable the default set of cleaning steps
ironic_automated_clean: yes
# Configure the neutron networks that Ironc should use
ironic_neutron_provisioning_network_name: "Ironic_Network"
ironic_neutron_cleaning_network_name: "Ironic_Network"
ironic_neutron_inspection_network_name: "Ironic_Network"
# Ensure ironic API (using uwsgi) listens on br-bmaas for agent callbacks
# as well as the mgmt interface for the loadbalancer
ironic_uwsgi_bind_address: 0.0.0.0
# Add ipa-insecure=1 to kernel parameters
# Needed when ironic endpoint is available over https with self-signed cert.
ironic_kernel_append_params: "ipa-debug=1 systemd.journald.forward_to_console=yes ipa-insecure=1"
# INI file overrides
ironic_ironic_conf_overrides:
# Disable full device erasure (slow) and just metadata erasure, and replace with "Express erasure"
# which tries to use firmware secure-erase command, but if that fails, reverts to metadata erasure.
# See: https://docs.openstack.org/ironic/yoga/admin/cleaning.html#storage-cleaning-options
deploy:
erase_devices_priority: 0
erase_devices_metadata_priority: 0
conductor:
clean_step_priority_override: "deploy.erase_devices_express:5"
# Direct IPA to callback directly to deploying ironic container (via BMAAS network)
# instead of going via HAProxy on mgmt network. Only applies when bmaas network is isolated.
service_catalog:
endpoint_override: "http://{{ container_networks['bmaas_address']['address'] }}:6385"
# Enable ipmitool's Serial-over-LAN terminal console for baremetal nodes
DEFAULT:
enabled_console_interfaces: "ipmitool-socat,no-console"
## IRONIC INSPECTOR ##
# Direct Inspector to callback directly to deploying ironic container (via BMAAS network)
# instead of going via HAProxy on mgmt network. Only applies when bmaas network is isolated.
ironic_inspector_callback_url: "{{ ironic_inspector_service_internaluri_proto }}://{{ container_networks['bmaas_address']['address'] }}:{{ ironic_inspector_service_port }}/v1/continue"
# Add ipa-insecure=1 to kernel parameters
# Needed when inspector is available over https with self-signed cert.
ironic_inspector_extra_callback_parameters: "ipa-collect-lldp=1 ipa-insecure=1"
# Ensure inspector API (using uwsgi) listens on br-bmaas for agent callbacks
# as well as the mgmt interface for the loadbalancer
ironic_inspector_service_address: "0.0.0.0"
# dnsmasq/dhcp information for inspector
ironic_inspector_dhcp_pool_range: 10.88.104.96 10.88.104.127
ironic_inspector_dhcp_subnet: 10.88.104.0/24
ironic_inspector_dhcp_subnet_mask: 255.255.255.0
ironic_inspector_dhcp_enable_gateway: False
ironic_inspector_dhcp_enable_nameservers: False
ironic_inspector_dhcp_interface: eth15 # connected to br-bmaas on the host
Deploy Neutron changes¶
openstack-ansible playbooks/os-neutron-install.yml
Deploy the ironic-specific nova services¶
This deploys nova compute and nova console services to the ironic compute containers.
playbooks/os-nova-install.yml --limit ironic_all
Deploy changes to HAProxy¶
This will bring up the required Ironic, Inspector, and console endpoints.
openstack-ansible playbooks/haproxy-install.yml --tags haproxy_server-config
Deploy the Ironic and Inspector services¶
openstack-ansible playbooks/os-ironic-install.yml
Deploy the Horizon dashbaords for Ironic¶
openstack-ansible playbooks/os-horizon-install.yml
Using Ironic¶
Please refer to the general instructions in the Configuring Ironic section of this documentation.
VXLAN project networks¶
In this example Ironic multitenancy is implemented using VLANs. In an OpenStack deployment where project networks are implemented using an overlay such as VXLAN, it will not be possible to attach Ironic nodes directly to these networks. In addition, it is not possible for an end user to request that the underlying implementation is VLAN when creating a project network.
In a cloud using overlay project networks it will be necessary for the cloud administrator to create VLAN provider networks for users to attach Ironic nodes to and to share these into individual projects using Neutron RBAC.