Infrastructure VMs

Kayobe can deploy infrastructure VMs to the seed-hypervisor. These can be used to provide supplementary services that do not run well within a containerised environment or are dependencies of the control plane.

Configuration

To deploy an infrastructure VM, add a new host to the the infra-vms group in the inventory:

$KAYOBE_CONFIG_PATH/inventory/infra-vms

 [infra-vms]
 an-example-vm

The configuration of the virtual machine should be done using host_vars. These override the group_vars defined for the infra-vms group. Most variables have sensible defaults defined, but there are a few variables which must be set.

Mandatory variables

All networks must have an interface defined, as described in Per-host Network Configuration. By default the VMs are attached to the admin overcloud network. If, for example, admin_oc_net_name was set to example_net, you would need to define example_net_interface. It is possible to change the list of networks that a VM is attached to by modifying infra_vm_network_interfaces. Additional interfaces can be added by setting infra_vm_network_interfaces_extra.

List of Kayobe applied defaults to required docker_container variables. Any of these variables can be overridden with a host_var.

---
###############################################################################
# Infrastructure VM configuration.

# Name of the infra VM.
infra_vm_name: "{{ inventory_hostname }}"

# Memory in MB.
infra_vm_memory_mb: "{{ 16 * 1024 }}"

# Number of vCPUs.
infra_vm_vcpus: 4

# List of volumes.
infra_vm_volumes:
  - "{{ infra_vm_root_volume }}"
  - "{{ infra_vm_data_volume }}"

# Root volume.
infra_vm_root_volume:
  name: "{{ infra_vm_name }}-root"
  pool: "{{ infra_vm_pool }}"
  capacity: "{{ infra_vm_root_capacity }}"
  format: "{{ infra_vm_root_format }}"
  image: "{{ infra_vm_root_image }}"

# Data volume.
infra_vm_data_volume:
  name: "{{ infra_vm_name }}-data"
  pool: "{{ infra_vm_pool }}"
  capacity: "{{ infra_vm_data_capacity }}"
  format: "{{ infra_vm_data_format }}"

# Name of the storage pool for the infra VM volumes.
infra_vm_pool: default

# Capacity of the infra VM root volume.
infra_vm_root_capacity: 50G

# Format of the infra VM root volume.
infra_vm_root_format: qcow2

# Base image for the infra VM root volume. Default is
# "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img"
# when os_distribution is "ubuntu", or
# https://dl.rockylinux.org/pub/rocky/8/images/x86_64/Rocky-8-GenericCloud.latest.x86_64.qcow2
# when os_distribution is "rocky",
# or
# "https://cloud.centos.org/centos/8-stream/x86_64/images/CentOS-Stream-GenericCloud-8-20220913.0.x86_64.qcow2"
# otherwise.
infra_vm_root_image: >-
  {%- if os_distribution == 'ubuntu' %}
  https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img
  {%- elif os_distribution == 'rocky' %}
  https://dl.rockylinux.org/pub/rocky/8/images/x86_64/Rocky-8-GenericCloud.latest.x86_64.qcow2
  {%- else -%}
  https://cloud.centos.org/centos/8-stream/x86_64/images/CentOS-Stream-GenericCloud-8-20220913.0.x86_64.qcow2
  {%- endif %}

# Capacity of the infra VM data volume.
infra_vm_data_capacity: 100G

# Format of the infra VM data volume.
infra_vm_data_format: qcow2

# List of network interfaces to attach to the infra VM.
infra_vm_interfaces: "{{ network_interfaces | sort | map('net_libvirt_vm_network') | list }}"

# Hypervisor that the VM runs on.
infra_vm_hypervisor: "{{ groups['seed-hypervisor'] | first }}"

# Customise ansible_ssh_extra_args for the test that checks SSH connectivity
# after provisioning. Defaults to disabling ssh host key checking.
infra_vm_wait_connection_ssh_extra_args: '-o StrictHostKeyChecking=no'

# OS family. Needed for config drive generation.
infra_vm_os_family: "{{ 'RedHat' if os_distribution == 'centos' else 'Debian' }}"

###############################################################################
# Infrastructure VM node configuration.

# User with which to access the infrastructure vm via SSH during bootstrap, in
# order to setup the Kayobe user account. Default is {{ os_distribution }}.
infra_vm_bootstrap_user: "{{ os_distribution }}"

###############################################################################
# Infrastructure VM network interface configuration.

# List of networks to which infrastructure vm nodes are attached.
infra_vm_network_interfaces: >
  {{ (infra_vm_default_network_interfaces +
      infra_vm_extra_network_interfaces) | select | unique | list }}

# List of default networks to which infrastructure vm nodes are attached.
infra_vm_default_network_interfaces: >
  {{ [admin_oc_net_name] | select | unique | list }}

# List of extra networks to which infrastructure vm nodes are attached.
infra_vm_extra_network_interfaces: []

###############################################################################
# Infrastructure VM node software RAID configuration.

# List of software RAID arrays. See mrlesmithjr.mdadm role for format.
infra_vm_mdadm_arrays: []

###############################################################################
# Infrastructure VM node encryption configuration.

# List of block devices to encrypt. See stackhpc.luks role for format.
infra_vm_luks_devices: []

###############################################################################
# Infrastructure VM node LVM configuration.

# List of infrastructure vm volume groups. See mrlesmithjr.manage_lvm role for
# format.
infra_vm_lvm_groups: "{{ infra_vm_lvm_groups_default + infra_vm_lvm_groups_extra }}"

# Default list of infrastructure vm volume groups. See mrlesmithjr.manage_lvm
# role for format.
infra_vm_lvm_groups_default: "{{ [infra_vm_lvm_group_data] if infra_vm_lvm_group_data_enabled | bool else [] }}"

# Additional list of infrastructure vm volume groups. See mrlesmithjr.manage_lvm
# role for format.
infra_vm_lvm_groups_extra: []

# Whether a 'data' LVM volume group should exist on the infrastructure vm. By
# default this contains a 'docker-volumes' logical volume for Docker volume
# storage. It will also be used for Docker container and image storage if
# 'docker_storage_driver' is set to 'devicemapper'. Default is true if
# 'docker_storage_driver' is set to 'devicemapper', or false otherwise.
infra_vm_lvm_group_data_enabled: "{{ docker_storage_driver == 'devicemapper' }}"

# Infrastructure VM LVM volume group for data. See mrlesmithjr.manage_lvm role
# for format.
infra_vm_lvm_group_data:
  vgname: data
  disks: "{{ infra_vm_lvm_group_data_disks }}"
  create: True
  lvnames: "{{ infra_vm_lvm_group_data_lvs }}"

# List of disks for use by infrastructure vm LVM data volume group. Default to
# an invalid value to require configuration.
infra_vm_lvm_group_data_disks:
  - changeme

# List of LVM logical volumes for the data volume group.
infra_vm_lvm_group_data_lvs:
  - "{{ infra_vm_lvm_group_data_lv_docker_volumes }}"

# Docker volumes LVM backing volume.
infra_vm_lvm_group_data_lv_docker_volumes:
  lvname: docker-volumes
  size: "{{ infra_vm_lvm_group_data_lv_docker_volumes_size }}"
  create: True
  filesystem: "{{ infra_vm_lvm_group_data_lv_docker_volumes_fs }}"
  mount: True
  mntp: /var/lib/docker/volumes

# Size of docker volumes LVM backing volume.
infra_vm_lvm_group_data_lv_docker_volumes_size: 75%VG

# Filesystem for docker volumes LVM backing volume. ext4 allows for shrinking.
infra_vm_lvm_group_data_lv_docker_volumes_fs: ext4

###############################################################################
# Infrastructure VM node sysctl configuration.

# Dict of sysctl parameters to set.
infra_vm_sysctl_parameters: {}

###############################################################################
# Infrastructure VM node tuned configuration.

# Builtin tuned profile to use. Format is same as that used by giovtorres.tuned
# role. Default is virtual-guest.
infra_vm_tuned_active_builtin_profile: "virtual-guest"

###############################################################################
# Infrastructure VM node user configuration.

# List of users to create. This should be in a format accepted by the
# singleplatform-eng.users role.
infra_vm_users: "{{ users_default }}"

###############################################################################
# Infrastructure VM node firewalld configuration.

# Whether to install and enable firewalld.
infra_vm_firewalld_enabled: false

# A list of zones to create. Each item is a dict containing a 'zone' item.
infra_vm_firewalld_zones: []

# A firewalld zone to set as the default. Default is unset, in which case the
# default zone will not be changed.
infra_vm_firewalld_default_zone:

# A list of firewall rules to apply. Each item is a dict containing arguments
# to pass to the firewalld module. Arguments are omitted if not provided, with
# the following exceptions:
# - offline: true
# - permanent: true
# - state: enabled
infra_vm_firewalld_rules: []

Customisations

Examples of common customisations are shown below.

By default the Ansible inventory name is used as the name of the VM. This may be overridden via infra_vm_name:

$KAYOBE_CONFIG_PATH/inventory/host_vars/an-example-vm

 # Name of the infra VM.
 infra_vm_name: "the-special-one"

By default the VM has 16G of RAM. This may be changed via infra_vm_memory_mb:

$KAYOBE_CONFIG_PATH/inventory/host_vars/an-example-vm

 # Memory in MB. Defaults to 16GB.
 infra_vm_memory_mb: "{{ 8 * 1024 }}"

The default network configuration attaches infra VMs to the admin network. If this is not appropriate, modify infra_vm_network_interfaces. At a minimum the network interface name for the network should be defined.

$KAYOBE_CONFIG_PATH/inventory/host_vars/an-example-vm

 # Network interfaces that the VM is attached to.
 infra_vm_network_interfaces:
   - aio

 # Mandatory: All networks must have an interface defined.
 aio_interface: eth0

 # By default kayobe will connect to a host via ``admin_oc_net``.
 # As we have not attached this VM to this network, we must override
 # ansible_host.
 ansible_host: "{{ 'aio' | net_ip }}"

Configuration for all VMs can be set using extra_vars defined in $KAYOBE_CONFIG_PATH/infra-vms.yml. Note that normal Ansible precedence rules apply and the variables will override any host_vars. If you need to override the defaults, but still maintain per-host settings, use group_vars instead.

Deploying the virtual machine

Once the initial configuration has been done follow the steps in Infrastructure VMs.