Configure PCIe Hot-Plug for Instances¶

About this task

PCIe-native machine types (Q35 and ARM virt) require pre-provisioned root ports to support hot-plugging devices into running instances. This topic describes how to configure Nova to reserve additional PCIe root ports so that instances can support multiple hot-plug operations.

Each hot-plugged device occupies a pcie-root-port controller slot. These slots are defined in the instance domain XML at creation and you cannot add them dynamically afterward. By default, libvirt reserves only 1-2 spare slots, which limits the number of devices you can hot-plug after boot.

This limitation applies to the following PCIe-native machine types:

Q35 family — all versioned variants (pc-q35-7.2, pc-q35-8.0, etc.)
ARM virt — the standard machine type for aarch64 and armv7 guests. Nova uses virt by default for ARM images that do not explicitly set hw_machine_type.

Note

This limitation does not apply to pc/i440fx (the default x86 machine type), which uses ACPI PCI hot-plug with approximately 31 available slots and does not require any pre-provisioning.

For more information, see QEMU documentation and Libvirt documentation

Prerequisites

Identify whether your instances use a PCIe-based machine type:
```
~(keystone_admin)]$ openstack image show <image-id> -f json | jq '.properties.hw_machine_type'
```
If the output is q35 or pc-q35-*, then instances launched from that image are subject to PCIe hot-plug slot limitations.

Configure PCIe Root Ports for New Instances ¶

Follow these steps to increase the number of pre-provisioned PCIe root ports for new instances.

Note

Choose the value based on expected hot-plug demand. Each empty root port consumes a PCI bus number and MMIO reservation. Values up to 16 are safe for modern virtio-based workloads (which avoid legacy IO BARs), while excessively large values waste address space.

Recommended value: 16 (suitable for typical workloads with multiple hot-plugged NICs or PCIe devices)
Maximum value: 28 (QEMU/libvirt architectural limit)

Procedure

Update the Nova configuration. Use one of the following options to update:

Option 1: Using a Helm override file

cat << EOF > ./pcie_ports.yaml
conf:
  nova:
    libvirt:
      num_pcie_ports: 16

Option 2: Directly overriding the config

~(keystone_admin)]$ system helm-override-update --reuse-values \
 --set conf.nova.libvirt.num_pcie_ports=16 \
 |prefix|-openstack nova openstack

Apply the updated application:

~(keystone_admin)]$ system application-apply |prefix|-openstack

Newly created instances will have the specified number of PCIe root ports pre-provisioned in their domain XML.

Attach Devices to Existing Instances ¶

The num_pcie_ports setting applies only to newly created instances. Existing instances retain their original root port count. To attach a device to an instance that was created before increasing num_pcie_ports, use a stop/start cycle:

Stop the instance.

~(keystone_admin)]$ openstack server stop <instance>

Attach the interface.

~(keystone_admin)]$ openstack server add port <instance> <port>

Start the instance.

~(keystone_admin)]$ openstack server start <instance>

At startup, Nova regenerates the domain XML and attaches the interface during boot, thereby bypassing the PCIe hot-plug slot limitation.