Driver BDM Data Structures¶
In addition to the API BDM data format there are also several internal data structures within Nova that map out how block devices are attached to instances. This document aims to outline the two general data structures and two additional specific data structures used by the libvirt virt driver.
Note
This document is based on an email to the openstack-dev mailing list by Matthew Booth below provided as a primer for developers working on virt drivers and interacting with these data structures.
http://lists.openstack.org/pipermail/openstack-dev/2016-June/097529.html
Note
References to local disks in the following document refer to any disk directly managed by nova compute. If nova is configured to use RBD or NFS for instance disks then these disks won’t actually be local, but they are still managed locally and referred to as local disks. As opposed to RBD volumes provided by Cinder that are not considered local.
Generic BDM data structures¶
BlockDeviceMapping
¶
The ‘top level’ data structure is the BlockDeviceMapping
(BDM) object. It
is a NovaObject
, persisted in the DB. Current code creates a BDM object for
every disk associated with an instance, whether it is a volume or not.
The BDM object describes properties of each disk as specified by the user. It is initially from a user request, for more details on the format of these requests please see the Block Device Mapping in Nova document.
The Compute API transforms and consolidates all BDMs to ensure that all disks,
explicit or implicit, have a BDM, and then persists them. Look in
nova.objects.block_device
for all BDM fields, but in essence they contain
information like (source_type=’image’, destination_type=’local’,
image_id=’<image uuid’>), or equivalents describing ephemeral disks, swap disks
or volumes, and some associated data.
Note
BDM objects are typically stored in variables called bdm
with lists
in bdms
, although this is obviously not guaranteed (and unfortunately
not always true: bdm
in libvirt.block_device
is usually a
DriverBlockDevice
object). This is a useful reading aid (except when
it’s proactively confounding), as there is also something else typically
called block_device_mapping
which is not a BlockDeviceMapping
object.
block_device_info
¶
Changed in version 24.0.0: (Xena)
The legacy block_device_info format is no longer supported.
Drivers do not directly use BDM objects. Instead, they are transformed into a
different driver-specific representation. This representation is normally
called block_device_info
, and is generated by
virt.driver.get_block_device_info()
. Its output is based on data in BDMs.
block_device_info
is a dict containing:
root_device_name
Hypervisor’s notion of the root device’s name
ephemerals
A list of all ephemeral disks
block_device_mapping
A list of all cinder volumes
swap
A swap disk, or None if there is no swap disk
Note
The disks were previously represented in one of two ways, depending on the specific driver in use. A legacy plain dict format or the currently used DriverBlockDevice format discussed below. Support for the legacy format was removed in Xena.
Disks are represented by subclasses of nova.block_device.DriverBlockDevice
.
These subclasses retain a reference to the underlying BDM object. This means
that by manipulating the DriverBlockDevice
object, the driver is able to
persist data to the BDM object in the DB.
Note
Common usage is to pull block_device_mapping
out of this
dict into a variable called block_device_mapping
. This is not a
BlockDeviceMapping
object, or a list of them.
Note
If block_device_info
was passed to the driver by compute manager, it
was probably generated by _get_instance_block_device_info()
.
By default, this function filters out all cinder volumes from
block_device_mapping
which don’t currently have connection_info
.
In other contexts this filtering will not have happened, and
block_device_mapping
will contain all volumes.
Note
Unlike BDMs, block_device_info
does not currently represent all
disks that an instance might have. Significantly, it will not contain any
representation of an image-backed local disk, i.e. the root disk of a
typical instance which isn’t boot-from-volume. Other representations used
by the libvirt driver explicitly reconstruct this missing disk.
libvirt driver specific BDM data structures¶
instance_disk_info
¶
The virt driver API defines a method get_instance_disk_info
, which returns
a JSON blob. The compute manager calls this and passes the data over RPC
between calls without ever looking at it. This is driver-specific opaque data.
It is also only used by the libvirt driver, despite being part of the API for
all drivers. Other drivers do not return any data. The most interesting aspect
of instance_disk_info
is that it is generated from the libvirt XML, not
from nova’s state.
Note
instance_disk_info
is often named disk_info
in code, which
is unfortunate as this clashes with the normal naming of the next
structure. Occasionally the two are used in the same block of code.
Note
RBD disks (including non-volume disks) and cinder volumes
are not included in instance_disk_info
.
instance_disk_info
is a list of dicts for some of an instance’s disks. Each
dict contains the following:
type
libvirt’s notion of the disk’s type
path
libvirt’s notion of the disk’s path
virt_disk_size
The disk’s virtual size in bytes (the size the guest OS sees)
backing_file
libvirt’s notion of the backing file path
disk_size
The file size of path, in bytes.
over_committed_disk_size
As-yet-unallocated disk size, in bytes.
disk_info
¶
Note
As opposed to instance_disk_info
, which is frequently called
disk_info
.
This data structure is actually described pretty well in the comment block at
the top of nova.virt.libvirt.blockinfo
. It is internal to the libvirt
driver. It contains:
disk_bus
The default bus used by disks
cdrom_bus
The default bus used by cdrom drives
mapping
Defined below
mapping
is a dict which maps disk names to a dict describing how that disk
should be passed to libvirt. This mapping contains every disk connected to the
instance, both local and volumes.
First, a note on disk naming. Local disk names used by the libvirt driver are well defined. They are:
disk
The root disk
disk.local
The flavor-defined ephemeral disk
disk.ephX
Where X is a zero-based index for BDM defined ephemeral disks
disk.swap
The swap disk
disk.config
The config disk
These names are hardcoded, reliable, and used in lots of places.
In disk_info
, volumes are keyed by device name, eg ‘vda’, ‘vdb’. Different
buses will be named differently, approximately according to legacy Linux
device naming.
Additionally, disk_info
will contain a mapping for ‘root’, which is the
root disk. This will duplicate one of the other entries, either ‘disk’ or a
volume mapping.
Each dict within the mapping
dict contains the following 3 required fields
of bus, dev and type with two optional fields of format and boot_index
:
bus
:The guest bus type (‘ide’, ‘virtio’, ‘scsi’, etc)
dev
:The device name ‘vda’, ‘hdc’, ‘sdf’, ‘xvde’ etc
type
:Type of device eg ‘disk’, ‘cdrom’, ‘floppy’
format
Which format to apply to the device if applicable
boot_index
Number designating the boot order of the device
Note
BlockDeviceMapping
and DriverBlockDevice
store boot index
zero-based. However, libvirt’s boot index is 1-based, so the value stored
here is 1-based.
Todo
Add a section for the per disk disk.info
file within instance
directory when using the libvirt driver.