ironic.common.image_format_inspector module

This is a python implementation of virtual disk format inspection routines gathered from various public specification documents, as well as qemu disk driver code. It attempts to store and parse the minimum amount of data required, and in a streaming-friendly manner to collect metadata about complex-format images.

class ironic.common.image_format_inspector.CaptureRegion(offset, length)

Bases: object

Represents a region of a file we want to capture.

A region of a file we want to capture requires a byte offset into the file and a length. This is expected to be used by a data processing loop, calling capture() with the most recently-read chunk. This class handles the task of grabbing the desired region of data across potentially multiple fractional and unaligned reads.

Parameters:

• offset – Byte offset into the file starting the region

• length – The length of the region

capture(chunk, current_position)

Process a chunk of data.

This should be called for each chunk in the read loop, at least until complete returns True.

Parameters:

• chunk – A chunk of bytes in the file

• current_position – The position of the file processed by the read loop so far. Note that this will be the position in the file after the chunk being presented.

property complete

Returns True when we have captured the desired data.

class ironic.common.image_format_inspector.FileInspector(tracing=False)

Bases: object

A stream-based disk image inspector.

This base class works on raw images and is subclassed for more complex types. It is to be presented with the file to be examined one chunk at a time, during read processing and will only store as much data as necessary to determine required attributes of the file.

property actual_size

Returns the total size of the file, usually smaller than virtual_size.

NOTE: this will only be accurate if the entire file is read and processed.

property complete

Returns True if we have all the information needed.

property context_info

Return info on amount of data held in memory for auditing.

This is a dict of region:sizeinbytes items that the inspector uses to examine the file.

eat_chunk(chunk)

Call this to present chunks of the file to the inspector.

property format_match

Returns True if the file appears to be the expected format.

classmethod from_file(filename)

Read as much of a file as necessary to complete inspection.

NOTE: Because we only read as much of the file as necessary, the actual_size property will not reflect the size of the file, but the amount of data we read before we satisfied the inspector.

Raises ImageFormatError if we cannot parse the file.

has_region(name)

Returns True if named region has been defined.

new_region(name, region)

Add a new CaptureRegion by name.

post_process()

Post-read hook to process what has been read so far.

This will be called after each chunk is read and potentially captured by the defined regions. If any regions are defined by this call, those regions will be presented with the current chunk in case it is within one of the new regions.

region(name)

Get a CaptureRegion by name.

safety_check()

Perform some checks to determine if this file is safe.

Returns True if safe, False otherwise. It may raise ImageFormatError if safety cannot be guaranteed because of parsing or other errors.

property virtual_size

Returns the virtual size of the disk image, or zero if unknown.

class ironic.common.image_format_inspector.ISOInspector(*a, **k)

Bases: FileInspector

ISO 9660 and UDF format

we need to check the first 32KB + descriptor size to look for the ISO 9660 or UDF signature.

http://wiki.osdev.org/ISO_9660 http://wiki.osdev.org/UDF mkisofs –help | grep udf

The Universal Disc Format or UDF is the filesystem used on DVDs and Blu-Ray discs.UDF is an extension of ISO 9660 and shares the same header structure and initial layout.

Like the CDFS(ISO 9660) file system, the UDF file system uses a 2048 byte sector size, and it designates that the first 16 sectors can be used by the OS to store proprietary data or boot logic.

That means we need to check the first 32KB + descriptor size to look for the ISO 9660 or UDF signature. both formats have an extent based layout, so we can’t determine ahead of time where the descriptor will be located.

fortunately, the ISO 9660 and UDF formats have a Primary Volume Descriptor located at the beginning of the image, which contains the volume size.

property format_match

Returns True if the file appears to be the expected format.

property virtual_size

Returns the virtual size of the disk image, or zero if unknown.

exception ironic.common.image_format_inspector.ImageFormatError

Bases: Exception

An unrecoverable image format error that aborts the process.

class ironic.common.image_format_inspector.InfoWrapper(source, fmt)

Bases: object

A file-like object that wraps another and updates a format inspector.

This passes chunks to the format inspector while reading. If the inspector fails, it logs the error and stops calling it, but continues proxying data from the source to its user.

close()

read(size)

class ironic.common.image_format_inspector.QEDInspector(tracing=False)

Bases: FileInspector

property format_match

Returns True if the file appears to be the expected format.

safety_check()

Perform some checks to determine if this file is safe.

Returns True if safe, False otherwise. It may raise ImageFormatError if safety cannot be guaranteed because of parsing or other errors.

class ironic.common.image_format_inspector.QcowInspector(*a, **k)

Bases: FileInspector

QEMU QCOW2 Format

This should only require about 32 bytes of the beginning of the file to determine the virtual size, and 104 bytes to perform the safety check.

BF_OFFSET = 8

BF_OFFSET_LEN = 8

I_FEATURES = 72

I_FEATURES_DATAFILE_BIT = 3

I_FEATURES_LEN = 8

I_FEATURES_MAX_BIT = 4

property format_match

Returns True if the file appears to be the expected format.

property has_backing_file

property has_data_file

property has_header

property has_unknown_features

safety_check()

Perform some checks to determine if this file is safe.

Returns True if safe, False otherwise. It may raise ImageFormatError if safety cannot be guaranteed because of parsing or other errors.

property virtual_size

Returns the virtual size of the disk image, or zero if unknown.

class ironic.common.image_format_inspector.TraceDisabled

Bases: object

A logger-like thing that swallows tracing when we do not want it.

debug(*a, **k)

error(*a, **k)

info(*a, **k)

warning(*a, **k)

class ironic.common.image_format_inspector.VDIInspector(*a, **k)

Bases: FileInspector

VirtualBox VDI format

This only needs to store the first 512 bytes of the image.

property format_match

Returns True if the file appears to be the expected format.

property virtual_size

Returns the virtual size of the disk image, or zero if unknown.

class ironic.common.image_format_inspector.VHDInspector(*a, **k)

Bases: FileInspector

Connectix/MS VPC VHD Format

This should only require about 512 bytes of the beginning of the file to determine the virtual size.

property format_match

Returns True if the file appears to be the expected format.

property virtual_size

Returns the virtual size of the disk image, or zero if unknown.

class ironic.common.image_format_inspector.VHDXInspector(*a, **k)

Bases: FileInspector

MS VHDX Format

This requires some complex parsing of the stream. The first 256KiB of the image is stored to get the header and region information, and then we capture the first metadata region to read those records, find the location of the virtual size data and parse it. This needs to store the metadata table entries up until the VDS record, which may consist of up to 2047 32-byte entries at max. Finally, it must store a chunk of data at the offset of the actual VDS uint64.

METAREGION = '8B7CA206-4790-4B9A-B8FE-575F050F886E'

VHDX_METADATA_TABLE_MAX_SIZE = 65536

VIRTUAL_DISK_SIZE = '2FA54224-CD1B-4876-B211-5DBED83BF4B8'

property format_match

Returns True if the file appears to be the expected format.

post_process()

Post-read hook to process what has been read so far.

This will be called after each chunk is read and potentially captured by the defined regions. If any regions are defined by this call, those regions will be presented with the current chunk in case it is within one of the new regions.

property virtual_size

Returns the virtual size of the disk image, or zero if unknown.

class ironic.common.image_format_inspector.VMDKInspector(*a, **k)

Bases: FileInspector

vmware VMDK format (monolithicSparse and streamOptimized variants only)

This needs to store the 512 byte header and the descriptor region which should be just after that. The descriptor region is some variable number of 512 byte sectors, but is just text defining the layout of the disk.

DESC_MAX_SIZE = 1048575

DESC_OFFSET = 512

GD_AT_END = 18446744073709551615

property format_match

Returns True if the file appears to be the expected format.

post_process()

Post-read hook to process what has been read so far.

This will be called after each chunk is read and potentially captured by the defined regions. If any regions are defined by this call, those regions will be presented with the current chunk in case it is within one of the new regions.

safety_check()

Perform some checks to determine if this file is safe.

Returns True if safe, False otherwise. It may raise ImageFormatError if safety cannot be guaranteed because of parsing or other errors.

property virtual_size

Returns the virtual size of the disk image, or zero if unknown.

ironic.common.image_format_inspector.chunked_reader(fileobj, chunk_size=512)

ironic.common.image_format_inspector.detect_file_format(filename)

Attempts to detect the format of a file.

This runs through a file one time, running all the known inspectors in parallel. It stops reading the file once all of them matches or all of them are sure they don’t match.

Parameters:

filename – The path to the file to inspect.

Returns:

A FormatInspector instance matching the file.

Raises:

ImageFormatError if multiple formats are detected.

ironic.common.image_format_inspector.get_inspector(format_name)

Returns a FormatInspector class based on the given name.

Parameters:

format_name – The name of the disk_format (raw, qcow2, etc).

Returns:

A FormatInspector or None if unsupported.