2023.2 Series (9.5.0 - 9.7.x) Release Notes

9.7.2

Upgrade Notes

  • Deployers implementing their own HardwareManagers must to audit their code for unsafe uses of qemu-img and related methods.

Security Issues

  • Ironic-Python-Agent now checks any supplied image format value against the detected format of the image file and will prevent deployments should the values mismatch.

  • Images previously misconfigured as raw despite being in another format, in some non-default configurations, may have been mistakenly converted if needed. Ironic-Python-Agent will no longer perform conversion in any case for images with metadata indicating in raw format.

  • Ironic-Python-Agent always inspects any non-raw user image content for safety before running any qemu-based utilities on the image. This is utilized to identify the format of the image and to verify the overall safety of the image. Any images with unknown or unsafe feature uses are explicitly rejected. This can be disabled in both IPA and Ironic by setting [conductor]disable_deep_image_inspection to True for the Ironic deployment. Image inspection is the primary mitigation for CVE-2024-44082 being tracked in bug 2071740. Operators may desire to set [conductor]conductor_always_validates_images on Ironic conductors to mitigate the issue before they have upgraded their Ironic-Python-Agent.

  • Ironic-Python-Agent now explicitly enforces a list of permitted image types for deployment, defaulting to “raw” and “qcow2”. Other image types may work, but are not explicitly supported and must be enabled. This can be modified by setting [conductor]permitted_image_formats for all Ironic services.

Bug Fixes

  • Fixes an issue where configuration drive volumes which are mounted by the operating system could remain mounted and cause a lock to be held, which may conflict with actions such as rebuild. The agent now always makes sure the folder used by Glean and Cloud-init is not mounted.

  • Fixes multiple issues in the handling of images as it related to execution of the qemu-img utility. When using this utility to convert an unsafe image, a malicious user can extract information from a node while Ironic-Python-Agent is deploying or converting an image. Ironic-Python-Agent now inspects all non-raw images for safety, and never runs qemu-based utilities on raw images. This fix is tracked as CVE-2024-44082 and bug 2071740.

  • Images with metadata indicating a “raw” disk format may have been transparently converted from another format. Now, these images will have their exact contents imaged to disk without modification.

  • Fixes bug 2066308, an issue where Ironic Python Agent would call evaluate_hardware_support multiple times on hardware manager plugins. Scanning for hardware and disks is time consuming, and caused timeouts on badly-performing nodes.

9.7.1

Bug Fixes

  • Fixes a failure case where downloads would not be retried when the checksum fails verification. the agent now includes the checksum activity as part of the file download operation, and will automatically retry downloads when the checksum fails in accordance with the existing download retry logic. This is largely in response to what appears to be intermittent transport failures at lower levels which we cannot otherwise detect.

  • Fixes missing Content-Type header when sending inspection data back to ironic-inspector or ironic. While ironic-inspector tolerates the missing header, it may cause issues with the new inspection implementation.

  • The default timeout value for the agent to lookup itself in an Ironic deployment has been extended to 600 seconds from 300 seconds. This is to provide better stability for Ironic deployments under heavy load which may be unable to service new requests. This is particularly true when the backing database is SQLite for Ironic due to the limited write concurrency of the database.

  • Fixes referencing to raid_device variable before assignment, is replaced by blk variable.

  • Inspection is now retried on HTTP 409 (conflict), which can be returned by the new implementation in Ironic.

  • Fixes the post data to inspector to retry in 50X errors.

  • The error handling of the multipathd service startup/discovery process. IPA handles both scenario when the multipathd service is already started and the scenario when the service has not been started and in the second scenario IPA will try to start the service. IPA is not pre checking whether multipathd is running already or not, it will start the multipathd service even if it is already running and expects 0 error code . It has been noticed that with certain combinations of Linux distros and multipathd versions the error code is not 0 when IPA tries to start multipathd in case an instance of multipathd is already running. When the expected return code is not 0 an exception will be thrown and that will cause the multipath device discovery to terminate prematurely and if the selected root device is a multipath device then IPA won’t be able to provision. This fix discards the exception that is caused by the non 0 error code returned by the multipathd startup process. In case there is a genuine issue with the multipath service, that would be caught when the actual multipath device listing command is executed (multipath -ll).

  • Fixes an issue with rebuilding instances on Software RAID with RAIDed ESP partitions.

9.7.0

New Features

  • Adds a new service extension which facilitates command handling for Ironic to retrieve a list of service steps.

  • Adds a new base method to base HardwareManager, get_service_steps which works the same as get_clean_steps and get_deploy_steps. These methods can be extended by hardware managers to permit them to signal what steps are permitted.

  • Extends reasonable deploy/clean steps to also be service steps which are embedded in the Ironic agent. For example, CPU, Network, and Memory burnin steps are available as service steps, but not the disk burnin step as that would likely result in the existing disk contents being damaged.

Bug Fixes

  • Fixes a failure case where a deployed instance may be unable to access the configuration drive post-deployment. This can occur when block devices only support 4KB IO interactions. When 4KB block IO sizes are in use, the ISO9660 filesystem driver in Linux cannot be used as it is modeled around a 2KB block. We now attempt to verify, and rebuild the configuration drive on a FAT filesystem when we cannot mount the supplied configuration drive. Operators can force the agent to write configuration drives using the FAT filesystem using the [DEFAULT]config_drive_rebuild option.

  • Fixes, or at least lessens the case where a running Ironic agent can stack up numerous lookup requests against an Ironic deployment when a node is locked. In particular, this is beause the lookup also drives generation of the agent token, which requires the conductor to allocate a worker, and generate the token, and return the result to the API client. Ironic’s retry logic will now wait up to 60 seconds, and if an HTTP Conflict (409) message is received, the agent will automatically pause lookup operations for thirty seconds as opposed continue to attempt lookups which could create more work for the Ironic deployment needlessly.

9.6.0

Bug Fixes

  • Fixes timeout declarations for Bandit 1.7.5 rule additions.

  • Adds a new configuration option http_request_timeout to allow for operators to set the amount of time to wait for a new request socket to wait. This helps prevent prevent a possible hanged connection should the initial packets be lost in tranist.

  • Fixes the nvidia hardware manager firmware upgrade support to permit URLs with an “https” schema.

Other Notes

  • Adds a new configuration option http_request_timeout which is also accessible utilizing the kernel command line option ipa-http-request-timeout. This option helps prevent failed connections from hanging the agent. The default is 30 seconds.

  • The agent now logs the size of data transferred when downloading images, which can be helpful in troubleshooting image download issues.

9.5.0

New Features

  • The hardware inventory now contains information about the system firmware: vendor, version and the build date.

  • The ironic-python-agent will now attempt to determine a checksum type by evaluating the length of the supplied checksum. This allows SHA512 (SHA-2) and SHA256 (SHA-2) checksums to be identified and utilized without an explicit declaration of the checksum type utilizing the os_hash_algo value.

  • Improved parsing of checksum files.

    • Added support for the ALGORITHM (FILENAME) = CHECKSUM format used by CentOS Stream.

    • Lines starting with # are ignored as comments.

    • If checksum file contain only the checksum itself, the content is validated to ensure it is one of the known checksum types.

  • Adds a new inspection collector lldp that collects LLDP information into the lldp_raw field.

  • The hardware inventory now contains supported network interface speed in Mbit/s.

Deprecation Notes

  • Support for MD5 checksums have been deprecated and will be removed after the 2024 Release.

  • The LLDP information as part of the general inventory is deprecated. Use the new lldp inspection collector to retrieve it.

  • The ipa-collect-lldp kernel parameter and the corresponding option are now deprecated.

Bug Fixes

  • Fixes UEFI NVRAM record handling with efibootmgr so we can accept and handle UTF-16 encoded data which is to be expected in UEFI NVRAM as the records are UTF-16 encoded.

  • Fixes handling of UEFI NVRAM records to allow for unexpected characters in the response, so it is non-fatal to Ironic.