2023.1 Series Release Notes

2023.1-eom

New Features

  • Added a new cinder-manage command to handle the situation where database purges would not complete due to the volumes table holding references to deleted services. The new command makes sure that all volumes have a reference only to the correct service_uuid, which will allow old service records to be purged from the database.

    Command: cinder-manage volume update_service

  • When Cinder creates a new cinder-volume service, it now also immediately updates the service_uuid for all volumes associated with that cinder-volume host. In some cases, this was preventing the database purge operation from completing successfully.

Upgrade Notes

  • Cinder now uses the RBD trash functionality to handle some volume deletions. Therefore, deployments must either a) enable scheduled RBD trash purging on the RBD backend or b) enable the Cinder RBD driver’s enable_deferred_deletion option to have Cinder purge the RBD trash. This adds the new configuration option ‘rbd_concurrent_flatten_operations’, which limits how many RBD flattens the driver will run simultaneously. This can be used to prevent flatten operations from consuming too much I/O capacity on the Ceph cluster. It defaults to 3.

Bug Fixes

  • Bug #2012246: Hide value of the [coordination] backend_url option from logs because it can contain credential.

  • Bug #2058596: Fixed broken backup_swift_service_auth=True which made swift backup driver consistently fail during object data access.

  • Bug #2031897: Fixed issues for volume backups with the Ceph driver where failures of the first process (“rbd export-diff”) were not caught. Instead, only the return code of the second process (“rbd import-diff”) was recognized.

    This change also preserves the stderr that was lost previously in order to ease debugging.

  • Hitachi driver bug #2072317: Fix potential data-loss due to a network issue during a volume deletion.

  • Hitachi driver bug #2024418: Fixed to raise correct exception when volume is busy while performing delete volume operation.

  • HPE XP and NEC V driver bug #2012515: Fixed to use correct Host group name.

  • Hitachi driver bug #2011810: Fixed to use correct pool number for secondary storage on GAD environment.

  • Hitachi driver bug #2071697 <https://bugs.launchpad.net/cinder/+bug/2071697>’_: Fix to set correct object ID as LDEV nickname when running host-assisted migration with ``retype` or migration commands.

  • Hitachi driver bug #2063317: Fix test scripts to avoid failing by unexpected response from psuedo REST API server

  • HPE 3PAR driver Bug #2068795: Fixed: Perform login before invoking getWsApiVersion

  • Bug #1988942: Increased size of volume image metadata values accepted by the Block Storage API. Volume image metadata values were limited to 255 characters but Glance allows up to 65535 bytes. This change does not affect the database tables which already allow up to 65535 bytes for image metadata values.

  • Pure iSCSI & FC driver bug #2006960: Fixed attaching LUNs greater than 255. Driver leverages new os-brick functionality to specify LUN addressing mode.

  • Bug #1969643: The RBD driver can now delete volumes with other volumes cloned from it (or its snapshots) in cases where deletion would previously fail. This uses the RBD trash functionality.

22.2.0

Security Issues

  • Images in the qcow2 format with an external data file are now rejected with an ImageUnacceptable error because such images could be used in an exploit to expose host information. Given that qcow2 external data files were never supported by Cinder, this change should have no impact on users. See Bug #2059809 for details.

Bug Fixes

  • Ceph backup driver Bug #1895035: Fixed restore full backups to non RBD volumes.

  • Bug #1912624: Corrected regression introduced by the refactoring of the backup service in the ussuri release, which prevented the creation of a volume backup in a different availability zone.

  • Bug #2059809: Fixed issue where a qcow2 format image with an external data file could expose host information. Such an image is now rejected with an ImageUnacceptable error if it is used to create a volume. Given that qcow2 external data files were never supported by Cinder, the only use for such an image previously was to attempt to steal host information, and hence this change should have no impact on users.

  • Dell PowerMax driver bug #2051828: The driver only recognized 10.0 as being Unisphere 10 and would try to use 9.2 for Unisphere 10.x (where x > 0), but now it correctly recognizes 10.x as being Unisphere 10.

  • HPE 3PAR driver Bug #1994521: Fixed: While performing a delete snapshot (s1) operation, the volumes (v2) dependent on the snapshot (s1) are converted to base volumes. This operation fails if these dependent volumes (v2) have their own dependent snapshots (s2). The errors during the failure were vague and not helpful. With this release, we added conditions to fail this operation early and also added useful error message.

  • HPE 3PAR driver Bug #2015034: Added handling for VLAN iscsi IPs in the 3PAR iSCSI driver.

22.1.2

Bug Fixes

  • Dell PowerFlex driver bug #1998136: When using self signed certificates, the option sent to os-brick via the connection_properties was not correctly handled. It has now been fixed by adding the ‘verify_certificate’ and ‘certificate_path’ to the driver when initializing the connection.

  • Bug #2045431: Fixed a data leak scenario where we preserve sparseness when reimaging the volume.

    We currently do a sparse copy when writing an image on the volume. This could be a potential data leak scenario where the zero blocks of the new image are not written on the existing volume and the data from the old image still exists on the volume. We fix the scenario by not doing sparse copy when reimaging the volume.

22.1.1

Bug Fixes

  • Bug #2025277: Fixed a regression in the fix for Cinder backup restoring into sparse volumes, where OpenStack’s integrated CLI triggered a traceback. The deprecated project-specific legacy CLI of Cinder continued to work.

  • Bug #1945500: The original attempt at fixing this bug did not account for differences in how glance and cinder store image metadata, and as a result some image properties were not filtered out. This new improved fix addresses those differences and makes the filtering more thorough.

  • HPE 3PAR driver Bug #2015746: Fixed: minor code changes to work with new wsapi.

  • Bug #1997980: RBD: Fixed failure to update rbd image features for multi-attach when features = 0.

22.1.0

Known Issues

  • For security reasons (Bug #2004555) manually deleting an attachment, manually doing the os-terminate_connection, os-detach or os-force_detach actions will no longer be allowed in most cases unless the request is coming from another OpenStack service on behalf of a user.

Upgrade Notes

  • Nova must be configured to send service tokens and cinder must be configured to recognize at least one of the roles that the nova service user has been assigned in keystone. By default, cinder will recognize the service role, so if the nova service user is assigned a differently named role in your cloud, you must adjust your cinder configuration file (service_token_roles configuration option in the keystone_authtoken section). If nova and cinder are not configured correctly in this regard, detaching volumes will no longer work (Bug #2004555).

Critical Issues

Security Issues

  • As part of the fix for Bug #2004555, cinder now rejects user attachment delete requests for attachments that are being used by nova instances to ensure that no leftover devices are produced on the compute nodes which could be used to access another project’s volumes. Terminate connection, detach, and force detach volume actions (calls that are not usually made by users directly) are, in most cases, not allowed for users.

Bug Fixes

  • Bug #2004555: Fixed issue where a user manually deleting an attachment, calling terminate connection, detach, or force detach, for a volume that is still used by a nova instance resulted in leftover devices on the compute node. These operations will now fail when it is believed to be a problem.

22.0.0

Prelude

Welcome to the 2023.1 (Antelope) release of the OpenStack Block Storage service (cinder). With this release, we added several drivers and driver features as follows:

  • Added HPE XP iSCSI and FC, Fungible NVMe-TCP, NetApp NVMe-TCP storage drivers.

  • Added Features like Trisync replication support for Pure driver, volume group snapshot support for IBM SVF driver, Unisphere 10 support for Dell EMC PowerMax driver, Host assisted migration and retype support for Hitachi VSP driver.

New Features

  • Added NVMe/TCP volume driver for NetApp ONTAP Storage Cluster.

  • Added NVMe-TCP volume driver for Fungible Storage Cluster.

  • Hitachi driver: Support Global-Active Device (GAD) volume. GAD is a one of Hitachi storage fucntion uses volume replication to provide a high-availability environment for hosts across storage systems and sites. New properties will be added in configuration. hbsd:topology sets to active_active_mirror_volumex would specify a GAD volume. hitachi_mirror_xxx parameters would specify a secondary storage for GAD volume.

  • Hitachi driver: Add a config option hitachi_group_name_format for hostgroup name format.

    When using this option, users can specify the name format of host groups or iSCSI targets. Rules of the format:

    • Usable characters are alphanumerics, “.”, “@”, “_”, “:”, “-“, “{” and “}”. “{” and “}” can be used only in variables.

    • The specified value must start with HBSD-.

    • You can use the following variables:

      {wwn}

      FC driver only. This is replaced with the smallest WWPN of the WWPNs of the connecting node.

      {ip}

      iSCSI driver only. This is replaced with the IP address of the connecting node.

      {host}

      This is replaced with the host name of the connecting node.

    • You can use each variable in the specified value no more than once.

    • The specified value must include the following variables:

      • FC driver: {wwn}

      • iSCSI driver: {ip}

    • The maximum length of a specified value is as follows:

      • FC driver: 64

      • iSCSI driver: 32

    • In the length calculation, use the following values as the length of each variable:

      • {wwn}: 16

      • {ip}: 15

      • {host}: 1

    • If the specified value includes {host}, the following rules apply:

      • characters that are not permitted for this parameter, they are replaced with _.

      • If the length of the name after variable replacement exceeds the maximum length of host group (iSCSI target) names, the host name is truncated so that the length of the host groups or iSCSI targets do not exceed the maximum length.

    If you specify this parameter, it is recommended that you specify True for the hitachi_group_create parameter to collect necessary information automatically.

    Examples:

    • FC driver: HBSD-{host}-{wwn}

    • iSCSI driver: HBSD-{host}-{ip}

  • Supported multi-pools for Hitachi driver and OEM storage drivers.

  • Hitachi driver: Support data deduplication and compression, by storage assist. The feature can be worked, if user enable deduplication and compression for the DP-pool, by Configuration Manager REST API, and set the extra spec hbsd:capacity_saving to deduplication_compression

  • Hitachi driver: Additionally support following storages, Hitachi VSP E590, Hitachi VSP E790 and Hitachi VSP E1090.

  • Hitachi driver: Update retype to different pool and support storage assisted migration. Storage assisted migration feature is also used when retype a volume, which doesn’t have any snapshots, to different pool.

  • Added backend driver for HPE XP storage.

  • IBM Spectrum Virtualize Family driver: Added –delete-volumes flag support for delete volumegroup operation. After adding support, the volumes can optionally be deleted when the volume group is deleted.

  • IBM Spectrum Virtualize Family driver: Added storwize_volume_group parameter in the cinder configuration to support volume group feature.

  • IBM Spectrum Virtualize Family driver: Added support for creation and deletion of volumegroup snapshots.

  • IBM Spectrum Virtualize Family driver: Added support for volumegroup for SVC Code Level 8.5.1.0 and above. User can now create, modify and delete volumegroup using the exising cinder CLI for group operations.

  • Infinidat: Added support for storage assisted volume migration within a same InfiniBox host (iSCSI and FC).

  • LVM nvmet target: Added support for new nvmeof connection properties format (version 2). Controlled with nvmeof_conn_info_version configuration option.

  • nvmet target driver: Added support to serve volumes on multiple addresses using the target_secondary_ip_addresses configuration option. This allows os-brick to iterate through them in search of one connection that works, and once os-brick supports NVMe-oF multipathing it will be automatically supported.

    This requires that nvmeof_conn_info_version configuration option is set to 2 as well.

  • nvmet target driver: Added support for shared subsystems/targets using the lvm_share_target configuration option. Defaults to non shared, e.g., each volume has its own subsystem/target.

  • Pure Storage driver: Added replication capability to backend pool information. Response will be `async`, `sync` or```trisync```. `sync` implies support for `async` and `trisync` implies support for `sync` and `async`.

  • Pure Storage driver: Added support for 3-site replication, aka trisync. Requires two replication devices to be created, one async and one sync, plus the addition of new parameters pure_trisync_enabled and pure_trisync_pg_name.

  • RBD driver: Sets the Ceph cluster FSID as the default value for the rbd_secret_uuid configuration option.

  • Dell PowerMax driver now supports Unisphere for PowerMax 10.0

Upgrade Notes

  • IET iSCSI target removed. IET iSCSI target was deprecated in the V release.

  • This release introduces a new configuration option, vmdk_allowed_types, that specifies the list of VMDK image subformats that Cinder will allow. The default setting allows only the ‘streamOptimized’ and ‘monolithicSparse’ subformats, which do not use named extents.

  • The cinder-manage db sync command for this verison of cinder will add additional database indexes. Depending on database size and complexity, this will take time to complete for every single index to be created. On MySQL or MariaDB, these indexes will only be created if an index does not already exist with the same name:

    • groups_deleted_project_id_idx

    • group_snapshots_deleted_project_id_idx

    • volumes_deleted_project_id_idx

    • volumes_deleted_host_idx

    • snapshots_deleted_project_id_idx

    • backups_deleted_project_id_idx

    An example of the SQL commands to generate these indexes can be found in the specific troubleshooting guide.

  • We introduced a new config parameter, reserved_image_namespaces, that allows operators to set the image properties to filter out from volume image metadata by namespace when uploading a volume to Glance. These properties, if not filtered out, cause failures when uploading images back to Glance. The error will happen on Glance side when the reserved namespaces are used. This option is also useful when an operator wants to use the Glance property protections feature to make some image properties read-only.

Deprecation Notes

  • Configuration option iscsi_secondary_ip_addresses is deprecated in favor of target_secondary_ip_addresses to follow the same naming convention of target_ip_address.

Security Issues

  • This release introduces a new configuration option, vmdk_allowed_types, that specifies the list of VMDK image subformats that Cinder will allow in order to prevent exposure of host information by modifying the named extents in a VMDK image. The default setting allows only the ‘streamOptimized’ and ‘monolithicSparse’ subformats, which do not use named extents.

  • As part of the fix for Bug #1996188, cinder is now more strict in checking that the disk_format recorded for an image (as revealed by the Image Service API image-show response) matches what cinder detects when it downloads the image. Thus, some requests to create a volume from a source image that had previously succeeded may fail with an ImageUnacceptable error.

Bug Fixes

  • Bug #2007615: the restore operation of the Cinder backup service now restores into sparse volumes, if possible. So, operators no longer need more space than used previously when they restore from a disaster.

  • Bug #1996049: Fixed bug where backup was not set to error on failure when volume did not exist.

  • Bug #1910767: Fixed the calculation of the allocated capacity for the volume manager. The fix takes into account all volumes that have a host setting, not just volumes with a status of ‘in-use’ or ‘available’.

  • Bug #1952805: Fixed the cinder-backup posix driver’s behavior with multiple backup hosts. Previously cinder-backup would frequently schedule incremental backups on the wrong host and immediately fail.

  • RBD Driver bug #1957073: Fixed snapshot deletion failure when its volume doesn’t exist.

  • PowerStore driver bug #1962824: Fixed Cinder volume caching mechanism for the driver. Now the driver correctly raises exception.SnapshotLimitReached when maximum snapshots are created for a given volume and the volume cache is invalidated to allow a new row of fast volume clones.

  • Bug #1978020: Fixed uploading a volume to a Cinder-backed Glance image; if a store name is set in the volume type’s extra specs, it must also be sent to Glance as part of the new image location URI. Please note that while the image_service:store_id extra spec is validated when it is set for the volume type, it is not validated later; it is the operator’s responsibility to make sure that the Glance store is not renamed or removed or that the volume types are updated accordingly.

  • IBM Spectrum Virtualize family driver Bug #1978290: Optimize lsmdiskgrp SSH calls in creation of replicated volumes to reduce the computational time.

  • Infinidat Driver bug #1982350: Fixed Infinidat driver multi-attach feature. Added a check if there are multiple attachments to the volume from the same connector and terminate connection only for the last attachment from the corresponding host.

  • Infinidat Driver bug #1982405: Fixed Infinidat driver to allow generic volume migration between two storage pools within the same cluster.

  • Infinidat Driver bug #1983287: Fixed Infinidat driver to allow backup of an attached volume.

  • Infinidat Driver bug #1984000: Fixed Infinidat driver to take into account the group identifier property when creating a new volume and add the volume to the consistency group.

  • IBM Spectrum Virtualize Family driver: Bug #1985065: Fixed to collect all the IP addresses for all the storage nodes given in lsip command response as volume of any iogrp should be available to the storage nodes in default scenario.

  • Bug #1996188: Fixed issue where a VMDK image file whose createType allowed named extents could expose host information. This change introduces a new configuration option, vmdk_allowed_types, that specifies the list of VMDK image subformats that Cinder will allow. The default setting allows only the ‘streamOptimized’ and ‘monolithicSparse’ subformats.

  • Bug #2008017: Fixed NetApp NFS driver to never spawn a native thread avoid thread starvation and other related issues.

  • Bug #1952443: Improve performance for creating volume from image, listing volumes, snapshots, backups, groups, and group_snapshots.

  • Bug #1945500: Fixed an error when uploading to Glance a previously downloaded glance image when glance multistore is enabled. Glance reserves image properties in the namespace ‘os_glance’ for its own use and will not allow images to be created with these properties. Additionally, there are image properties, such as those associated with image signature verification, that are stored in a volume’s image metadata, which should not be added to a new image when a volume is being uploaded as an image. Thus Cinder will no longer include any volume image metadata in the namespaces os_glance and img_signature when it creates an image in Glance. Furthermore, because the Glance property protections feature allows an operator to configure specific image properties as read-only, this fix adds a configuration option, reserved_image_namespaces, that allows an operator to exclude additional image properties by namespace (the os_glance and img_signature namespaces are always excluded).

  • bug #2000724: Handled the case when glance is calling online extend and external events were being sent to nova. Now Cinder will only send external events when the volume, to be extended, is attached to a nova instance.

  • Hitachi, NEC V, HPE XP drivers bug #2004140: Fixed KeyError when a backend is down.

  • LVM nvmet target bug #1964391: Fixed temporary disconnection of all volumes from all hosts when creating and removing volume exports.

  • LVM nvmet target bug #1964394: Fixed annoying kernel log message when exporting a volume.

  • nvmeof target bug #1966513: Fixed LVM failing on terminate_connection if the connecting host doesn’t have an iSCSI initiator name setup, for example if LVM is using the nvmet target.

  • Pure Storage FlashArray driver bug #1969784: Fixed array failover incorrectly handles loss of an array due to network issue

  • RBD driver bug #1960206: Fixed total_capacity reported by the driver to the scheduler on Ceph clusters that have renamed the bytes_used field to stored. (e.g., Nautilus).

  • Bug #2008259: Fixed the volume create functionality where non-admin users were able to create multiattach volumes by providing the multiattach parameter in the request body. Now we can only create multiattach volumes using a multiattach volume type, which is also the recommended way.

Other Notes

  • Removed the ability to create multiattach volumes by specifying multiattach parameter in the request body of a volume create operation. This functionality is unsafe, can lead to data loss, and has been deprecated since the Queens release. The recommended method for creating a multiattach volume is to use a volume type that supports multiattach. By default, volume types can only be created by the operator. Users who have a need for multiattach volumes should contact their operator if a suitable volume type is not available.