Rocky Series Release Notes

13.0.9-7

Bug Fixes

  • PowerMax Driver - Issue with upgrades from pre Pike to Pike and later. The device is not found when trying to snapshot a legacy volume.

13.0.9

Bug Fixes

  • Fixed issue where all Ceph RBD backups would be incremental after the first one. The driver now honors whether --incremental is specified or not.

13.0.8

Bug Fixes

  • Fixes a bug that prevented distributed file system drivers from creating snapshots during volume clone operations (NFS, WindowsSMBFS, VZstorage and Quobyte drivers). Fixing this allows creating snapshot based backups.

  • Fix DetachedInstanceError is not bound to a Session for VolumeAttachments. This affected VolumeList.get_all, and could make a service fail on startup and make it stay in down state.

13.0.7

Bug Fixes

  • Bug 1809249 - 3PAR driver adds the config option hpe3par_target_nsp that can be set to the 3PAR backend to use when multipath is not enabled and the Fibre Channel Zone Manager is not used.

  • Dell EMC SC Driver: Fixes bug 1822229 to handle the volume mappings in the backend when a volume is attached to multiple instances on the same host.

13.0.6

Bug Fixes

  • Kaminario FC and iSCSI drivers: Fixed bug 1829398 where force detach would fail.

  • NetApp iSCSI drivers no longer use the discovery mechanism for multipathing and they always return all target/portals when attaching a volume. Thanks to this, volumes will be successfully attached even if the target/portal selected as primary is down, this will be the case for both, multipath and single path connections.

13.0.4

Bug Fixes

  • Fixed group availability zone-backend host mismatch [Bug 1773446].

  • Fixed NetApp SolidFire bug that avoided multiatached volumes to be deleted.

  • Dell EMC VNX Driver: Fixes bug 1817385 to make sure the sg can be created again after it was destroyed under destroy_empty_storage_group setting to True.

13.0.3

New Features

  • VMAX driver support for new configuration option - vmax_snapvx_unlink_limit for specifying the maximum number of unlinks which will be performed before a clone operation. Default value is 3

Upgrade Notes

  • Beginning with Cinder version 12.0.0, as part of the Queens release “policies in code” community effort, Cinder has had the ability to run without a policy file because sensible default values are specified in the code. Customizing the policies in effect at your site, however, still requires a policy file. The default location of this file has been /etc/cinder/policy.json (although the documentation has indicated otherwise). With this release, the default location of this file is changed to /etc/cinder/policy.yaml.

    Some points to keep in mind:

    • The policy file to be used may be specified in the /etc/cinder/cinder.conf file in the [oslo_policy] section as the value of the policy_file configuration option. That way there’s no question what file is being used.

    • To find out what policies are available and what their default values are, you can generate a sample policy file. To do this, you must have a local copy of the Cinder source code repository. From the top level directory, run the command:

      tox -e genpolicy

      This will generate a file named policy.yaml in the etc/cinder directory of your checked-out Cinder repository.

    • The sample file is YAML (because unlike JSON, YAML allows comments). If you prefer, you may use a JSON policy file.

    • Beginning with Cinder 12.0.0, you only need to specify policies in your policy file that you want to differ from the default values. Unspecified policies will use the default values defined in the code. Given that a default value must be specified in the code when a new policy is introduced, the default policy, which was formerly used as a catch-all for policy targets that were not defined elsewhere in the policy file, has no effect. We mention this because an old upgrade strategy was to use the policy file from the previous release with "default": "role:admin" (or "default": "!") so that newly introduced actions would be blocked from end users until the operator had time to assess the implications of exposing these actions. This strategy no longer works. Hopefully this isn’t a problem because we’re defining sensible defaults in the code. It would be a good idea, however, to generate the sample policy file with each release (see instructions above) to verify this for yourself.

Bug Fixes

  • PowerMax driver - Workload support was dropped in ucode 5978. If a VMAX All Flash array is upgraded to 5978 or greater and existing volume types leveraged workload e.g. DSS, DSS_REP, OLTP and OLTP_REP, certain operations will no longer work and the volume type will be unusable. This fix addresses these issues and fixes problems with using old volume types with workloads included in the volume type pool_name.

  • Fixed a bug which could create volumes with invalid content in case of unhandled errors from glance client (Bug #1799221).

  • The Solidfire cinder driver has been fixed to ensure delete happens on the correct volume.

13.0.2

Upgrade Notes

  • The cinder-manage db online_data_migrations command now returns exit status 2 in the case where some migrations failed (raised exceptions) and no others were completed successfully from the last batch attempted. This should be considered a fatal condition that requires intervention. Exit status 1 will be returned in the case where the --max-count option was used and some migrations failed but others succeeded (updated at least one row), because more work may remain for the non-failing migrations, and their completion may be a dependency for the failing ones. The command should be reiterated while it returns exit status 1, and considered completed successfully only when it returns exit status 0.

13.0.1

Known Issues

  • Multiattach support is disabled for the LVM driver when using the LIO iSCSI target. This functionality will be fixed in a later release.

Bug Fixes

  • Fixed support for IPv6 on management and data paths for NFS, iSCSI and FCP NetApp ONTAP drivers.

  • Fixes a bug in NetApp SolidFire where the deletion of group snapshots was failing.

  • Fixes force_detach behavior for volumes in NetApp SolidFire driver.

13.0.0

New Features

  • Now scheduler plugins are aware of operation type via operation attribute in RequestSpec dictionary, plugins can support backend filtering according to backend status as well as operation type. Current possible values for operation are:

    • create_volume

    • extend_volume

    • create_snapshot

    • retype_volume

    • migrate_volume

    • manage_existing

    • manage_existing_snapshot

    • create_group

  • Added boolean conf option ‘split_loggers’ in [default] section of cinder.conf to enable split logging functionality. The default value of split_loggers option is set to False. Operator can set it’s value to True to split HTTP content into subloggers to allow for fine-grained control of what is logged and how. This new config option ‘split_loggers’ should be enabled only when keystoneauth log level is set to DEBUG in ‘default_log_levels’ config option.

  • A new target, NVMET, is added for the LVM driver over RDMA, it allows cinder to use nvmetcli in order to create/delete subsystems on attaching/detaching an LVM volume to/from an instance.

  • Add support to force detach a volume from all hosts on 3PAR.

  • Cinder now allows for capacity based QoS which can be useful in environments where storage performance scales with consumption (such as RBD backed storage). The newly added QoS specs are read_iops_sec_per_gb, write_iops_sec_per_gb, total_iops_sec_per_gb, read_bytes_sec_per_gb, write_bytes_sec_per_gb and total_bytes_sec_per_gb. These values will be multiplied by the size of the volume and passed to the consumer. For example, setting total_iops_sec_per_gb to 30 and setting total_bytes_sec_per_gb to 1048576 (1MB) then creating a 100 GB volume with that QoS will result in a volume with 3,000 total IOPs and 100MB/s throughput limit.

  • Cinder now allows for a minimum value when using the capacity based QoS in order to make sure small volumes can get a minimum allocation for them to be usable. The newly added QoS specs are read_iops_sec_per_gb_min, write_iops_sec_per_gb_min, total_iops_sec_per_gb_min, read_bytes_sec_per_gb_min, write_bytes_sec_per_gb_min and total_bytes_sec_per_gb_min

  • A new cinder-manage command, reset_active_backend, was added to promote a failed-over backend participating in replication. This allows you to reset a backend without manually editing the database. A backend undergoing promotion using this command is expected to be in a disabled and frozen state. Support for both standalone and clustered backend configurations are supported.

  • Added dell_api_async_rest_timeout option to the Dell EMC SC driver. This is the timeout used for asynchronous REST calls to the Dell EMC SC REST API. Default is 15 seconds.

  • Added dell_api_sync_rest_timeout option to the Dell EMC SC driver. This is the timeout used for synchronous REST calls to the Dell EMC SC REST API. Default is 30 seconds.

  • Enabled Cinder Multi-Attach capability in the Dell EMC Storage Center Cinder driver.

  • IBM DS8K driver has added multiattach support.

  • Added flag ‘backend_state’ which will give backend state info in service list.

  • NetApp ONTAP iSCSI and FCP drivers multiattach capability enabled.

  • NetApp ONTAP NFS multiattach capability enabled.

  • Support backup restore cancelation by changing the backup status to anything other than restoring using cinder backup-reset-state.

  • Cinder backup creation can now (since microversion 3.51) receive the availability zone where the backup should be stored.

  • Cinder backup now supports running multiple processes to make the most of the available CPU cores. Performance gains will be significant when running multiple concurrent backups/restores with compression. The number of processes is set with backup_workers configuration option.

  • When using the RBD pool exclusively for Cinder we can now set rbd_exclusive_cinder_pool to true and Cinder will use DB information to calculate provisioned size instead of querying all volumes in the backend, which will reduce the load on the Ceph cluster and the volume service.

  • Google backup driver now supports google-auth library, and is the preferred library if both google-auth (together with google-auth-httplib2) and oauth2client libraries are present in the system.

  • Support for volume multi-attach in the INFINIDAT InfiniBox driver.

  • New FC Cinder volume driver for Inspur Instorage.

  • The NetApp ONTAP driver supports a new configuration option netapp_api_trace_pattern to enable filtering backend API interactions to log. This option must be specified in the backend section when desired and it accepts a valid python regular expression.

  • Added backend driver for Nexenta Edge iSCSI storage.

  • Added support to Pure Storage Volume Drivers for Active Cluster using the standard replication API’s for the Block Storage Service.

  • Pure Storage FlashArray driver has added multiatach support.

  • QNAP Cinder driver added support for QES fw 2.1.0.

  • Added support for active-active replication to the RBD driver. This allows users to configure multiple volume backends that are all a member of the same cluster participating in replication.

  • Allow rbd driver to list manageable snapshots.

  • Allow rbd driver to report backend state.

  • Added flag ‘backend_state: up/down’ which will give backend state info in service list.

  • Added flag ‘backend_state’ which will give backend state info in service list.

  • Dell EMC ScaleIO has been renamed to Dell EMC VxFlex OS. Documentation for the driver can be found under the new name. The driver maintains full backwards compatability with prior ScaleIO releases and no configuration changes are needed upon upgrade to the new version of the driver.

  • The SMBFS driver now supports the ‘snapshot attach’ feature. Special care must be taken when attaching snapshots though, as writing to a snapshot will corrupt the differencing image chain.

  • Added data reduction pool support for thin-provisoned and compressed volume in Storwize cinder driver.

  • Now availability zone is supported in volume type as below.

    • RESKEY:availability_zones now is a reserved spec key for AZ volume type, and administrator can create AZ volume type that includes AZ restrictions by adding a list of Az’s to the extra specs similar to: RESKEY:availability_zones: az1,az2.

    • Extra spec RESKEY:availability_zones will only be used for filtering backends when creating and retyping volumes.

    • Volume type can be filtered within extra spec: /types?extra_specs={“key”:”value”} since microversion “3.52”.

  • Added image signature verification support when creating volume from image. This depends on signature metadata from glance. This feature is turned on by default, administrators can change behaviour by updating option verify_glance_signatures. Also, an additional image metadata signature_verified has been added to indicate whether signature verification was performed during creating process.

  • Cinder-manage DB sync command can now bump the RPC and Objects versions of the services to avoid a second restart when doing offline upgrades.

  • Adds support to configure the size of the native thread pool used by the cinder volume and backup services. For the backup we use backup_native_threads_pool_size in the [DEFAULT] section, and for the backends we use backend_native_threads_pool_size in the driver section.

  • Support transfer volume with snapshots by default in new V3 API ‘v3/volume_transfers’. After microverison 3.55, if users don’t want to transfer snapshots, they could use the new optional argument no_snapshots=True in request body of new transfer creation API.

  • Dell EMC Unity driver: Add compressed volume support.

  • Dell EMC Unity Cinder driver allows enabling/disabling the SSL verification. Admin can set True or False for driver_ssl_cert_verify to enable or disable this function, alternatively set the driver_ssl_cert_path=<PATH> for customized CA path. Both above 2 options should go under the driver section.

  • Dell EMC Unity Driver: Adds support for removing empty host. The new option named remove_empty_host could be configured as True to notify Unity driver to remove the host after the last LUN is detached from it.

  • Added ISCSI based driver for Veritas Access.

  • Dell EMC VMAX driver has added multiattach support.

  • Dell EMC VMAX driver has added list manageable volumes and snapshots support.

  • Log VMAX specific metadata of a volume if debug is enabled.

  • Support for retype (storage-assisted migration) of replicated volumes on VMAX cinder driver.

  • VMware VMDK driver and FCD driver now support a config option vmware_datastore_regex to specify the regular expression pattern to match the name of datastores where backend volumes are created.

  • VMware VMDK driver and FCD driver now support NFS 4.1 datastores in vCenter server.

  • Added support to revert a volume to a snapshot with the Dell EMC VNX driver.

  • The Cinder Volume Backup service can now be run on Windows. It supports backing up volumes exposed by SMBFS/iSCSI Windows Cinder Volume backends, as well as any other Cinder backend that’s accessible on Windows (e.g. SANs exposing volumes via iSCSI/FC).

    The Swift and Posix backup drivers are known to be working on Windows.

  • Dell EMC XtremIO driver has added multiattach support.

Known Issues

  • If RBD stats collection is taking too long in your environment maybe even leading to the service appearing as down you’ll want to use the rbd_exclusive_cinder_pool = true configuration option if you are using the pool exclusively for Cinder and maybe even if you are not and can live with the innacuracy.

Upgrade Notes

  • The Dell EMC CoprHD drivers have been marked as unsupported and are now deprecated. enable_unsupported_driver will need to be set to True in the driver’s section in cinder.conf to continue to use it.

  • The DataCore drivers have been marked as unsupported and are now deprecated. enable_unsupported_driver will need to be set to True in the driver’s section in cinder.conf to continue to use it.

  • The Disco driver has been marked as unsupported and is now deprecated. enable_unsupported_driver will need to be set to True in the driver’s section in cinder.conf to continue to use it.

  • The HGST driver has been marked as unsupported and is now deprecated. enable_unsupported_driver will need to be set to True in the driver’s section in cinder.conf to continue to use it.

  • In NEC driver, the number of volumes in a storage pool is no longer limited to 1024. More volumes can be created with storage firmware revision 1015 or later.

  • The following commands are no longer required to be listed in your rootwrap configuration: cgcreate; and cgset.

  • Backup service to driver mapping is removed. If you use old values like ‘cinder.backup.services.swift’ or ‘cinder.backup.services.ceph’ it should be changed to ‘cinder.backup.drivers.swift’ or ‘cinder.backup.drivers.ceph’ accordingly to get your backup service working.

  • Removed the option allow_inuse_volume_type_modification which had been deprecated in Ocata release.

  • The LVM driver specific lvm_max_over_subscription_ratio setting had been deprecated and is now removed. Over subscription should now be managed using the generic max_over_subscription_ratio setting.

  • VMAX driver - Removed deprecated option cinder_dell_emc_config_file

  • On offline upgrades, due to the rolling upgrade mechanism we need to restart the cinder services twice to complete the installation just like in the rolling upgrades case. First you stop the cinder services, then you upgrade them, you sync your DB, then you start all the cinder services, and then you restart them all. To avoid this last restart we can now instruct the DB sync to bump the services after the migration is completed, the command to do this is cinder-manage db sync –bump-versions

Deprecation Notes

  • Deprecate option check_max_pool_luns_threshold. The VNX driver will always check the threshold.

  • The Dell EMC CoprHD drivers have been marked as unsupported and are now deprecated. enable_unsupported_driver will need to be set to True in the driver’s section in cinder.conf to continue to use it. If its support status does not change, they will be removed in the Stein development cycle.

  • The DataCore drivers have been marked as unsupported and are now deprecated. enable_unsupported_driver will need to be set to True in the driver’s section in cinder.conf to continue to use it. If its support status does not change, they will be removed in the Stein development cycle.

  • VMAX driver - configuration tag san_rest_port will be replaced by san_api_port in the next release.

  • The Disco driver has been marked as unsupported and is now deprecated. enable_unsupported_driver will need to be set to True in the driver’s section in cinder.conf to continue to use it. If its support status does not change, it will be removed in the Stein development cycle.

  • Cinder’s Google backup driver is now called gcs, so backup_driver configuration for Google Cloud Storage should be updated from cinder.backup.drivers.google to cinder.backup.driver.gcs.

  • The HGST driver has been marked as unsupported and is now deprecated. enable_unsupported_driver will need to be set to True in the driver’s section in cinder.conf to continue to use it. If its support status does not change, it will be removed in the Stein development cycle.

  • The NetApp E-Series drivers are deprecated as of the Rocky release and will be removed in the Stein release. Other configurations of the NetApp driver, i.e Clustered Data ONTAP and Solidfire, are unaffected.

Security Issues

  • Removed the ability to create volumes in a ScaleIO Storage Pool that has zero-padding disabled. A new configuration option had been added to override this new behavior and allow volume creation, but should not be enabled if multiple tenants will utilize volumes from a shared Storage Pool.

  • Privsep transitions. Cinder is transitioning from using the older style rootwrap privilege escalation path to the new style Oslo privsep path. This should improve performance and security of Cinder in the long term.

  • Privsep daemons are now started by Cinder when required. These daemons can be started via rootwrap if required. rootwrap configs therefore need to be updated to include new privsep daemon invocations.

  • Removed the ability to create volumes in a ScaleIO Storage Pool that has zero-padding disabled. A new configuration option sio_allow_non_padded_volumes has been added to override this new behavior and allow unpadded volumes, but should not be enabled if multiple tenants will utilize volumes from a shared Storage Pool.

Bug Fixes

  • Add option max_luns_per_storage_group back. The max LUNs per storage group was set to 255 before. With the new option, admin can set it to a larger number.

  • NetApp ONTAP NFS (bug 1690954): Fix wrong usage of export path as volume name when deleting volumes and snapshots.

  • NetApp ONTAP iSCSI (bug 1712651): Fix ONTAP NetApp iSCSI driver not raising a proper exception when trying to extend an attached volume beyond its max geometry.

  • NetApp ONTAP (bug 1762424): Fix ONTAP NetApp driver not being able to extend a volume to a size greater than the corresponding LUN max geometry.

  • NetApp ONTAP (bug 1765182): Make ONTAP NetApp iSCSI driver and FC driver report to the Cinder scheduler that they don’t support online volume extending.

  • Make Cinder scheduler check if backend reports online_extend_support before performing an online extend operation.

  • NetApp ONTAP (bug 1765182): Make ONTAP NetApp NFS driver report to the Cinder scheduler that it doesn’t support online volume extending.

  • Fixed QNAP driver failures to create volume and snapshot in some cases.

  • Fixed QNAP driver failures to detach iscsi device while uploading volume to image.

  • The XtremIO driver has been fixed to correctly report the “free_capacity_gb” size.

  • Dell EMC SC driver correctly returns initialize_connection data when more than one IQN is attached to a volume. This fixes some random Nova Live Migration failures where the connection information being returned was for an IQN other than the one for which it was being requested.

  • Dell EMC Unity Driver: Fixes bug 1759175 to detach the lun correctly when auto zone was enabled and the lun was the last one attached to the host.

  • We no longer leave orphaned chunks on the backup backend or leave a temporary volume/snapshot when aborting a backup.

  • Resolve issue with cross AZ migrations and retypes where the destination volume kept the source volume’s AZ, so we ended up with a volume where the AZ does not match the backend. (bug 1747949)

  • [bug 1772421] INFINIDAT fixed a bug in volume extension feature where volumes were not extended to target size but added the given target size.

  • Cinder will now consume quota when importing new backup resource.

  • Fix a quota usage error triggered by a non-admin user backing up an in-use volume. The forced backup uses a temporary volume, and quota usage was incorrectly updated when the temporary volume was deleted after the backup operation completed. Fixes bug 1778774.

  • Volume “force delete” was introduced with the 3.23 API microversion, however the check for in the service was incorrectly looking for microversion 3.2. That check has now been fixed. It is possible that an API call using a microversion below 3.23 would previously work for this call, which will now fail. This closes bug #1783028.

  • Google backup driver now works when using google-api-python-client version 1.6.0 or higher.

  • When encryption keys based on the ConfKeyManager’s fixed_key are migrated to Barbican, ConfKeyManager keys stored in the Backup table are included in the migration process. Fixes bug 1757235.

  • Storwize SVC Driver: Fixes bug 1749687 previously lsvdisk() was called separately for every ‘in-use’ volume in order to check if the volume exists on the storage. In order to avoid problem of too long driver initialization now lsvdisk() is called once per pool.

  • Fixed bug #1783582, where calls to os-force_detach were failing on NetApp ONTAP iSCSI/FC drivers.

  • Fixed bug #1731474 on NetApp Data ONTAP driver that was causing LUNs to be created with larger size than requested. This fix requires version 9.1 of ONTAP or later.

  • Two new policies “volume_extension:type_get” and “volume_extension:type_get_all” have been added to control type show and type list APIs.

  • Added a new optional cache of volumes generated from snapshots for the Quobyte backend. Enabling this cache speeds up creation of multiple volumes from a single snapshot at the cost of a slight increase in creation time for the first volume generated for this given snapshot. The quobyte_volume_from_snapshot_cache option is off by default.

  • VMAX driver - fixes SSL certificate verification error.

  • Updated the parameter storwize_preferred_host_site from StrOpt to DictOpt in cinder back-end configuration, and removed it from volume type configuration.

  • After an offline upgrade we had to restart all Cinder services twice, now with the cinder-manage db sync –bump-versions command we can avoid the second restart.

  • Fixes concurrency issue on backups, where only 20 native threads could be concurrently be executed. Now default will be 60, and can be changed with backup_native_threads_pool_size.

  • RBD driver can have bottlenecks if too many slow operations are happening at the same time (for example many huge volume deletions), we can now use the backend_native_threads_pool_size option in the RBD driver section to resolve the issue.

  • Dell EMC Unity Driver: Fixes bug 1773305 to return the targets which connect to the logged-out initiators. Then the zone manager could clean up the FC zone based on the correct target wwns.

Other Notes

  • Added schema validation support using jsonschema [json-schema-validation] for all supported v3 APIs.

    Following APIs were accepting boolean parameters with leading and trailing white spaces (for e.g. “ true “). But now with schema validation support, all these boolean parameters henceforth will not accept leading and trailing whitespaces to maintain consistency.

    • Generic volume groups:

      • delete group: “POST /v3/{project_id}/groups/{group_id}/action”

      • failover replication: “POST /v3/{project_id}/groups/{group_id}/action”

    • Volume Snapshots:

      • create a snapshot: “POST /v3/{project_id}/snapshots”

    • Volume_actions:

      • set bootable: “POST /v3/{project_id}/volumes/{volume_id}/action”

      • volume readonly update: “POST /v3/{project_id}/volumes/{volume_id}/action”

  • The “cinder-manage logs” commands have been removed. Information previously gathered by these commands may be found in cinder service and syslog logs.

  • Dell EMC VNX driver: Enhances the performance of create/delete volume.