Support deferred volume deletion in the RBD driver¶
https://blueprints.launchpad.net/cinder/+spec/rbd-deferred-volume-deletion
This spec proposes configurable support for deferred volume deletion in the RBD driver, using the trash_* API calls, as introduced with the Luminous version of Ceph.
Problem description¶
Volume deletion in Cinder is synchronous: the c-vol worker thread deleting a volume will return only when the backend confirmed the removal of the volume. This approach ensures that the space used on the backend will at most be equal to the quota of the corresponding project, and hence allows for proper resource management. The current implementation of the RBD driver follows this approach by using RBD’s ‘remove()’ API call.
For deployments with thousands of volumes/users and large volumes (10-100TB) this can create two potential issues:
a c-vol worker thread will be blocked during deletion and depending on the size of the volume, the speed of the backend, the number of concurrent requests, and the total number of worker threads, this may block other operations, makes the service seem unresponsive, and can lead to timeouts.
[1] was proposed to mitigate this problem by making the number of native threads configurable and profit more efficiently from the backend’s potential capabilities.
users have to wait until the deletion is completed on the backend before they can re-use the freed up quota/space – despite having signalled to Cinder that the volume should be removed and the quota be freed. For large volumes, this waiting time can be significant (hours for large volumes) and has hence a corresponding negative impact on the user experience.
To address these issues, this spec proposes to optionally support asynchronous volume deletion in the RBD driver.
Use Cases¶
Administrators can optionally enable a deferred deletion: the Ceph backend will acknowledge the deletion immediately adressing the two issues above.
Proposed change¶
The proposal is to
amend the RBD driver to support deferred deletion by optionally calling RBD’s ‘trash_move()’ rather than RBD’s ‘remove()’ in the driver’s implementation of ‘delete_volume()’;
add a periodic timer in the RBD driver to trigger a regular purge.
For this, the following additional config options would be needed:
- ‘enable_deferred_deletion’ (default: ‘False’)
If enabled, RBD’s ‘thrash_move’ will be called rather than ‘remove’.
- ‘deferred_deletion_delay’ (default: 0)
Number of seconds before an image moved to trash is regarded as expired and can be purged.
- ‘deferred_deletion_purge_interval’ (default: 3600)
Number of seconds between runs of the periodic timer to purge the backend’s trash.
Alternatives¶
None
Data model impact¶
None
REST API impact¶
None.
Cinder-client impact¶
None.
Security impact¶
None.
Notifications impact¶
None.
Other end user impact¶
For the end users, deletions will appear to be instantaneous. The freed up quota will become available immediately.
Performance Impact¶
Volume deletions will become independent from the backend’s speed or the size of the volumes.
Other deployer impact¶
As the deletion is deferred, the space occupied will not be freed immediately. Deployers need to be aware that the maximum space needed can exceed the sum of the allocated quota of all projects.
Developer impact¶
None
Implementation¶
Assignee(s)¶
- Primary assignee:
Arne Wiebalck (arne.wiebalck@cern.ch)
Work Items¶
Update RBD driver to respect options for deferred deletions and call ‘trash_move’ if enabled.
Add periodic task to call ‘trash_purge’ for backend which have deferred deletion enabled.
Add related unit testcases.
Dependencies¶
None
Testing¶
Add unit tests to cover this change.
Documentation Impact¶
Add administrator documentation to advertise the option of deferred deletions for RBD and explain why&when&how this should be used, emphasizing in particular that by enabling this option the space of deleted volumes will not be freed on the RBD backend until the trash is purged.
