Restore a Subcloud/Group of Subclouds from Backup Data Using DCManager CLI¶
A subcloud can be restored from its backup data previously stored centrally on the system controller or locally on the subcloud using dcmanager command line interface (CLI). The subcloud install data must be available for this operation to proceed. The subcloud must support Redfish Virtual Media Service (version 1.2 or higher) if remote installation is required.
About this task
The CLI command dcmanager subcloud-backup restore can be used to restore a subcloud or a group of subclouds. By default, the restore is done from subcloud backup data on the central systemController. The command accepts the following parameters/options:
--with-installPerform remote installation of the subcloud prior to execution of restore procedure. The subcloud must support Redfish Virtual Media Service (version 1.2 or higher) to use this option.
--local-onlyUse the local backup archive (default local storage
/opt/platform-backup/backups/<release-version>). If not specified, the subcloud backup archive on the central systemController will be used.--registry-imagesRestore saved container images to registry.local as part of restore procedure (local storage only)
--subcloud <subcloud-name>The subcloud to restore.
--group <subcloud-group-name>The group of subclouds to restore.
--restore-values <yaml-file>The yaml file containing the customization parameters.
wipe_ceph_osds=false: To keep the Ceph cluster data intact.wipe_ceph_osds=true: To wipe the Ceph cluster entirely.on_box_data=true: To indicate that the backup data file is under /opt/platform-backup directory on the local machine.bootstrap_address: List of subclouds and their corresponding bootstrap addresses for connectivity.bootstrap_address: <subcloud_name1>: <subcloud_bootstrap_address1> <subcloud_name2>: <subcloud_bootstrap_address2>
Note
The
bootstrap_addresskey is only necessary for the restore of manually installed subclouds. For the subclouds installed via Redfish, thebootstrap_addressis already available in the install values.
See Run Restore Playbook Locally on the Controller for the list of configurable system restore parameters.
--sysadmin-password <sysadmin-password>If not specified, user will be prompted for the password. Recommend that this option is ONLY used for automation; i.e., for interactive use, don’t use option and specify password on prompting, so as to avoid sysadmin password getting into log files.
The --subcloud/--group is a mandatory parameter.
“When --registry-images option is applied, the entire registry filesystem
which contains both platform and user container images will be restored.
After the subcloud has been re-installed with the desired release version, the
backup archive for that release will be transferred to the subcloud for the
restore operation by default. If --local-only option is specified, the local
backup archive for the release will be used instead.
It is possible to specify a custom location of the backup file that resides on
the subcloud using --restore-values option and by setting
initial_backup_dir and backup_filename in the provided
restore_values yaml file. Please ensure this custom backup file is not
corrupted and is compatible with software release the subcloud was installed
with.
To restore images from a custom backup file on the subcloud using
--restore-values <yaml-file> option, the registry_backup_filename parameter
must be set in restore_values yaml file.
Restore a single subcloud¶
Prerequisites
The System Controller is healthy and ready to accept dcmanager related commands.
The subcloud is unmanaged and is in a valid state for restore operation (i.e. not being restored, installed, bootstrapped, deployed or rehomed).
The subcloud install data is available.
The backup file(s) exists and is compatible with the software release the subcloud is being restored to.
Procedure
To restore a subcloud, including remote installation, from system backup data in central storage:
~(keystone_admin)]$ dcmanager subcloud-backup restore --subcloud <subcloud-name> --with-install --sysadmin-password <sysadmin-password>
To restore a pre-installed subcloud from system backup data in central storage:
~(keystone_admin)]$ dcmanager subcloud-backup restore --subcloud <subcloud-name> --sysadmin-password <sysadmin-password>
To restore a subcloud, including remote installation, from system backup data stored in default local storage:
~(keystone_admin)]$ dcmanager subcloud-backup restore --subcloud <subcloud-name> --with-install --local-only --sysadmin-password <sysadmin-password>
To restore a subcloud, including remote installation, from system backup and images backup data in default local storage:
~(keystone_admin)]$ dcmanager subcloud-backup restore --subcloud <subcloud-name> --local-only --registry-images --sysadmin-password <sysadmin-password>
Note
The --registry-images option can only be used with --local-only
option.
To restore a pre-installed subcloud from system and images backup data stored at custom location on the subcloud:
Create a yaml file e.g.
restore_overrides.yamlwith the following content:initial_backup_dir: /home/sysadmin/mybackup_dir backup_filename: test_platform_backup.tgz registry_backup_filename: test_images_backup.tgz
Then, run the command:
~(keystone_admin)]$ dcmanager subcloud-backup restore --subcloud subcloud1 --local-only –-registry-images --restore-values restore_overrides.yaml --sysadmin-password <sysadmin-password>
Sample response to a single subcloud restore:
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 8 |
| name | subcloud1 |
| description | None |
| location | None |
| software_version | 22.12 |
| management | unmanaged |
| availability | offline |
| deploy_status | restore-failed |
| management_subnet | fd01:15::0/64 |
| management_start_ip | fd01:15::2 |
| management_end_ip | fd01:15::11 |
| management_gateway_ip | fd01:15::1 |
| systemcontroller_gateway_ip | fd01:1::1 |
| group_id | 2 |
| created_at | 2022-12-12 05:29:23.807243 |
| updated_at | 2022-12-13 16:39:48.904037 |
| backup_status | unknown |
| backup_datetime | None |
+-----------------------------+----------------------------+
Note
The subcloud can be restored or restored again while in a failed deploy state such as:
data-migration-failed (upgrade failure)
restore-failed (previous restore attempt failed due to a bad backup file)
rehome-failed
To view the progress of subcloud restore, please use dcmanager subcloud show or dcmanager subcloud list command:
~(keystone_admin)]$ dcmanager subcloud show subcloud1
+-----------------------------+----------------------------+
| Field | Value |
+-----------------------------+----------------------------+
| id | 9 |
| name | subcloud2 |
| description | None |
| location | None |
| software_version | 22.12 |
| management | unmanaged |
| availability | offline |
| deploy_status | restoring |
| management_subnet | fd01:176::0/64 |
| management_start_ip | fd01:176::2 |
| management_end_ip | fd01:176::11 |
| management_gateway_ip | fd01:176::1 |
| systemcontroller_gateway_ip | fd01:1::1 |
| group_id | 2 |
| created_at | 2022-12-13 00:09:44.543494 |
| updated_at | 2022-12-13 18:23:20.659138 |
| backup_status | unknown |
| backup_datetime | None |
| dc-cert_sync_status | unknown |
| firmware_sync_status | unknown |
| identity_sync_status | unknown |
| kubernetes_sync_status | unknown |
| kube-rootca_sync_status | unknown |
| load_sync_status | unknown |
| patching_sync_status | unknown |
| platform_sync_status | unknown |
+-----------------------------+----------------------------+
If the restore operation completes successfully, the subcloud will become
online and the deploy_status will be set to ‘complete’.
Please continue with Post restore procedure.
If the restore operation fails, dcmanager subcloud errors command can be used to view the error.
Restore a group of subclouds¶
The above subcloud-backup restore operations can be performed for a group of
subclouds simultaneously by replacing --subcloud option with --group
option. For instance, to restore a group of subclouds with remote installation
from their system data in central storage:
~(keystone_admin)]$ dcmanager subcloud-backup restore --group <group> --with-install --sysadmin-password <sysadmin-password>
If all subclouds in the group are not in the valid state for restore, an error message will be displayed. If some of the subclouds in the group meet restore operation criteria, a list will be displayed.
Sample group restore response:
+----+-----------+-------------+----------+------------------+------------+--------------+---------------+-------------------+---------------------+-------------------+-----------------------+-----------------------------+----------+----------------------------+----------------------------+----------------+----------------------------+
| id | name | description | location | software_version | management | availability | deploy_status | management_subnet | management_start_ip | management_end_ip | management_gateway_ip | systemcontroller_gateway_ip | group_id | created_at | updated_at | backup_status | backup_datetime |
+----+-----------+-------------+----------+------------------+------------+--------------+---------------+-------------------+---------------------+-------------------+-----------------------+-----------------------------+----------+----------------------------+----------------------------+----------------+----------------------------+
| 8 | subcloud6 | None | None | 22.12 | unmanaged | online | complete | fd01:15::0/64 | fd01:15::2 | fd01:15::11 | fd01:15::1 | fd01:1::1 | 2 | 2022-12-13 18:23:03.883068 | 2022-12-13 22:14:39.331199 | complete-local | 2022-12-13 22:04:06.232043 |
| 9 | subcloud8 | None | None | 22.12 | unmanaged | online | complete | fd01:176::0/64 | fd01:176::2 | fd01:176::11 | fd01:176::1 | fd01:1::1 | 2 | 2022-12-13 19:27:55.115604 | 2022-12-13 22:15:09.287665 | complete-local | 2022-12-13 22:05:03.785280 |
+----+-----------+-------------+----------+------------------+------------+--------------+---------------+-------------------+---------------------+-------------------+-----------------------+-----------------------------+----------+----------------------------+----------------------------+----------------+----------------------------+
After group restore is complete, continue with Post restore procedure for each subcloud in the group.
Post restore procedure¶
AIO-SX subcloud
Resume subcloud audit with the command:
~(keystone_admin)]$ dcmanager subcloud manage
AIO-DX/Standard subcloud
If the restore playbook completes successfully, the subcloud will be online and
deploy_status will be set to complete. Only controller-0 will be in
unlocked and online state. To complete the restore operation, follow the
procedure available in Restore Platform System Data and Storage for
restoring the remaining subcloud nodes.
Resume subcloud audit with the command:
~(keystone_admin)]$ dcmanager subcloud manage
