Migrate an AIO-SX to an AIO-DX Subcloud¶
About this task
You can migrate an AIO-SX subcloud to an AIO-DX subcloud without reinstallation. This operation involves updating the system mode, adding the OAM unit IP addresses of each controller, and installing the second controller.
Prerequisites
A distributed cloud system is setup with at least a system controller and an AIO-SX subcloud. The subcloud must be online and managed by dcmanager. Both the management network and cluster-host network need to be configured and cannot be on the loopback interface.
Reconfigure Loopback Assigned Interfaces¶
If any interface is on the loopback interface, use the following procedure to reconfigure it on to a physical interface.
Procedure
List interfaces assigned to loopback.
~(keystone_admin)$ system interface-network-list 1 | awk '$6 == "lo" { print $8 }'Lock the active controller.
~(keystone_admin)$ system host-lock controller-0
Change the class attribute to ‘none’ for the loopback interface.
~(keystone_admin)$ system host-if-modify controller-0 lo -c none
For each network listed in the first step, delete its interface-network configuration. For example, for cluster-host:
~(keystone_admin)$ IFNET_UUID=$(system interface-network-list controller-0 | awk '{if ($8 =="cluster-host") print $4;}') ~(keystone_admin)$ system interface-network-remove $IFNET_UUIDFor each network listed in the first step, assign it to the new interface. This example assumes the network is cluster-host and the interface name is mgmt0.
~(keystone_admin)$ system interface-network-assign controller-0 mgmt0 cluster-host
Postrequisites
Continue with the AIO-SX to AIO-DX subcloud migration, using one of the following procedures:
Use Ansible Playbook to Migrate a Subcloud from AIO-SX to AIO-DX, or Manually Migrate a Subcloud from AIO-SX to AIO-DX.
Use Ansible Playbook to Migrate a Subcloud from AIO-SX to AIO-DX¶
Use the following procedure to migrate a subcloud from AIO-SX to AIO-DX using the ansible playbook.
Prerequisites
the subcloud must be online and managed from the System Controller
the subcloud’s controller-0 may be locked or unlocked; the ansible playbook will lock the subcloud controller-0 as part of migrating the subcloud
Procedure
Use the ansible-vault create migrate-subcloud1-overrides-EXAMPLE.yml command to securely specify the OAM unit IP addresses and the ansible ssh password. The existing OAM IP address of the AIO-SX system will be used as the OAM floating IP address of the new AIO-DX system.
In the following example, 10.10.10.13 and 10.10.10.14 are the new OAM unit IP addresses for controller-0 and controller-1 respectively and oam-ipv4 is the current OAM address-pool name. The same logic applies to management, cluster-host and pxeboot networks.
{ "ansible_ssh_pass": "St8rlingXCloud*", "ansible_become_pass": "St8rlingXCloud*", "external_oam_node_0_address": "10.10.10.13", "external_oam_node_1_address": "10.10.10.14", "external_oam_addrpool_name": "oam-ipv4", "management_node_0_address": "192.168.204.3", "management_node_1_address": "192.168.204.4", "management_addrpool_name": "management-ipv4", "cluster_host_node_0_address": "192.168.206.2", "cluster_host_node_1_address": "192.168.206.3", "cluster_host_addrpool_name": "cluster-host-subnet-ipv4", "pxeboot_node_0_address": "169.254.202.2", "pxeboot_node_1_address": "169.254.202.3", "pxeboot_addrpool_name": "pxeboot", }All network parameters in the previous example are required. In case there is an admin or a storage network configured, each one needs to be added to the overrides file as well. For admin network, add:
"admin_node_0_address": "192.168.210.3", "admin_node_1_address": "192.168.210.4", "admin_addrpool_name": "admin-ipv4",
For storage network, add:
"storage_node_0_address": "10.10.20.2", "storage_node_1_address": "10.10.20.3", "storage_addrpool_name": "storage-pool",
Note
When dual-stack network is configured, use comma separated values for the primary and secondary networks. For example:
"management_node_0_address": "192.168.204.3,fd01::3", "management_node_1_address": "192.168.204.4,fd01::4", "management_addrpool_name": "management-ipv4,management-ipv6",
On the system controller, run the ansible playbook to migrate the AIO-SX subcloud to an AIO-DX.
Note
Run the following command on an AIO-SX system before changing to AIO-DX, as this does not apply to AIO-DX systems.
~(keystone_admin)$ system storage-backend-modify ceph-store replication=2
For example, if the subcloud name is ‘subcloud1’, enter:
~(keystone_admin)$ ansible-playbook --ask-vault-pass /usr/share/ansible/stx-ansible/playbooks/migrate_sx_to_dx.yml -e @migrate-subcloud1-overrides-EXAMPLE.yml -i subcloud1, -v
The ansible playbook will lock the subcloud’s controller-0, if it not already locked, apply the configuration changes to convert the subcloud to an AIO-DX system with a single controller, and unlock controller-0. Wait for the controller to reset and come back up to an operational state.
Software install and configure the second controller for the subcloud.
From the System Controller, reconfigure the subcloud, using dcmanager. Specify the sysadmin password and the deployment configuration file, using the dcmanager subcloud deploy config command.
~(keystone_admin)$ dcmanager subcloud deploy config --sysadmin-password <sysadmin_password> --deploy-config deployment-config-subcloud1-duplex.yaml <subcloud1>
where <sysadmin_password> is assumed to be the login password and <subcloud1> is the name of the subcloud
Note
--deploy-configmust reference a deployment configuration file for a AIO-DX subcloud.For example, deployment-config-subcloud1-duplex.yaml should only include changes for controller-1 as changing fields for other nodes/ resources may cause them to go out of sync.
Manually Migrate a Subcloud from AIO-SX to AIO-DX¶
As an alternative to using the Ansible playbook, use the following procedure to manually migrate a subcloud from AIO-SX to AIO-DX. Perform the following commands on the AIO-SX subcloud.
Procedure
If not already locked, lock the active controller.
~(keystone_admin)$ system host-lock controller-0
Run the following command on an AIO-SX system before changing to AIO-DX:
For Ceph installation:
~(keystone_admin)$ system storage-backend-modify ceph-store replication=2
For Rook Container-based Ceph installation:
~(keystone_admin)$ system storage-backend-modify ceph-rook-store replication=2
Change the system mode to ‘duplex’.
~(keystone_admin)$ system modify --system_mode=duplex
Add the node IP addresses (controller-0 and controller-1) to the address pool for the required networks.
For example, if the OAM subnet is 10.20.2.0/24 and AIO-SX system’s OAM IP address is 10.20.2.3, this address will continue being used as the floating address and 10.20.2.4 and 10.20.2.5 could be used for controller-0 and controller-1 respectively. The same logic applies to management, admin, cluster-host, pxeboot and storage networks. The address pool modification is always required for OAM, management, cluster-host and pxeboot. If the system is suing admin and storage networks they need to be changed as well. Please note that when using dual-stack, two address pools exist for each network, one for IPv4 and one for IPv6, with the exception of pxeboot that works only with IPv4.
Note
The node addresses for controller-0 and controller-1 have to be different from each other and from the floating address.
To find all address pools that need to be changed, first using ‘system network-addrpool-list’ command list the address poll names filtered by the network names:
~(keystone_admin)$ system network-addrpool-list | awk -F'[| ]+' ' NR > 3 && !/^+--/ && $3 ~ "(oam|mgmt|admin|cluster-host|pxeboot|storage-net)" { print $3,"|",$4}' | column -t admin admin-ipv4 admin admin-ipv6 cluster-host cluster-host-subnet-ipv4 cluster-host cluster-host-subnet-ipv6 mgmt management-ipv4 mgmt management-ipv6 oam oam-ipv4 oam oam-ipv6 pxeboot pxeboot storage-net storage-pool-ipv4 storage-net storage-pool-ipv6In this example, the non-required admin and storage networks are present. Then find the address pool’s UUIDs of the required pools by listing all address pools and filtering the entries according to the second column in the previous example. Please note that some of your address pool names could be different, so change the command accordingly.
~(keystone_admin)$ system addrpool-list --nowrap | grep -E "(oam-|management|admin|cluster-host|pxeboot|storage)" | 68817b0b-01ed-46cd-98ef-25f4d23bef9b | cluster-host-subnet-ipv4 | 192.168.206.0 | 24 | random | ['192.168.206.1-192.168.206.50'] | 192.168.206.1 | None | None | None | | 3cfd5254-a543-4b17-bfac-c2cde950f152 | cluster-host-subnet-ipv6 | fd02:: | 64 | random | ['fd02::1-fd02::50'] | fd02::1 | None | None | None | | 866cef95-e95f-4ad4-be87-f8716fdfee7e | management-ipv4 | 192.168.204.0 | 24 | random | ['192.168.204.2-192.168.204.50'] | 192.168.204.2 | None | None | 192.168.204.1 | | fa6d8ce8-4e65-43e0-895c-326b029e68e6 | management-ipv6 | fd01:: | 64 | random | ['fd01::2-fd01::50'] | fd01::2 | None | None | fd01::1 | | 01b44ff2-0961-41a5-873c-eed95525988c | oam-ipv4 | 10.20.2.0 | 24 | random | ['10.20.2.1-10.20.2.254'] | 10.20.2.3 | None | None | 10.20.2.1 | | 8e2cc358-af2c-4bfe-bd20-32caf3b675d2 | oam-ipv6 | fd00:: | 64 | random | ['fd00::1-fd00::ffff:ffff:ffff:ffff'] | fd00::a14:203 | None | None | fd00::1 | | cfb5bea9-f12f-4ac7-b612-cc63a2fe278b | pxeboot | 169.254.202.0 | 24 | random | ['169.254.202.1-169.254.202.254'] | 169.254.202.1 | None | None | None | # In case admin and/or storage pools are configured they will also appear in the previous output: | e5f4f7b7-f388-4733-b91d-61864126cead | admin-ipv4 | 192.168.210.0 | 24 | random | ['192.168.210.2-192.168.210.50'] | 192.168.210.2 | None | None | 192.168.210.1 | | 3c5ccebb-b13b-4a89-b86e-1fdfc2e75d40 | admin-ipv6 | fdff:: | 64 | random | ['fdff::2-fdff::50'] | fdff::2 | None | None | fdff::1 | | ea9a6a9a-f96b-4d52-a22c-1922b2bb77a2 | storage-pool-ipv4 | 10.10.20.0 | 24 | random | ['10.10.20.1-10.10.20.100'] | 10.10.20.1 | None | None | None | | 47553723-b090-4324-bd6e-d4cc7e212da4 | storage-pool-ipv6 | fdfe:: | 64 | random | ['fdfe::1-fdfe::100'] | fdfe::1 | None | None | None |
Now add the node addresses for each of listed address pool entries:
### IPv4 pools POOL=$(system addrpool-list --nowrap | awk '$4 == "oam-ipv4" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address 10.20.2.4 --controller1-address 10.20.2.5 POOL=$(system addrpool-list --nowrap | awk '$4 == "management-ipv4" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address 192.168.204.3 --controller1-address 192.168.204.4 POOL=$(system addrpool-list --nowrap | awk '$4 == "cluster-host-subnet-ipv4" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address 192.168.206.2 --controller1-address 192.168.206.3 POOL=$(system addrpool-list --nowrap | awk '$4 == "pxeboot" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address 169.254.202.2 --controller1-address 169.254.202.3 # In case admin and/or storage pools are configured: POOL=$(system addrpool-list --nowrap | awk '$4 == "admin-ipv4" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address 192.168.210.3 --controller1-address 192.168.210.4 POOL=$(system addrpool-list --nowrap | awk '$4 == "storage-pool-ipv4" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address 10.10.20.2 --controller1-address 10.10.20.3 ### IPv6 pools POOL=$(system addrpool-list --nowrap | awk '$4 == "oam-ipv6" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address fd00::a14:204 --controller1-address fd00::a14:205 POOL=$(system addrpool-list --nowrap | awk '$4 == "management-ipv6" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address fd01::3 --controller1-address fd01::4 POOL=$(system addrpool-list --nowrap | awk '$4 == "cluster-host-subnet-ipv6" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address fd02::2 --controller1-address fd02::3 # In case admin and/or storage pools are configured: POOL=$(system addrpool-list --nowrap | awk '$4 == "admin-ipv6" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address fdff::3 --controller1-address fdff::4 POOL=$(system addrpool-list --nowrap | awk '$4 == "storage-pool-ipv6" { print $2 }') && \ system addrpool-modify "${POOL}" --controller0-address fdfe::2 --controller1-address fdfe::3 ### Check if the node all node addresses were added correctly: ~(keystone_admin)$ system addrpool-list --nowrap | grep -E "(oam-|management|admin|cluster-host|pxeboot|storage)" | 68817b0b-01ed-46cd-98ef-25f4d23bef9b | cluster-host-subnet-ipv4 | 192.168.206.0 | 24 | random | ['192.168.206.1-192.168.206.50'] | 192.168.206.1 | 192.168.206.2 | 192.168.206.3 | None | | 3cfd5254-a543-4b17-bfac-c2cde950f152 | cluster-host-subnet-ipv6 | fd02:: | 64 | random | ['fd02::1-fd02::50'] | fd02::1 | fd02::2 | fd02::3 | None | | 866cef95-e95f-4ad4-be87-f8716fdfee7e | management-ipv4 | 192.168.204.0 | 24 | random | ['192.168.204.2-192.168.204.50'] | 192.168.204.2 | 192.168.204.3 | 192.168.204.4 | 192.168.204.1 | | fa6d8ce8-4e65-43e0-895c-326b029e68e6 | management-ipv6 | fd01:: | 64 | random | ['fd01::2-fd01::50'] | fd01::2 | fd01::3 | fd01::4 | fd01::1 | | 01b44ff2-0961-41a5-873c-eed95525988c | oam-ipv4 | 10.20.2.0 | 24 | random | ['10.20.2.1-10.20.2.254'] | 10.20.2.3 | 10.20.2.4 | 10.20.2.5 | 10.20.2.1 | | 8e2cc358-af2c-4bfe-bd20-32caf3b675d2 | oam-ipv6 | fd00:: | 64 | random | ['fd00::1-fd00::ffff:ffff:ffff:ffff'] | fd00::a14:203 | fd00::a14:204 | fd00::a14:205 | fd00::1 | | cfb5bea9-f12f-4ac7-b612-cc63a2fe278b | pxeboot | 169.254.202.0 | 24 | random | ['169.254.202.1-169.254.202.254'] | 169.254.202.1 | 169.254.202.2 | 169.254.202.3 | None | # In case admin and/or storage pools are configured they will also appear in the previous output: | e5f4f7b7-f388-4733-b91d-61864126cead | admin-ipv4 | 192.168.210.0 | 24 | random | ['192.168.210.2-192.168.210.50'] | 192.168.210.2 | 192.168.210.3 | 192.168.210.4 | 192.168.210.1 | | 3c5ccebb-b13b-4a89-b86e-1fdfc2e75d40 | admin-ipv6 | fdff:: | 64 | random | ['fdff::2-fdff::50'] | fdff::2 | fdff::3 | fdff::4 | fdff::1 | | ea9a6a9a-f96b-4d52-a22c-1922b2bb77a2 | storage-pool-ipv4 | 10.10.20.0 | 24 | random | ['10.10.20.1-10.10.20.100'] | 10.10.20.1 | 10.10.20.2 | 10.10.20.3 | None | | 47553723-b090-4324-bd6e-d4cc7e212da4 | storage-pool-ipv6 | fdfe:: | 64 | random | ['fdfe::1-fdfe::100'] | fdfe::1 | fdfe::2 | fdfe::3 | None |
Additional Steps for Ceph Installation¶
It is necessary to add OSDs for controller-1. Both controllers must have the same number of OSDs (and also the same disk specs) to work properly. When you migrate an AIO-SX subcloud to an AIO-DX subcloud with Ceph installations, follow the additional steps below:
List host’s disks and identify disks you want to use for Ceph OSDs. Ensure that you note the UUIDs.
~(keystone_admin)$ system host-disk-list controller-1
Add disk as an OSD storage.
~(keystone_admin)$ system host-stor-add controller-1 osd <disk-uuid>
List OSD storage devices.
~(keystone_admin)$ system host-stor-list controller-1
Additional Steps for Rook Container-based Ceph Installation¶
When you migrate an AIO-SX subcloud to an AIO-DX subcloud with Rook container-based Ceph installations, follow the additional steps below:
After configuring controller-1:
Add a new Ceph monitor on controller-1.
~(keystone_admin)$ system host-fs-add controller-1 ceph=<size>
Add an OSD on controller-1.
List host’s disks and identify disks you want to use for Ceph OSDs. Ensure you note the UUIDs.
~(keystone_admin)$ system host-disk-list controller-1
Add disk as an OSD storage.
~(keystone_admin)$ system host-stor-add controller-1 osd <disk-uuid>
List OSD storage devices.
~(keystone_admin)$ system host-stor-list controller-1
After unlocking controller-1:
Wait until Ceph has two active monitors. To verify, run ceph -s and ensure that the output shows mon: 2 daemons, quorum a,b. This confirms that both the local monitors are active.
~(keystone_admin)$ ceph -s cluster: id: c55813c6-4ce5-470b-b9f5-e3c1fa0c35b1 health: HEALTH_WARN insufficient standby MDS daemons available services: mon: 2 daemons, quorum a,b (age 2m) mgr: a(active, since 114s), standbys: b mds: 1/1 daemons up osd: 4 osds: 4 up (since 46s), 4 in (since 65s)Add the floating monitor.
~(keystone_admin)$ system host-lock controller-1 ~(keystone_admin)$ system controllerfs-add ceph-float=<size> ~(keystone_admin)$ system host-unlock controller-1
Wait for the controller to reset and come back up to an operational state.
Re-apply the rook-ceph application.
~(keystone_admin)$ system application-apply rook-ceph
