Ceph - upgrade monolithic ceph-osd chart to multiple ceph charts

This document captures the steps to move from installed monolithic ceph-osd chart to mutlitple ceph osd charts.

this work will bring flexibility on site update as we will have more control on osds.

Install single ceph-osd chart:

step 1: Setup:

  • Follow OSH single node or multinode guide to bring up OSH environment.

Note

we will have single ceph osd chart and here are the override values for ceph disks
osd:
  • data:

    type: block-logical location: /dev/vdb

    journal:

    type: block-logical location: /dev/vda1

  • data:

    type: block-logical location: /dev/vdc

    journal:

    type: block-logical location: /dev/vda2

Step 2: Setup the OSH environment and check ceph cluster health

Note

Make sure we have healthy ceph cluster running

Ceph status:

ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph -s
    cluster:
      id:     61a4e07f-8b4a-4c47-8fc7-a0e7345ac0b0
      health: HEALTH_OK

    services:
       mon: 3 daemons, quorum k8smaster,k8sslave1,k8sslave2
       mgr: k8sslave2(active), standbys: k8sslave1
       mds: cephfs-1/1/1 up  {0=mds-ceph-mds-5bf9fdfc6b-8nq4p=up:active}, 1 up:standby
       osd: 6 osds: 6 up, 6 in
    data:
       pools:   18 pools, 186 pgs
       objects: 377  objects, 1.2 GiB
       usage:   4.2 GiB used, 116 GiB / 120 GiB avail
       pgs:     186 active+clean

  • Ceph cluster is in HEALTH_OK state with 3 MONs and 6 OSDs.

Note

Make sure we have single ceph osd chart only

Helm status:

ubuntu@k8smaster:/opt/openstack-helm$  helm list | grep -i osd
ceph-osd            1     Tue Mar 26 03:21:07 2019        DEPLOYED        ceph-osd-vdb-0.1.0

  • single osd chart deployed sucessfully.

Upgrade to multiple ceph osd charts:

step 1: setup

  • create multiple ceph osd charts as per requirement

Note

copy ceph-osd folder to multiple ceph osd charts in openstack-helm-infra folder

ubuntu@k8smaster:/opt/openstack-helm-infra$  cp -r ceph-osd ceph-osd-vdb
ubuntu@k8smaster:/opt/openstack-helm-infra$  cp -r ceph-osd ceph-osd-vdc

Note

make sure to correct chart name in each osd chart folder created above, need to update it in Charts.yaml .

  • create script to install multiple ceph osd charts

Note

create new installation scripts to reflect new ceph osd charts.

ubuntu@k8smaster:/opt/openstack-helm$  cp ./tools/deployment/multinode/030-ceph.sh
./tools/deployment/multinode/030-ceph-osd-vdb.sh

ubuntu@k8smaster:/opt/openstack-helm$  cp ./tools/deployment/multinode/030-ceph.sh
./tools/deployment/multinode/030-ceph-osd-vdc.sh

Note

make sure to delete all other ceph charts from above scripts and have only new ceph osd chart. and also have correct overrides as shown below.

example1: for CHART in ceph-osd-vdb; do helm upgrade –install ${CHART} ${OSH_INFRA_PATH}/${CHART} –namespace=ceph –values=/tmp/ceph.yaml ${OSH_EXTRA_HELM_ARGS} ${OSH_EXTRA_HELM_ARGS_CEPH_DEPLOY}

osd:
  • data:

    type: block-logical location: /dev/vdb

    journal:

    type: block-logical location: /dev/vda1

example2: for CHART in ceph-osd-vdc; do helm upgrade –install ${CHART} ${OSH_INFRA_PATH}/${CHART} –namespace=ceph –values=/tmp/ceph.yaml ${OSH_EXTRA_HELM_ARGS} ${OSH_EXTRA_HELM_ARGS_CEPH_DEPLOY}

osd:
  • data:

    type: block-logical location: /dev/vdc

    journal:

    type: block-logical location: /dev/vda2

step 2: Scale down applications using ceph pvc

Note

Scale down all the applications who are using pvcs so that we will not have any writes on ceph rbds .

ubuntu@k8smaster:/opt/openstack-helm$  sudo kubectl scale statefulsets -n openstack
mariadb-server --replicas=0

ubuntu@k8smaster:/opt/openstack-helm$  sudo kubectl scale statefulsets -n openstack
rabbitmq-rabbitmq --replicas=0

  • just gave one example but we need to do it for all the applications using pvcs

step 3: Setup ceph cluster flags to prevent rebalance

Note

setup few flags on ceph cluster to prevent rebalance during this process.

ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd set
noout

ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd set
nobackfill

ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd set
norecover

ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd set
pause

step 4: Delete single ceph-osd chart

Note

Delete the single ceph-osd chart.

ubuntu@k8smaster:/opt/openstack-helm$ helm delete --purge ceph-osd

step 5: install new ceph-osd charts

Note

Now we can install multiple ceph osd releases.

ubuntu@k8smaster:/opt/openstack-helm$ ./tools/deployment/multinode/030-ceph-osd-vdb.sh
ubuntu@k8smaster:/opt/openstack-helm$ ./tools/deployment/multinode/030-ceph-osd-vdc.sh
ubuntu@k8smaster:/opt/openstack-helm# helm list | grep -i osd
ceph-osd-vdb            1            Tue Mar 26 03:21:07 2019        DEPLOYED  ceph-osd-vdb-0.1.0
ceph-osd-vdc            1            Tue Mar 26 03:22:13 2019        DEPLOYED  ceph-osd-vdc-0.1.0

  • wait and check for healthy ceph cluster , if there are any issues need to sort out until we see healthy ceph cluster.

step 6: Unset ceph cluster flags

Note

unset the flags we set on the ceph cluster in above steps.

ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd unset
noout

ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd unset
nobackfill

ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd unset
norecover

ubuntu@k8smaster:/opt/openstack-helm$ kubectl exec -n ceph ceph-mon-5qn68 -- ceph osd unset
pause

step 7: Scale up the applications using pvc

Note

Since ceph cluster is back to healthy status, now scale up the applications.

ubuntu@k8smaster:/opt/openstack-helm$  sudo kubectl scale statefulsets -n openstack
mariadb-server --replicas=3

ubuntu@k8smaster:/opt/openstack-helm$  sudo kubectl scale statefulsets -n openstack
rabbitmq-rabbitmq --replicas=3