Manual Kubernetes Upgrade in AIO-SX

You can upgrade the Kubernetes version on a running system from one supported version to another.

Note

Each StarlingX release supports two or more consecutive Kubernetes releases. The default version on a fresh install will always be the latest Kubernetes release supported by a StarlingX release. Upgrades from previous releases will always start with the same Kubernetes version as the highest version from the release you are upgrading from.

Kubernetes upgrades previously required sequential version updates, one at a time. The upgrade system now checks for Kubernetes version skew, allowing kubelet components to run up to three minor versions behind the control plane. This enables multi-version upgrades in a single cycle, eliminating the need to upgrade kubelet through each intermediate version. The system marks all valid versions as available, simplifying version selection and reducing the number of upgrades steps.

About this task

To complete this task, you will apply the following updates (patches) and upgrade various systems.

Platform update

The platform update contains metadata for the new Kubernetes version and the Kubernetes networking pods templates for the new Kubernetes version.

Kubernetes Component Packages

Each supported Kubernetes version is packaged separately and includes the corresponding versions of kubeadm, kubelet, and kubectl binaries. The system uses Debian packages to manage Kubernetes components.

Prerequisites

  • The system must be clear of alarms.

  • All hosts must be unlocked, enabled, and available.

  • All Kubernetes pods must be ready.

  • The installed applications must be compatible with the new Kubernetes versions that the system will upgrade to.

  • If you are using NetApp Trident in StarlingX r10.0 and if you have upgraded from the StarlingX previous version, ensure that your NetApp backend version is compatible with Trident 25.02.1. Follow the steps in Upgrade the NetApp Trident Software to upgrade Trident to 25.02.1 before upgrading Kubernetes to version 1.32.

Note

The sysadmin and admin passwords must be set to the same value prior to starting an upgrade from StarlingX Release r9 to StarlingX Release r10.

Procedure

  1. Upload, apply and install the platform update.

    For more details, see the section Updates and Upgrade section.

  2. Confirm that the system is healthy.

    Check the current system health status, resolve any alarms and other issues reported by the system health-query-kube-upgrade command, and recheck the system health status to confirm that all the System Health fields are set to OK.

    ~(keystone_admin)]$ system health-query-kube-upgrade
    System Health:
    All hosts are provisioned: [OK]
    All hosts are unlocked/enabled: [OK]
    All hosts have current configurations: [OK]
    All hosts are patch current: [OK]
    Ceph Storage Healthy: [OK]
    No alarms: [OK]
    All kubernetes nodes are ready: [OK]
    All kubernetes control plane pods are ready: [OK]
    Required patches are applied: [OK]
    License valid for upgrade: [OK]
    All kubernetes applications are in a valid state: [OK]
    Active controller is controller-0: [OK]

  3. List the available Kubernetes versions, for example:

    On a fresh install of StarlingX r10.0, the following output appears:

    ~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| Version | Target | State       |
+---------+--------+-------------+
| v1.29.2 | False  | unavailable |
| v1.30.6 | False  | unavailable |
| v1.31.5 | False  | unavailable |
| v1.32.2 | True   | active      |
+---------+--------+-------------+

    If StarlingX was upgraded to r10.0, the following output appears:

    ~(keystone_admin)]$ system kube-version-list
+---------+--------+-------------+
| Version | Target | State       |
+---------+--------+-------------+
| v1.29.2 | True   | active      |
| v1.30.6 | False  | available   |
| v1.31.5 | False  | available   |
| v1.32.2 | False  | available   |
+---------+--------+-------------+

    The following meanings apply to the output shown:

    Target

    Target is either true or false. Target will be true only for the active Kubernetes version.

    State

    State can be one of the following:

    active: The version is running everywhere.

    partial: The version is running somewhere.

    available: The version can be upgraded.

    unavailable: The version is not available for upgrading.

  4. Start the Kubernetes upgrade.

    Specify the desired target version available to upgrade.

    For example:

    ~(keystone_admin)]$ system kube-upgrade-start v1.32.2
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrade-started                      |
| created_at   | 2025-08-24T02:33:47.049826+00:00     |
| updated_at   | None                                 |
+--------------+--------------------------------------+

    The upgrade process checks the applied/available updates, the upgrade path, the system health, the installed applications compatibility, and validates that the system is ready for an upgrade.

    Warning

    The command system kube-upgrade-start --force causes the upgrade process to ignore non-management-affecting alarms. Kubernetes cannot be upgraded if there are management-affecting alarms.

  5. Download the Kubernetes images.

    For example:

    ~(keystone_admin)]$ system kube-upgrade-download-images
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | downloading-images                   |
| created_at   | 2025-08-24T02:33:47.049826+00:00     |
| updated_at   | None                                 |
+--------------+--------------------------------------+

  6. Confirm that the download has completed.

    ~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | downloaded-images                    |
| created_at   | 2025-08-24T02:33:47.049826+00:00     |
| updated_at   | 2025-08-24T02:38:16.374677+00:00     |
+--------------+--------------------------------------+

  7. Update all applications that require updating before performing the Kubernetes version upgrade. This will update all applications that have the timing: pre metadata setting.

    ~(keystone_admin)]$ system kube-pre-application-update
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | pre-updating-apps                    |
| created_at   | 2025-03-26T18:44:46.854319+00:00     |
| updated_at   | 2025-03-26T18:51:29.486448+00:00     |
+--------------+--------------------------------------+

    Note

    The state will change to pre-updated-apps when the app update has completed.

  8. Upgrade Kubernetes networking, for example:

    ~(keystone_admin)]$ system kube-upgrade-networking
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrading-networking                 |
| created_at   | 2025-08-24T02:33:47.049826+00:00     |
| updated_at   | 2025-08-24T02:38:16.374677+00:00     |
+--------------+--------------------------------------+

    The state upgraded-networking will be entered when the networking upgrade has completed.

    ~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgraded-networking                  |
| created_at   | 2025-08-24T02:33:47.049826+00:00     |
| updated_at   | 2025-08-24T02:42:40.543522+00:00     |
+--------------+--------------------------------------+

  9. Update the Kubernetes storage components to align with the target version. This ensures compatibility and stability for persistent volumes and storage-related services throughout the upgrade process.

    ~(keystone_admin)]$ system kube-upgrade-storage
 +--------------+--------------------------------------+
 | Property     | Value                                |
 +--------------+--------------------------------------+
 | uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
 | from_version | v1.29.2                              |
 | to_version   | v1.32.2                              |
 | state        | upgrading-storage                    |
 | created_at   | 2025-08-24T02:33:47.049826+00:00     |
 | updated_at   | 2025-08-24T02:42:40.543522+00:00     |
 +--------------+--------------------------------------+

    The state upgraded-storage will be entered when the storage upgrade has completed.

    ~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgraded-storage                     |
| created_at   | 2025-08-24T02:33:47.049826+00:00     |
| updated_at   | 2025-08-24T02:42:40.543522+00:00     |
+--------------+--------------------------------------+

  10. Run the kube-host-cordon command to evict the regular pods from the host. This command will prevent the application from running on intermediate versions.

    Note

    This command will permanently evict the pods which are not in namespaces.

    ~(keystone_admin)]$ system kube-host-cordon controller-0
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | cordon-started                       |
| created_at   | 2025-08-24T02:45:32.257231+00:00     |
| updated_at   | 2025-08-24T02:45:32.257231+00:00     |
+--------------+--------------------------------------+

    ~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | cordon-complete                      |
| created_at   | 2025-08-24T02:45:32.257231+00:00     |
| updated_at   | 2025-08-24T11:47:56.178266+00:00     |
+--------------+--------------------------------------+

    The state cordon-complete will be entered when the host cordon has completed.

    Note

    Repeat the following upgrade steps for each Kubernetes version (up to three minor versions) until you reach the target version or hit the version skew policy limit.

  11. Upgrade the control plane on controller-0.

    ~(keystone_admin)]$ system kube-host-upgrade controller-0 control-plane
+-----------------------+-------------------------+
| Property              | Value                   |
+-----------------------+-------------------------+
| control_plane_version | v1.29.2                 |
| hostname              | controller-0            |
| id                    | 1                       |
| kubelet_version       | v1.29.2                 |
| personality           | controller              |
| status                | upgrading-control-plane |
| target_version        | v1.32.2                 |
+-----------------------+-------------------------+

    Check if the control plane version upgrade status is changed to None. This verifies that the control plane has been successfully upgraded to the next version.

    ~(keystone_admin)]$ system kube-host-upgrade-list
+----+---------------+------------+----------------+-----------------------+-----------------+--------------+
| id | hostname      | personality| target_version | control_plane_version | kubelet_version | status       |
+----+---------------+------------+----------------+-----------------------+-----------------+--------------+
| 1  | controller-0  | controller | v1.30.6        | v1.30.6               | v1.29.2         | None         |
+----+---------------+---+--------+----------------+-----------------------+-----------------+--------------+

  12. Upgrade kubelet on controller-0.

    ~(keystone_admin)]$ system kube-host-upgrade controller-0 kubelet
+-----------------------+-------------------------+
| Property              | Value                   |
+-----------------------+-------------------------+
| control_plane_version | v1.30.6                 |
| hostname              | controller-0            |
| id                    | 1                       |
| kubelet_version       | v1.29.2                 |
| personality           | controller              |
| status                | upgrading-kubelet       |
| target_version        | v1.30.6                 |
+-----------------------+-------------------------+

    Check the status of the kubelet upgrade.

    ~(keystone_admin)]$ system kube-host-upgrade-list
+----+---------------+------------+----------------+-----------------------+-----------------+------------------+
| id | hostname      | personality| target_version | control_plane_version | kubelet_version | status           |
+----+---------------+------------+----------------+-----------------------+-----------------+------------------+
| 1  | controller-0  | controller | v1.30.6        | v1.30.6               | v1.30.6         | upgraded-kubelet |
+----+---------------+---+--------+----------------+-----------------------+-----------------+------------------+

    The status upgraded-kubelet will be entered when the kubelet upgrade has completed.

    Repeat the upgrade steps from Upgrade control plane for each version in the sequence until you reach the target Kubernetes version. This approach keeps the upgrade within the Kubernetes version skew policy, which supports up to three minor version upgrades at a time.

  13. Optional step: Run the uncordon subcommand.

    The kube-host-uncordon command will allow the regular pods on the host again.

    ~(keystone_admin)]$ system kube-host-uncordon controller-0
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | uncordon-started                     |
| created_at   | 2025-08-24T11:56:56.178266+00:00     |
| updated_at   | 2025-08-24T11:56:56.178266+00:00     |
+--------------+--------------------------------------+

~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | uncordon-complete                    |
| created_at   | 2025-08-24T11:56:56.178266+00:00     |
| updated_at   | 2025-08-24T11:58:35.136866+00:00     |
+--------------+--------------------------------------+

    The state uncordon-complete will be entered when the host uncordon has completed.

  14. Complete the Kubernetes upgrade.

    ~(keystone_admin)]$ system kube-upgrade-complete
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrade-complete                     |
| created_at   | 2025-08-24T02:33:47.049826+00:00     |
| updated_at   | 2025-08-24T02:55:18.122620+00:00     |
+--------------+--------------------------------------+

  15. Optional step: Update all applications that require updating after the K8S Version Upgrade. This step will update all applications that have the timing: post metadata setting.

    ~(keystone_admin)]$ system kube-post-application-update

  16. Delete the temporary resources associated with a Kubernetes upgrade procedure. This step also help clear Alarm 900.007 if it is still running after the upgrade.

    ~(keystone_admin)]$ system kube-upgrade-delete

Kubernetes Upgrade Abort

If you want to abort the Kubernetes upgrade after the upgrade has started, run the following command:

~(keystone_admin)]$ system kube-upgrade-abort
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrade-aborting                     |
| created_at   | 2025-06-26T18:44:46.854319+00:00     |
| updated_at   | 2025-08-24T02:55:18.122620+00:00     |
+--------------+--------------------------------------+

To check the status of the abort operation, run the following command:

~(keystone_admin)]$ system kube-upgrade-show
+--------------+--------------------------------------+
| Property     | Value                                |
+--------------+--------------------------------------+
| uuid         | bf3f9c80-0cec-49a0-91ef-dd86c9bb8fe8 |
| from_version | v1.29.2                              |
| to_version   | v1.32.2                              |
| state        | upgrade-aborted                      |
| created_at   | 2025-08-24T07:10:02.578787+00:00     |
| updated_at   | 2025-08-24T07:24:00.429794+00:00     |
+--------------+--------------------------------------+

Note

  • The upgrade abort operation reverts all the Kubernetes version upgrades and shows the same state the Kubernetes was in before the upgrade started.

  • Once the Kubernetes upgrade is completed, it cannot be aborted.