Update system-local-ca or Migrate Platform Certificates to use Cert Manager¶

The playbook described in this section can be used to either update system-local-ca or migrate platform Certificates to use Cert Manager. For updating system-local-ca, this playbook will update the system-local-ca Secret and Issuer, re-sign all the Platform certificates using this issuer, and in a Distributed Cloud environment, iterate thru all of the subclouds and do the same updates and re-signing on each Subcloud. In the migration use case, this playbook can be used to switch from using Platform certificates generated from the legacy certificate management APIs/CLIs, which will be deprecated in a future release, to the new method of configuring Platform certificates using Cert-Manager which enables auto-renewals of Platform certificates. And again in a Distributed Cloud environment will iterate thru all of the Subclouds.

About this task

This playbook can be configured to execute on any deployment configuration supported by StarlingX (AIO, standard, and distributed cloud configurations), allowing you to update system-local-ca and re-sign certificates or migrate certificates at scale.

The certificates (if they exist) that will be updated / migrated with this playbook are:

system-local-ca
system-openldap-local-certificate
REST API & Web Server certificate ( system-restapi-gui-certificate signed by system-local-ca)
Docker Registry certificate (system-registry-local-certificate signed by system-local-ca)
OIDC-Auth-Apps certificate (<user-specified> signed by system-local-ca)

Procedure

Create an inventory file using Ansible-Vault.

You must create an inventory file to specify the playbook parameters. Using ansible-vault is highly recommended in order to securely store the contents of the inventory file which includes the system-local-ca public certificate and private key, and the Root CA public certificate for system-local-ca. Ansible vault will ask for a password in this step, which is used for subsequent ansible-vault access and ansible-playbook commands.

For example:
```
~(keystone_admin)]$ ansible-vault create migration-inventory.yml
```
This will open up an editor in which you need to manually add or paste your inventory parameters, as specified in the example below.

An example migration-inventory.yaml file is shown below:
```
all:
  vars:
    system_local_ca_cert: <base64_cert>
    system_local_ca_key: <base64_key>
    system_root_ca_cert: <base64_cert>
  children:
    target_group:
      vars:
        system_platform_certificate:
          dns_domain: xyz.com
          duration: 2160h # 90d
          renewBefore: 360h # 15d
          subject_C: CA
          subject_ST: Ontario
          subject_L: Ottawa
          subject_O: myorganization
          subject_OU: engineering
          subject_CN: myorganization.com
        # SSH password to connect to all subclouds
        ansible_ssh_user: sysadmin
        ansible_ssh_pass: <sysadmin-passwd>
        # Sudo password
        ansible_become_pass: <sysadmin-passwd>
```
The inventory parameters have the following meanings:
system_local_ca_cert and system_local_ca_key
Platform issuer (system-local-ca) public certificate and private key pair, both values being the single-line base64 encoding of the corresponding pem file; i.e. the output of base64 -w0 <pem-file>.

It is highly recommended that you use an Intermediate CA system-local-ca, where the system-local-ca’s certificate is generated and signed by an external trusted Root CA. Refer to the documentation for the external trusted Root CA that you are using, on how to create an Intermediate CA public certificate and private key pair.

The system_local_ca_cert override must provide either:
- A single certificate, directly signed by the Root CA; or
- A bundle of Intermediate CA certificates where every certificate is signed by the next one, starting with system-local-ca’s public certificate and ending with the Intermediate CA certificate that is directly signed by the Root CA certificate. The Root CA should not be included in this bundle.
The system_local_ca_key override must provide only the private key for system-local-ca. Only RSA is supported for the key, which must be provided in unencrypted PEM format.

The duration of the Intermediate CA public certificate should be at least 3 years. See ica_duration to modify this semantic check.

Warning

The private key for system-local-ca should be handled carefully, avoiding any possibility of transference between the cluster and the external CA.

Note

It is recommended that the private key is generated locally and used to create a Certificate Signing Request (CSR). The CSR and its challenge password should be exchanged securely with the external CA to obtain the signed public certificate for the Intermediate CA, while the private key is never transferred.

Using OpenSSL, the private key and the CSR can be generated via the CLI using the following command:
$ openssl req -newkey rsa:4096 -keyout private_key.key -out cert.csr
You will be prompted to fill the certificate subject fields and to provide a challenge password. Refer to your external CA to generate an Intermediate CA public certificate that matches this private key that can be used to issue the internal platform certificates. The certificate should have the following extensions:
subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer basicConstraints = critical, CA:true keyUsage = critical, digitalSignature, cRLSign, keyCertSign
For example, if you are running a private PKI, you can generate the Intermediate CA certificate cert.pem from cert.csr using the following extensions:
cat <<eof> extensions.conf [v3_req] subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer basicConstraints = critical, CA:true keyUsage = critical, digitalSignature, cRLSign, keyCertSign eof $ openssl x509 -req -CA <CA-certificate-path> -CAkey <CA-private-key> -days <expiration-days> -in cert.csr -out cert.pem -extensions v3_req -extfile extensions.conf
system_root_ca_cert
The public certificate of the Root CA that signed system_local_ca_cert.

rca_duration
RCA duration validation parameter. This will be used against system_root_ca_cert to ensure that it have sufficient duration remaining. It defaults to 3 years, as this is typical for CA certificates and this certificate must be renewed manually. Only override if necessary.

ica_duration
ICA duration validation parameter. This will be used against system_local_ca_cert to ensure that it have sufficient duration remaining. It defaults to 3 years, as this is typical for CA certificates and this certificate must be renewed manually. Only override if necessary.

system_platform_certificate.dns_domain
The DNS domain that will be used to build a full DNS name for the SANs List of the Platform Certificates. E.g. starlingx-restapi-gui.<dns_domain> would appear in the SANs list of the REST API & Web Server certificate. in the server certificates.

system_platform_certificate.duration
The duration of certificate validity to use in all Platform Certificates, in hours; defaults to 2160h (or 90 days). The Platform Server Certificates will be auto-renewed by Cert-Manager.

system_platform_certificate.renewBefore
The number of hours before certificate expiry that the Platform Certificate should be auto-renewed by Cert-Manager; defaults to 360h (or 15 days).

system_platform_certificate.subject_*fields
Subject related fields that will be added to all platform certificates:
- system_platform_certificate.subject_C: country
- system_platform_certificate.subject_ST: State or Province
- system_platform_certificate.subject_L: Location
- system_platform_certificate.subject_O: Organization
- system_platform_certificate.subject_OU: Organization Unit
- system_platform_certificate.subject_CN: Common Name
ansible_ssh_user
The username to use to connect to the target system using ssh.

ansible_ssh_pass
The password to use to connect to the target system using ssh.

ansible_become_pass
The ansible_ssh_user’s sudo password.
If a separate set of overrides are required for a group of hosts, children groups can be added under target_group.

Note

The Certificate usage of Cert-manager Documentation (https://cert-manager.io/docs/usage/certificate/) states that one should “Take care when setting the renewBefore field to be very close to the duration as this can lead to a renewal loop, where the Certificate is always in the renewal period.”

In the light of the statement above, you must not set renewBefore to a value very close to the “duration” value, such as a renewBefore of 29 days and a duration of 30 days. Instead, you could set values such as renewBefore=15 days and duration=30 days to avoid renewal loops.

The following example illustrates using one set of ssh/sudo passwords for subcloud1 and subcloud2 and another set of ssh/sudo passwords for subcloud3.
```
all:
  vars:
    ...
  children:
    target_group:
      vars:
        ...
      children:
        different_password_group:
          vars:
            ansible_ssh_user: sysadmin
            ansible_ssh_pass: <sysadmin-passwd>
            ansible_become_pass: <sysadmin-passwd>
          hosts:
            subcloud1:
            subcloud2:
        different_password_group2:
          vars:
            ansible_ssh_user: sysadmin
            ansible_ssh_pass: <different-sysadmin-passwd>
            ansible_become_pass: <different-sysadmin-passwd>
          hosts:
            subcloud3:
```
Run the playbook.

Execute the Ansible playbook to start the migration process. You will be prompted for the vault password created in the previous step.

For example:
```
~(keystone_admin)]$ ansible-playbook /usr/share/ansible/stx-ansible/playbooks/update_platform_certificates.yml -i migration-inventory.yml --extra-vars "target_list=localhost,subcloud1 mode=update ignore_alarms=yes" --ask-vault-pass
```
Note
- In Distributed Cloud systems, the playbook must be run from the System Controller, and the target_list parameter should be used to target the desired subclouds.
- The target_list parameter should include localhost within the targeted subcloud, to keep the certificates consistent with the SystemController.
The behavior of the update/migration can be customized using the following --extra-vars parameter options:
mode
- update: Creates or updates platform certificates. Also supports ongoing updates, which is useful for operations such as replacing the ICA or changing other parameters.
- check: Gathers certificates from all subclouds and prints them on the system controller
target_list
- localhost: Will target the localhost (standalone systems or system controller). The target_list parameter should include this value to keep consistency with the SystemController.
- subcloud1, subcloud2: A comma separated list of hosts the playbook will target.
- all_online_subclouds: Will query dcmanager subcloud list and retrieve a list of online subclouds to target.
ignore_alarms
yes/no: When not specified defaults to no. Using ignore_alarms=yes should be avoided as much as possible. Only use it after a careful analysis of the alarm in question and for specific hosts.