TLS

This guide describes how to configure Kolla Ansible to deploy OpenStack with TLS enabled. Enabling TLS on the provided internal and/or external VIP address allows OpenStack clients to authenticate and encrypt network communication with OpenStack services.

When an OpenStack service exposes an API endpoint, Kolla Ansible will configure HAProxy for that service to listen on the internal and/or external VIP address. The HAProxy container load-balances requests on the VIPs to the nodes running the service container.

There are two different layers of TLS configuration for OpenStack APIs:

1. Enabling TLS on the internal and/or external VIP, so communication between an OpenStack client and the HAProxy listening on the VIP is secure.

2. Enabling TLS on the backend network, so communication between HAProxy and the backend API services is secure.

Note

TLS authentication is based on certificates that have been signed by trusted Certificate Authorities. Examples of commercial CAs are Comodo, Symantec, GoDaddy, and GlobalSign. Letsencrypt.org is a CA that will provide trusted certificates at no charge. If using a trusted CA is not possible for your project, you can use a private CA, e.g. Hashicorp Vault, to create a certificate for your domain, or see Generating TLS certificates with Let’s Encrypt to use a Kolla Ansible generated private CA.

For details on ACME-enabled CAs, such as letsencrypt.org, please see ACME http-01 challenge support.

Quick Start

Note

The certificates generated by Kolla Ansible use a simple Certificate Authority setup and are not suitable for a production deployment. Only certificates signed by a trusted Certificate Authority should be used in a production deployment.

To deploy OpenStack with TLS enabled for the external, internal and backend APIs, configure the following in globals.yml:

kolla_enable_tls_internal: "yes"
kolla_enable_tls_external: "yes"
kolla_enable_tls_backend: "yes"
kolla_copy_ca_into_containers: "yes"

If deploying on Debian or Ubuntu:

openstack_cacert: "/etc/ssl/certs/ca-certificates.crt"

If on CentOS or Rocky:

openstack_cacert: "/etc/pki/tls/certs/ca-bundle.crt"

The Kolla Ansible certificates command generates a private test Certificate Authority, and then uses the CA to sign the generated certificates for the enabled VIP(s) to test TLS in your OpenStack deployment. Assuming you are using the multinode inventory:

kolla-ansible -i ~/multinode certificates

TLS Configuration for internal/external VIP

The configuration variables that control TLS for the internal and/or external VIP are:

• kolla_enable_tls_external

• kolla_enable_tls_internal

• kolla_internal_fqdn_cert

• kolla_external_fqdn_cert

Note

If TLS is enabled only on the internal or external network, then kolla_internal_vip_address and kolla_external_vip_address must be different.

If there is only a single network configured in your topology (as opposed to separate internal and external networks), TLS can only be enabled using the internal network configuration variables.

The default state for TLS networking is disabled. To enable external TLS encryption:

kolla_enable_tls_external: "yes"

To enable internal TLS encryption:

kolla_enable_tls_internal: "yes"

Two certificate files are required to use TLS securely with authentication, which will be provided by your Certificate Authority:

• server certificate with private key

• CA certificate with any intermediate certificates

The combined server certificate and private key needs to be provided to Kolla Ansible, with the path configured via kolla_external_fqdn_cert or kolla_internal_fqdn_cert. These paths default to {{ kolla_certificates_dir }}/haproxy.pem and {{ kolla_certificates_dir }}/haproxy-internal.pem respectively, where kolla_certificates_dir is /etc/kolla/certificates by default.

If the server certificate provided is not already trusted by clients, then the CA certificate file will need to be distributed to the clients. This is discussed in more detail in Configuring the OpenStack Client for TLS and Adding CA Certificates to the Service Containers.

Configuring the OpenStack Client for TLS

The location for the CA certificate for the admin-openrc.sh file is configured with the kolla_admin_openrc_cacert variable, which is not set by default. This must be a valid path on all hosts where admin-openrc.sh is used.

When TLS is enabled on a VIP, and kolla_admin_openrc_cacert is set to /etc/pki/tls/certs/ca-bundle.crt, an OpenStack client will have settings similar to this configured by admin-openrc.sh:

export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin
export OS_PASSWORD=demoPassword
export OS_AUTH_URL=https://mykolla.example.net:5000
export OS_INTERFACE=internal
export OS_ENDPOINT_TYPE=internalURL
export OS_MISTRAL_ENDPOINT_TYPE=internalURL
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_AUTH_PLUGIN=password
# os_cacert is optional for trusted certificates
export OS_CACERT=/etc/pki/tls/certs/ca-bundle.crt

Adding CA Certificates to the Service Containers

To copy CA certificate files to the service containers:

kolla_copy_ca_into_containers: "yes"

When kolla_copy_ca_into_containers is configured to “yes”, the CA certificate files in /etc/kolla/certificates/ca will be copied into service containers to enable trust for those CA certificates. This is required for any certificates that are either self-signed or signed by a private CA, and are not already present in the service image trust store. Kolla will install these certificates in the container system wide trust store when the container starts.

All certificate file names will have the kolla-customca- prefix prepended to them when they are copied into the containers. For example, if a certificate file is named internal.crt, it will be named kolla-customca-internal.crt in the containers.

For Debian and Ubuntu containers, the certificate files will be copied to the /usr/local/share/ca-certificates/ directory.

For CentOS and Rocky containers, the certificate files will be copied to the /etc/pki/ca-trust/source/anchors/ directory.

In both cases, valid certificates will be added to the system trust store - /etc/ssl/certs/ca-certificates.crt on Debian and Ubuntu, and /etc/pki/tls/certs/ca-bundle.crt on CentOS and Rocky.

Configuring a CA bundle

OpenStack services do not always trust CA certificates from the system trust store by default. To resolve this, the openstack_cacert variable should be configured with the path to the CA Certificate in the container.

To use the system trust store on Debian or Ubuntu:

openstack_cacert: /etc/ssl/certs/ca-certificates.crt

For CentOS or Rocky:

openstack_cacert: /etc/pki/tls/certs/ca-bundle.crt

Back-end TLS Configuration

Enabling TLS on the backend services secures communication between the HAProxy listing on the internal/external VIP and the OpenStack services. It also enables secure end-to-end communication between OpenStack services that support TLS termination. The OpenStack services that support backend TLS termination in Victoria are: Nova, Ironic, Neutron, Keystone, Glance, Heat, Placement, Horizon, Barbican, and Cinder.

The configuration variables that control back-end TLS for service endpoints are:

• kolla_enable_tls_backend

• kolla_tls_backend_cert

• kolla_tls_backend_key

• haproxy_backend_cacert

• haproxy_backend_cacert_dir

The default state for back-end TLS is disabled. To enable TLS for the back-end communication:

kolla_enable_tls_backend: "yes"

It is also possible to enable back-end TLS on a per-service basis. For example, to enable back-end TLS for Keystone, set keystone_enable_tls_backend to yes.

The default values for haproxy_backend_cacert and haproxy_backend_cacert_dir should suffice if the certificate is in the system trust store. Otherwise, they should be configured to a location of the CA certificate installed in the service containers.

Each backend service requires a certificate and private key. In many cases it is necessary to use a separate certificate and key for each host, or even per-service. The following precedence is used for the certificate:

• {{ kolla_certificates_dir }}/{{ inventory_hostname }}/{{ project_name }}-cert.pem

• {{ kolla_certificates_dir }}/{{ inventory_hostname }}-cert.pem

• {{ kolla_certificates_dir }}/{{ project_name }}-cert.pem

• {{ kolla_tls_backend_cert }}

And for the private key:

• {{ kolla_certificates_dir }}/{{ inventory_hostname }}/{{ project_name }}-key.pem

• {{ kolla_certificates_dir }}/{{ inventory_hostname }}-key.pem

• {{ kolla_certificates_dir }}/{{ project_name }}-key.pem

• {{ kolla_tls_backend_key }}

The default for kolla_certificates_dir is /etc/kolla/certificates.

kolla_tls_backend_cert and kolla_tls_backend_key, default to {{ kolla_certificates_dir }}/backend-cert.pem and {{ kolla_certificates_dir }}/backend-key.pem respectively.

project_name is the name of the OpenStack service, e.g. keystone or cinder.

Note

The back-end TLS cert/key can be the same certificate that is used for the VIP, as long as those certificates are configured to allow requests from both the VIP and internal networks.

By default, the TLS certificate will be verified as trustable by the OpenStack services. Although not recommended for production, it is possible to disable verification of the backend certificate:

kolla_verify_tls_backend: "no"

Generating TLS certificates with Let’s Encrypt

Let’s Encrypt is a free, automated, and open certificate authority.

To enable OpenStack to deploy the Let’s Encrypt container to fetch certificates from the Let’s Encrypt certificate authority, the following must be configured in globals.yml:

enable_letsencrypt: "yes"
letsencrypt_email: "<The email used for registration and recovery contact>"

The Let’s Encrypt container will attempt to renew your certificates every 12 hours. If the certificates are renewed, they will automatically be deployed to the HAProxy containers using SSH.

Note

If letsencrypt_email is not valid email, letsencrypt role will not work correctly.

Note

If enable_letsencrypt is set to true, haproxy’s socket will run with admin access level. This is needed so Let’s Encrypt can interact with HAProxy.

You can configure separate ACME servers for internal and external certificate requests.

letsencrypt_external_cert_server: "<ACME server URL for external cert>"
letsencrypt_internal_cert_server: "<ACME server URL for internal cert>"

Note

The letsencrypt_external_cert_server has a default value of https://acme-v02.api.letsencrypt.org/directory. Ensure that letsencrypt_internal_cert_server is reachable from the controller if you configure it for internal certificate requests.

Generating a Private Certificate Authority

Note

The certificates generated by Kolla Ansible use a simple Certificate Authority setup and are not suitable for a production deployment. Only certificates signed by a trusted Certificate Authority should be used in a production deployment.

It’s not always practical to get a certificate signed by a trusted CA. In a development or internal test OpenStack deployment, it can be useful to generate certificates locally to enable TLS.

For convenience, the kolla-ansible command will generate the necessary certificate files based on the information in the globals.yml configuration file and the inventory file:

kolla-ansible -i multinode certificates

The certificates role performs the following actions:

1. Generates a test root Certificate Authority

2. Generates the internal/external certificates which are signed by the root CA.

3. If back-end TLS is enabled, generate the back-end certificate signed by the root CA.

The combined certificate and key file haproxy.pem (which is the default value for kolla_external_fqdn_cert) will be generated and stored in the /etc/kolla/certificates/ directory, and a copy of the CA certificate (root.crt) will be stored in the /etc/kolla/certificates/ca/ directory.

Generating your certificates without kolla-ansible

If you want to manage your TLS certificates outside kolla-ansible directly on your hosts, you can do it by setting kolla_externally_managed_cert to true. This will make kolla-ansible ignore any copy of certificate from the operator to kolla-ansible managed hosts and will keep other configuration options for TLS as is.

If using this option, make sure that all certificates are present on the appropriate hosts in the appropriate location.

HAProxy TLS related settings

You can select between different SSL/TLS ciphers by setting the following in /etc/kolla/globals.yml:

kolla_haproxy_ssl_settings: "modern" # or "intermediate" or "legacy"

The default value is “modern”. These settings are adapted from the Mozilla SSL Configuration Generator.

The setting “modern” is recommended for most deployments. The setting “intermediate” is recommended for deployments that need to support older clients. The setting “legacy” is not recommended, but is left as a compatibility option for older deployments.

See the Mozilla SSL Configuration Generator for more information on exact supported client versions.

The kolla_haproxy_ssl_settings setting also affects the glance and neutron haproxy TLS settings, if these proxy services are enabled.