Object Encryption¶

Encryption Root Secret in External Key Management System¶

The benefits of using a dedicated system for storing the encryption root secret include the auditing and access control infrastructure that are already in place in such a system, and the fact that an encryption root secret stored in a key management system (KMS) may be backed by a hardware security module (HSM) for additional security. Another significant benefit of storing the root encryption secret in an external KMS is that it is in this case never stored on a disk in the Swift cluster.

Make sure the required dependencies are installed for retrieving an encryption root secret from an external KMS. This can be done when installing Swift (add the -e flag to install as a development version) by changing to the Swift directory and running the following command to install Swift together with the kms_keymaster extra dependencies:

sudo pip install .[kms_keymaster]

Another way to install the dependencies is by making sure the following lines exist in the requirements.txt file, and installing them using pip install -r requirements.txt:

cryptography>=1.6                       # BSD/Apache-2.0
castellan>=0.6.0

To make use of an encryption root secret stored in an external KMS, replace the keymaster middleware with the kms_keymaster middleware in the proxy server WSGI pipeline in proxy-server.conf, in the order shown in this example:

<other middleware> kms_keymaster encryption proxy-logging proxy-server

and add a section to the same file:

[filter:kms_keymaster]
use = egg:swift#kms_keymaster
keymaster_config_path = file_with_kms_keymaster_config

Create or edit the file file_with_kms_keymaster_config referenced above. For further details on the middleware configuration options, see the keymaster.conf-sample file. An example of the content of this file, with optional parameters omitted, is below:

[kms_keymaster]
key_id = changeme
username = swift
password = password
project_name = swift
auth_endpoint = http://keystonehost:5000/v3

The encryption root secret shall be created and stored in the external key management system before it can be used by the keymaster. It shall be stored as a symmetric key, with content type application/octet-stream, base64 content encoding, AES algorithm, bit length 256, and secret type symmetric. The mode ctr may also be stored for informational purposes - it is not currently checked by the keymaster.

The following command can be used to store the currently configured encryption_root_secret value from the proxy-server.conf file in Barbican:

openstack secret store --name swift_root_secret \
--payload-content-type="application/octet-stream" \
--payload-content-encoding="base64" --algorithm aes --bit-length 256 \
--mode ctr --secret-type symmetric --payload <base64_encoded_root_secret>

Alternatively, the existing root secret can also be stored in Barbican using curl.

Instead of using an existing root secret, Barbican can also be asked to generate a new 256-bit root secret, with content type application/octet-stream and algorithm AES (the mode parameter is currently optional):

openstack secret order create --name swift_root_secret \
--payload-content-type="application/octet-stream" --algorithm aes \
--bit-length 256 --mode ctr key

The order create creates an asynchronous request to create the actual secret. The order can be retrieved using openstack secret order get, and once the order completes successfully, the output will show the key id of the generated root secret. Keys currently stored in Barbican can be listed using the openstack secret list command.

The keymaster uses the explicitly configured username and password (and project name etc.) from the keymaster.conf file for retrieving the encryption root secret from an external key management system. The Castellan library is used to communicate with Barbican.

For the proxy server, reading the encryption root secret directly from the proxy-server.conf file, from the keymaster.conf file pointed to from the proxy-server.conf file, or from an external key management system such as Barbican, are all functionally equivalent. In case reading the encryption root secret from the external key management system fails, the proxy server will not start up. If the encryption root secret is retrieved successfully, it is cached in memory in the proxy server.

For further details on the configuration options, see the [filter:kms_keymaster] section in the proxy-server.conf-sample file, and the keymaster.conf-sample file.

Upgrade Considerations¶

When upgrading an existing cluster to deploy encryption, the following sequence of steps is recommended:

1. Upgrade all object servers
2. Upgrade all proxy servers
3. Add keymaster and encryption middlewares to every proxy server’s middleware pipeline with the encryption disable_encryption option set to True and the keymaster encryption_root_secret value set as described above.
4. If required, follow the steps for Container sync configuration.
5. Finally, change the encryption disable_encryption option to False

Objects that existed in the cluster prior to the keymaster and encryption middlewares being deployed are still readable with GET and HEAD requests. The content of those objects will not be encrypted unless they are written again by a PUT or COPY request. Any user metadata of those objects will not be encrypted unless it is written again by a PUT, POST or COPY request.

Disabling Encryption¶

Once deployed, the keymaster and encryption middlewares should not be removed from the pipeline. To do so will cause encrypted object data and/or metadata to be returned in response to GET or HEAD requests for objects that were previously encrypted.

Encryption of inbound object data may be disabled by setting the encryption disable_encryption option to True, in which case existing encrypted objects will remain encrypted but new data written with PUT, POST or COPY requests will not be encrypted. The keymaster and encryption middlewares should remain in the pipeline even when encryption of new objects is not required. The encryption middleware is needed to handle GET requests for objects that may have been previously encrypted. The keymaster is needed to provide keys for those requests.

Container sync configuration¶

If container sync is being used then the keymaster and encryption middlewares must be added to the container sync internal client pipeline. The following configuration steps are required:

1. Create a custom internal client configuration file for container sync (if one is not already in use) based on the sample file internal-client.conf-sample. For example, copy internal-client.conf-sample to /etc/swift/container-sync-client.conf.

2. Modify this file to include the middlewares in the pipeline in the same way as described above for the proxy server.

3. Modify the container-sync section of all container server config files to point to this internal client config file using the internal_client_conf_path option. For example:

   internal_client_conf_path = /etc/swift/container-sync-client.conf
   

Encryption scheme¶

Plaintext data is encrypted to ciphertext using the AES cipher with 256-bit keys implemented by the python cryptography package. The cipher is used in counter (CTR) mode so that any byte or range of bytes in the ciphertext may be decrypted independently of any other bytes in the ciphertext. This enables very simple handling of ranged GETs.

In general an item of unencrypted data, plaintext, is transformed to an item of encrypted data, ciphertext:

ciphertext = E(plaintext, k, iv)

where E is the encryption function, k is an encryption key and iv is a unique initialization vector (IV) chosen for each encryption context. For example, the object body is one encryption context with a randomly chosen IV. The IV is stored as metadata of the encrypted item so that it is available for decryption:

plaintext = D(ciphertext, k, iv)

where D is the decryption function.

The implementation of CTR mode follows NIST SP800-38A, and the full IV passed to the encryption or decryption function serves as the initial counter block.

In general any encrypted item has accompanying crypto-metadata that describes the IV and the cipher algorithm used for the encryption:

crypto_metadata = {"iv": <16 byte value>,
                   "cipher": "AES_CTR_256"}

This crypto-metadata is stored either with the ciphertext (for user metadata and etags) or as a separate header (for object bodies).

Key management¶

A keymaster middleware is responsible for providing the keys required for each encryption and decryption operation. Two keys are required when handling object requests: a container key that is uniquely associated with the container path and an object key that is uniquely associated with the object path. These keys are made available to the encryption middleware via a callback function that the keymaster installs in the WSGI request environ.

The current keymaster implementation derives container and object keys from the encryption_root_secret in a deterministic way by constructing a SHA256 HMAC using the encryption_root_secret as a key and the container or object path as a message, for example:

object_key = HMAC(encryption_root_secret, "/a/c/o")

Other strategies for providing object and container keys may be employed by future implementations of alternative keymaster middleware.

During each object PUT, a random key is generated to encrypt the object body. This random key is then encrypted using the object key provided by the keymaster. This makes it safe to store the encrypted random key alongside the encrypted object data and metadata.

This process of key wrapping enables more efficient re-keying events when the object key may need to be replaced and consequently any data encrypted using that key must be re-encrypted. Key wrapping minimizes the amount of data encrypted using those keys to just other randomly chosen keys which can be re-wrapped efficiently without needing to re-encrypt the larger amounts of data that were encrypted using the random keys.

Encryption middleware¶

The encryption middleware is composed of an encrypter component and a decrypter component.

X-Object-Meta-Private1: value1
X-Object-Meta-Private2: value2

X-Object-Transient-Sysmeta-Crypto-Meta-Private1:
  E(value1, object_key, header_iv_1); swift_meta={"iv": header_iv_1,
                                                  "cipher": "AES_CTR_256"}
X-Object-Transient-Sysmeta-Crypto-Meta-Private2:
  E(value2, object_key, header_iv_2); swift_meta={"iv": header_iv_2,
                                                  "cipher": "AES_CTR_256"}

body_ciphertext = E(body_plaintext, body_key, body_iv)

wrapped_body_key = E(body_key, object_key, body_key_iv)

X-Object-Sysmeta-Crypto-Body-Meta:
    {"iv": body_iv,
     "cipher": "AES_CTR_256",
     "body_key": {"key": wrapped_body_key,
                  "iv": body_key_iv}}

X-Object-Sysmeta-Crypto-Etag:
  E(md5(plaintext), object_key, etag_iv); swift_meta={"iv": etag_iv,
                                                      "cipher": "AES_CTR_256"}

X-Object-Sysmeta-Container-Update-Override-Etag:
    E(md5(plaintext), container_key, override_etag_iv);
    meta={"iv": override_etag_iv, "cipher": "AES_CTR_256"}

X-Object-Sysmeta-Crypto-Etag-Mac: HMAC(object_key, md5(plaintext))

If-Match: match_etag

If-Match: match_etag,HMAC(object_key, match_etag)
X-Backend-Etag-Is-At: X-Object-Sysmeta-Crypto-Etag-Mac

Impact on other Swift services and features¶

Encryption has no impact on Object Versioning other than that any previously unencrypted objects will be encrypted as they are copied to or from the versions container. Keymaster and encryption middlewares should be placed after versioned_writes in the proxy server pipeline, as described in Deployment and operation.

Container Sync uses an internal client to GET objects that are to be sync’d. This internal client must be configured to use the keymaster and encryption middlewares as described above.

Encryption has no impact on the object-auditor service. Since the ETag header saved with the object at rest is the md5 sum of the encrypted object body then the auditor will verify that encrypted data is valid.

Encryption has no impact on the object-expirer service. X-Delete-At and X-Delete-After headers are not encrypted.

Encryption has no impact on the object-replicator and object-reconstructor services. These services are unaware of the object or EC fragment data being encrypted.

Encryption has no impact on the container-reconciler service. The container-reconciler uses an internal client to move objects between different policy rings. The destination object has the same URL as the source object and the object is moved without re-encryption.

Considerations for developers¶

Developers should be aware that keymaster and encryption middlewares rely on the path of an object remaining unchanged. The included keymaster derives keys for containers and objects based on their paths and the encryption_root_secret. The keymaster does not rely on object metadata to inform its generation of keys for GET and HEAD requests because when handling Conditional Requests it is required to provide the object key before any metadata has been read from the object.

Developers should therefore give careful consideration to any new features that would relocate object data and metadata within a Swift cluster by means that do not cause the object data and metadata to pass through the encryption middlewares in the proxy pipeline and be re-encrypted.

The crypto-metadata associated with each encrypted item does include some key_id metadata that is provided by the keymaster and contains the path used to derive keys. This key_id metadata is persisted in anticipation of future scenarios when it may be necessary to decrypt an object that has been relocated without re-encrypting, in which case the metadata could be used to derive the keys that were used for encryption. However, this alone is not sufficient to handle conditional requests and to decrypt container listings where objects have been relocated, and further work will be required to solve those issues.