Object Encryption

Swift supports the optional encryption of object data at rest on storage nodes. The encryption of object data is intended to mitigate the risk of users’ data being read if an unauthorised party were to gain physical access to a disk.

Note

Swift’s data-at-rest encryption accepts plaintext object data from the client, encrypts it in the cluster, and stores the encrypted data. This protects object data from inadvertently being exposed if a data drive leaves the Swift cluster. If a user wishes to ensure that the plaintext data is always encrypted while in transit and in storage, it is strongly recommended that the data be encrypted before sending it to the Swift cluster. Encrypting on the client side is the only way to ensure that the data is fully encrypted for its entire lifecycle.

Encryption of data at rest is implemented by middleware that may be included in the proxy server WSGI pipeline. The feature is internal to a Swift cluster and not exposed through the API. Clients are unaware that data is encrypted by this feature internally to the Swift service; internally encrypted data should never be returned to clients via the Swift API.

The following data are encrypted while at rest in Swift:

• Object content i.e. the content of an object PUT request’s body

• The entity tag (ETag) of objects that have non-zero content

• All custom user object metadata values i.e. metadata sent using X-Object-Meta- prefixed headers with PUT or POST requests

Any data or metadata not included in the list above are not encrypted, including:

• Account, container and object names

• Account and container custom user metadata values

• All custom user metadata names

• Object Content-Type values

• Object size

• System metadata

Note

This feature is intended to provide confidentiality of data that is at rest i.e. to protect user data from being read by an attacker that gains access to disks on which object data is stored.

This feature is not intended to prevent undetectable modification of user data at rest.

This feature is not intended to protect against an attacker that gains access to Swift’s internal network connections, or gains access to key material or is able to modify the Swift code running on Swift nodes.

Deployment and operation

Encryption is deployed by adding two middleware filters to the proxy server WSGI pipeline and including their respective filter configuration sections in the proxy-server.conf file. Additional steps are required if the container sync feature is being used.

The keymaster and encryption middleware filters must be to the right of all other middleware in the pipeline apart from the final proxy-logging middleware, and in the order shown in this example:

<other middleware> keymaster encryption proxy-logging proxy-server

[filter:keymaster]
use = egg:swift#keymaster
encryption_root_secret = your_secret

[filter:encryption]
use = egg:swift#encryption
# disable_encryption = False

See the proxy-server.conf-sample file for further details on the middleware configuration options.

Keymaster middleware

The keymaster middleware must be configured with a root secret before it is used. By default the keymaster middleware will use the root secret configured using the encryption_root_secret option in the middleware filter section of the proxy-server.conf file, for example:

[filter:keymaster]
use = egg:swift#keymaster
encryption_root_secret = your_secret

Root secret values MUST be at least 44 valid base-64 characters and should be consistent across all proxy servers. The minimum length of 44 has been chosen because it is the length of a base-64 encoded 32 byte value.

Note

The encryption_root_secret option holds the master secret key used for encryption. The security of all encrypted data critically depends on this key and it should therefore be set to a high-entropy value. For example, a suitable encryption_root_secret may be obtained by base-64 encoding a 32 byte (or longer) value generated by a cryptographically secure random number generator.

The encryption_root_secret value is necessary to recover any encrypted data from the storage system, and therefore, it must be guarded against accidental loss. Its value (and consequently, the proxy-server.conf file) should not be stored on any disk that is in any account, container or object ring.

The encryption_root_secret value should not be changed once deployed. Doing so would prevent Swift from properly decrypting data that was encrypted using the former value, and would therefore result in the loss of that data.

One method for generating a suitable value for encryption_root_secret is to use the openssl command line tool:

openssl rand -base64 32

Separate keymaster configuration file

The encryption_root_secret option may alternatively be specified in a separate config file at a path specified by the keymaster_config_path option, for example:

[filter:keymaster]
use = egg:swift#keymaster
keymaster_config_path = /etc/swift/keymaster.conf

This has the advantage of allowing multiple processes which need to be encryption-aware (for example, proxy-server and container-sync) to share the same config file, ensuring that consistent encryption keys are used by those processes. It also allows the keymaster configuration file to have different permissions than the proxy-server.conf file.

A separate keymaster config file should have a [keymaster] section containing the encryption_root_secret option:

[keymaster]
encryption_root_secret = your_secret

Note

Alternative keymaster middleware is available to retrieve encryption root secrets from an external key management system such as Barbican rather than storing root secrets in configuration files.

Once deployed, the encryption filter will by default encrypt object data and metadata when handling PUT and POST requests and decrypt object data and metadata when handling GET and HEAD requests. COPY requests are transformed into GET and PUT requests by the Server Side Copy middleware before reaching the encryption middleware and as a result object data and metadata is decrypted and re-encrypted when copied.

Changing the encryption root secret

From time to time it may be desirable to change the root secret that is used to derive encryption keys for new data written to the cluster. The keymaster middleware allows alternative root secrets to be specified in its configuration using options of the form:

encryption_root_secret_<secret_id> = <secret value>

where secret_id is a unique identifier for the root secret and secret value is a value that meets the requirements for a root secret described above.

Only one root secret is used to encrypt new data at any moment in time. This root secret is specified using the active_root_secret_id option. If specified, the value of this option should be one of the configured root secret secret_id values; otherwise the value of encryption_root_secret will be taken as the default active root secret.

Note

The active root secret is only used to derive keys for new data written to the cluster. Changing the active root secret does not cause any existing data to be re-encrypted.

Existing encrypted data will be decrypted using the root secret that was active when that data was written. All previous active root secrets must therefore remain in the middleware configuration in order for decryption of existing data to succeed. Existing encrypted data will reference previous root secret by the secret_id so it must be kept consistent in the configuration.

Note

Do not remove or change any previously active <secret value> or <secret_id>.

For example, the following keymaster configuration file specifies three root secrets, with the value of encryption_root_secret_2 being the current active root secret:

[keymaster]
active_root_secret_id = 2
encryption_root_secret = your_secret
encryption_root_secret_1 = your_secret_1
encryption_root_secret_2 = your_secret_2

Note

To ensure there is no loss of data availability, deploying a new key to your cluster requires a two-stage config change. First, add the new key to the encryption_root_secret_<secret_id> option and restart the proxy-server. Do this for all proxies. Next, set the active_root_secret_id option to the new secret id and restart the proxy. Again, do this for all proxies. This process ensures that all proxies will have the new key available for decryption before any proxy uses it for encryption.

Encryption middleware

Once deployed, the encryption filter will by default encrypt object data and metadata when handling PUT and POST requests and decrypt object data and metadata when handling GET and HEAD requests. COPY requests are transformed into GET and PUT requests by the Server Side Copy middleware before reaching the encryption middleware and as a result object data and metadata is decrypted and re-encrypted when copied.

Encryption Root Secret in External Key Management System

The benefits of using a dedicated system for storing the encryption root secret include the auditing and access control infrastructure that are already in place in such a system, and the fact that an encryption root secret stored in a key management system (KMS) may be backed by a hardware security module (HSM) for additional security. Another significant benefit of storing the root encryption secret in an external KMS is that it is in this case never stored on a disk in the Swift cluster.

Swift supports fetching encryption root secrets from a Barbican service or a KMIP service using the kms_keymaster or kmip_keymaster middleware respectively.

Encryption Root Secret in a Barbican KMS

Make sure the required dependencies are installed for retrieving an encryption root secret from an external KMS. This can be done when installing Swift (add the -e flag to install as a development version) by changing to the Swift directory and running the following command to install Swift together with the kms_keymaster extra dependencies:

sudo pip install .[kms_keymaster]

Another way to install the dependencies is by making sure the following lines exist in the requirements.txt file, and installing them using pip install -r requirements.txt:

cryptography>=1.6                       # BSD/Apache-2.0
castellan>=0.6.0

Note

If any of the required packages is already installed, the --upgrade flag may be required for the pip commands in order for the required minimum version to be installed.

To make use of an encryption root secret stored in an external KMS, replace the keymaster middleware with the kms_keymaster middleware in the proxy server WSGI pipeline in proxy-server.conf, in the order shown in this example:

<other middleware> kms_keymaster encryption proxy-logging proxy-server

and add a section to the same file:

[filter:kms_keymaster]
use = egg:swift#kms_keymaster
keymaster_config_path = file_with_kms_keymaster_config

Create or edit the file file_with_kms_keymaster_config referenced above. For further details on the middleware configuration options, see the keymaster.conf-sample file. An example of the content of this file, with optional parameters omitted, is below:

[kms_keymaster]
key_id = changeme
username = swift
password = password
project_name = swift
auth_endpoint = http://keystonehost:5000/v3

The encryption root secret shall be created and stored in the external key management system before it can be used by the keymaster. It shall be stored as a symmetric key, with content type application/octet-stream, base64 content encoding, AES algorithm, bit length 256, and secret type symmetric. The mode ctr may also be stored for informational purposes - it is not currently checked by the keymaster.

The following command can be used to store the currently configured encryption_root_secret value from the proxy-server.conf file in Barbican:

openstack secret store --name swift_root_secret \
--payload-content-type="application/octet-stream" \
--payload-content-encoding="base64" --algorithm aes --bit-length 256 \
--mode ctr --secret-type symmetric --payload <base64_encoded_root_secret>

Alternatively, the existing root secret can also be stored in Barbican using curl.

Note

The credentials used to store the secret in Barbican shall be the same ones that the proxy server uses to retrieve the secret, i.e., the ones configured in the keymaster.conf file. For clarity reasons the commands shown here omit the credentials - they may be specified explicitly, or in environment variables.

Instead of using an existing root secret, Barbican can also be asked to generate a new 256-bit root secret, with content type application/octet-stream and algorithm AES (the mode parameter is currently optional):

openstack secret order create --name swift_root_secret \
--payload-content-type="application/octet-stream" --algorithm aes \
--bit-length 256 --mode ctr key

The order create creates an asynchronous request to create the actual secret. The order can be retrieved using openstack secret order get, and once the order completes successfully, the output will show the key id of the generated root secret. Keys currently stored in Barbican can be listed using the openstack secret list command.

Note

Both the order (the asynchronous request for creating or storing a secret), and the actual secret itself, have similar unique identifiers. Once the order has been completed, the key id is shown in the output of the order get command.

The keymaster uses the explicitly configured username and password (and project name etc.) from the keymaster.conf file for retrieving the encryption root secret from an external key management system. The Castellan library is used to communicate with Barbican.

For the proxy server, reading the encryption root secret directly from the proxy-server.conf file, from the keymaster.conf file pointed to from the proxy-server.conf file, or from an external key management system such as Barbican, are all functionally equivalent. In case reading the encryption root secret from the external key management system fails, the proxy server will not start up. If the encryption root secret is retrieved successfully, it is cached in memory in the proxy server.

For further details on the configuration options, see the [filter:kms_keymaster] section in the proxy-server.conf-sample file, and the keymaster.conf-sample file.

Encryption Root Secret in a KMIP service

This middleware enables Swift to fetch a root secret from a KMIP service. The root secret is expected to have been previously created in the KMIP service and is referenced by its unique identifier. The secret should be an AES-256 symmetric key.

To use this middleware Swift must be installed with the extra required dependencies:

sudo pip install .[kmip_keymaster]

Add the -e flag to install as a development version.

Edit the swift proxy-server.conf file to insert the middleware in the wsgi pipeline, replacing any other keymaster middleware:

[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck proxy-logging \
    <other middleware> kmip_keymaster encryption proxy-logging proxy-server

and add a new filter section:

[filter:kmip_keymaster]
use = egg:swift#kmip_keymaster
key_id = <unique id of secret to be fetched from the KMIP service>
host = <KMIP server host>
port = <KMIP server port>
certfile = /path/to/client/cert.pem
keyfile = /path/to/client/key.pem
ca_certs = /path/to/server/cert.pem
username = <KMIP username>
password = <KMIP password>

Apart from use and key_id the options are as defined for a PyKMIP client. The authoritative definition of these options can be found at https://pykmip.readthedocs.io/en/latest/client.html.

The value of the key_id option should be the unique identifier for a secret that will be retrieved from the KMIP service.

The keymaster configuration can alternatively be defined in a separate config file by using the keymaster_config_path option:

[filter:kmip_keymaster]
use = egg:swift#kmip_keymaster
keymaster_config_path = /etc/swift/kmip_keymaster.conf

In this case, the filter:kmip_keymaster section should contain no other options than use and keymaster_config_path. All other options should be defined in the separate config file in a section named kmip_keymaster. For example:

[kmip_keymaster]
key_id = 1234567890
host = 127.0.0.1
port = 5696
certfile = /etc/swift/kmip_client.crt
keyfile = /etc/swift/kmip_client.key
ca_certs = /etc/swift/kmip_server.crt
username = swift
password = swift_password

Changing the encryption root secret of external KMS’s

Because the KMS and KMIP keymaster’s derive from the default KeyMaster they also have to ability to define multiple keys. The only difference is the key option names. Instead of using the form encryption_root_secret_<secret_id> both external KMS’s use key_id_<secret_id>, as it is an extension of their existing configuration. For example:

...
key_id = 1234567890
key_id_foo = 0987654321
key_id_bar = 5432106789
active_root_secret_id = foo
...

Other then that, the process is the same as Changing the encryption root secret.

Upgrade Considerations

When upgrading an existing cluster to deploy encryption, the following sequence of steps is recommended:

1. Upgrade all object servers

2. Upgrade all proxy servers

3. Add keymaster and encryption middlewares to every proxy server’s middleware pipeline with the encryption disable_encryption option set to True and the keymaster encryption_root_secret value set as described above.

4. If required, follow the steps for Container sync configuration.

5. Finally, change the encryption disable_encryption option to False

Objects that existed in the cluster prior to the keymaster and encryption middlewares being deployed are still readable with GET and HEAD requests. The content of those objects will not be encrypted unless they are written again by a PUT or COPY request. Any user metadata of those objects will not be encrypted unless it is written again by a PUT, POST or COPY request.

Disabling Encryption

Once deployed, the keymaster and encryption middlewares should not be removed from the pipeline. To do so will cause encrypted object data and/or metadata to be returned in response to GET or HEAD requests for objects that were previously encrypted.

Encryption of inbound object data may be disabled by setting the encryption disable_encryption option to True, in which case existing encrypted objects will remain encrypted but new data written with PUT, POST or COPY requests will not be encrypted. The keymaster and encryption middlewares should remain in the pipeline even when encryption of new objects is not required. The encryption middleware is needed to handle GET requests for objects that may have been previously encrypted. The keymaster is needed to provide keys for those requests.

Container sync configuration

If container sync is being used then the keymaster and encryption middlewares must be added to the container sync internal client pipeline. The following configuration steps are required:

1. Create a custom internal client configuration file for container sync (if one is not already in use) based on the sample file internal-client.conf-sample. For example, copy internal-client.conf-sample to /etc/swift/container-sync-client.conf.

2. Modify this file to include the middlewares in the pipeline in the same way as described above for the proxy server.

3. Modify the container-sync section of all container server config files to point to this internal client config file using the internal_client_conf_path option. For example:

   internal_client_conf_path = /etc/swift/container-sync-client.conf
   

Note

The encryption_root_secret value is necessary to recover any encrypted data from the storage system, and therefore, it must be guarded against accidental loss. Its value (and consequently, the custom internal client configuration file) should not be stored on any disk that is in any account, container or object ring.

Note

These container sync configuration steps will be necessary for container sync probe tests to pass if the encryption middlewares are included in the proxy pipeline of a test cluster.

Implementation

Encryption scheme

Plaintext data is encrypted to ciphertext using the AES cipher with 256-bit keys implemented by the python cryptography package. The cipher is used in counter (CTR) mode so that any byte or range of bytes in the ciphertext may be decrypted independently of any other bytes in the ciphertext. This enables very simple handling of ranged GETs.

In general an item of unencrypted data, plaintext, is transformed to an item of encrypted data, ciphertext:

ciphertext = E(plaintext, k, iv)

where E is the encryption function, k is an encryption key and iv is a unique initialization vector (IV) chosen for each encryption context. For example, the object body is one encryption context with a randomly chosen IV. The IV is stored as metadata of the encrypted item so that it is available for decryption:

plaintext = D(ciphertext, k, iv)

where D is the decryption function.

The implementation of CTR mode follows NIST SP800-38A, and the full IV passed to the encryption or decryption function serves as the initial counter block.

In general any encrypted item has accompanying crypto-metadata that describes the IV and the cipher algorithm used for the encryption:

crypto_metadata = {"iv": <16 byte value>,
                   "cipher": "AES_CTR_256"}

This crypto-metadata is stored either with the ciphertext (for user metadata and etags) or as a separate header (for object bodies).

Key management

A keymaster middleware is responsible for providing the keys required for each encryption and decryption operation. Two keys are required when handling object requests: a container key that is uniquely associated with the container path and an object key that is uniquely associated with the object path. These keys are made available to the encryption middleware via a callback function that the keymaster installs in the WSGI request environ.

The current keymaster implementation derives container and object keys from the encryption_root_secret in a deterministic way by constructing a SHA256 HMAC using the encryption_root_secret as a key and the container or object path as a message, for example:

object_key = HMAC(encryption_root_secret, "/a/c/o")

Other strategies for providing object and container keys may be employed by future implementations of alternative keymaster middleware.

During each object PUT, a random key is generated to encrypt the object body. This random key is then encrypted using the object key provided by the keymaster. This makes it safe to store the encrypted random key alongside the encrypted object data and metadata.

This process of key wrapping enables more efficient re-keying events when the object key may need to be replaced and consequently any data encrypted using that key must be re-encrypted. Key wrapping minimizes the amount of data encrypted using those keys to just other randomly chosen keys which can be re-wrapped efficiently without needing to re-encrypt the larger amounts of data that were encrypted using the random keys.

Note

Re-keying is not currently implemented. Key wrapping is implemented in anticipation of future re-keying operations.

Encryption middleware

The encryption middleware is composed of an encrypter component and a decrypter component.

Encrypter operation

Custom user metadata

The encrypter encrypts each item of custom user metadata using the object key provided by the keymaster and an IV that is randomly chosen for that metadata item. The encrypted values are stored as Object Transient-Sysmeta with associated crypto-metadata appended to the encrypted value. For example:

X-Object-Meta-Private1: value1
X-Object-Meta-Private2: value2

are transformed to:

X-Object-Transient-Sysmeta-Crypto-Meta-Private1:
  E(value1, object_key, header_iv_1); swift_meta={"iv": header_iv_1,
                                                  "cipher": "AES_CTR_256"}
X-Object-Transient-Sysmeta-Crypto-Meta-Private2:
  E(value2, object_key, header_iv_2); swift_meta={"iv": header_iv_2,
                                                  "cipher": "AES_CTR_256"}

The unencrypted custom user metadata headers are removed.

Object body

Encryption of an object body is performed using a randomly chosen body key and a randomly chosen IV:

body_ciphertext = E(body_plaintext, body_key, body_iv)

The body_key is wrapped using the object key provided by the keymaster and a randomly chosen IV:

wrapped_body_key = E(body_key, object_key, body_key_iv)

The encrypter stores the associated crypto-metadata in a system metadata header:

X-Object-Sysmeta-Crypto-Body-Meta:
    {"iv": body_iv,
     "cipher": "AES_CTR_256",
     "body_key": {"key": wrapped_body_key,
                  "iv": body_key_iv}}

Note that in this case there is an extra item of crypto-metadata which stores the wrapped body key and its IV.

Entity tag

While encrypting the object body the encrypter also calculates the ETag (md5 digest) of the plaintext body. This value is encrypted using the object key provided by the keymaster and a randomly chosen IV, and saved as an item of system metadata, with associated crypto-metadata appended to the encrypted value:

X-Object-Sysmeta-Crypto-Etag:
  E(md5(plaintext), object_key, etag_iv); swift_meta={"iv": etag_iv,
                                                      "cipher": "AES_CTR_256"}

The encrypter also forces an encrypted version of the plaintext ETag to be sent with container updates by adding an update override header to the PUT request. The associated crypto-metadata is appended to the encrypted ETag value of this update override header:

X-Object-Sysmeta-Container-Update-Override-Etag:
    E(md5(plaintext), container_key, override_etag_iv);
    meta={"iv": override_etag_iv, "cipher": "AES_CTR_256"}

The container key is used for this encryption so that the decrypter is able to decrypt the ETags in container listings when handling a container request, since object keys may not be available in that context.

Since the plaintext ETag value is only known once the encrypter has completed processing the entire object body, the X-Object-Sysmeta-Crypto-Etag and X-Object-Sysmeta-Container-Update-Override-Etag headers are sent after the encrypted object body using the proxy server’s support for request footers.

Conditional Requests

In general, an object server evaluates conditional requests with If[-None]-Match headers by comparing values listed in an If[-None]-Match header against the ETag that is stored in the object metadata. This is not possible when the ETag stored in object metadata has been encrypted. The encrypter therefore calculates an HMAC using the object key and the ETag while handling object PUT requests, and stores this under the metadata key X-Object-Sysmeta-Crypto-Etag-Mac:

X-Object-Sysmeta-Crypto-Etag-Mac: HMAC(object_key, md5(plaintext))

Like other ETag-related metadata, this is sent after the encrypted object body using the proxy server’s support for request footers.

The encrypter similarly calculates an HMAC for each ETag value included in If[-None]-Match headers of conditional GET or HEAD requests, and appends these to the If[-None]-Match header. The encrypter also sets the X-Backend-Etag-Is-At header to point to the previously stored X-Object-Sysmeta-Crypto-Etag-Mac metadata so that the object server evaluates the conditional request by comparing the HMAC values included in the If[-None]-Match with the value stored under X-Object-Sysmeta-Crypto-Etag-Mac. For example, given a conditional request with header:

If-Match: match_etag

the encrypter would transform the request headers to include:

If-Match: match_etag,HMAC(object_key, match_etag)
X-Backend-Etag-Is-At: X-Object-Sysmeta-Crypto-Etag-Mac

This enables the object server to perform an encrypted comparison to check whether the ETags match, without leaking the ETag itself or leaking information about the object body.

Decrypter operation

For each GET or HEAD request to an object, the decrypter inspects the response for encrypted items (revealed by crypto-metadata headers), and if any are discovered then it will:

1. Fetch the object and container keys from the keymaster via its callback

2. Decrypt the X-Object-Sysmeta-Crypto-Etag value

3. Decrypt the X-Object-Sysmeta-Container-Update-Override-Etag value

4. Decrypt metadata header values using the object key

5. Decrypt the wrapped body key found in X-Object-Sysmeta-Crypto-Body-Meta

6. Decrypt the body using the body key

For each GET request to a container that would include ETags in its response body, the decrypter will:

1. GET the response body with the container listing

2. Fetch the container key from the keymaster via its callback

3. Decrypt any encrypted ETag entries in the container listing using the container key

Impact on other Swift services and features

Encryption has no impact on Versioned Writes other than that any previously unencrypted objects will be encrypted as they are copied to or from the versions container. Keymaster and encryption middlewares should be placed after versioned_writes in the proxy server pipeline, as described in Deployment and operation.

Container Sync uses an internal client to GET objects that are to be sync’d. This internal client must be configured to use the keymaster and encryption middlewares as described above.

Encryption has no impact on the object-auditor service. Since the ETag header saved with the object at rest is the md5 sum of the encrypted object body then the auditor will verify that encrypted data is valid.

Encryption has no impact on the object-expirer service. X-Delete-At and X-Delete-After headers are not encrypted.

Encryption has no impact on the object-replicator and object-reconstructor services. These services are unaware of the object or EC fragment data being encrypted.

Encryption has no impact on the container-reconciler service. The container-reconciler uses an internal client to move objects between different policy rings. The reconciler’s pipeline MUST NOT have encryption enabled. The destination object has the same URL as the source object and the object is moved without re-encryption.

Considerations for developers

Developers should be aware that keymaster and encryption middlewares rely on the path of an object remaining unchanged. The included keymaster derives keys for containers and objects based on their paths and the encryption_root_secret. The keymaster does not rely on object metadata to inform its generation of keys for GET and HEAD requests because when handling Conditional Requests it is required to provide the object key before any metadata has been read from the object.

Developers should therefore give careful consideration to any new features that would relocate object data and metadata within a Swift cluster by means that do not cause the object data and metadata to pass through the encryption middlewares in the proxy pipeline and be re-encrypted.

The crypto-metadata associated with each encrypted item does include some key_id metadata that is provided by the keymaster and contains the path used to derive keys. This key_id metadata is persisted in anticipation of future scenarios when it may be necessary to decrypt an object that has been relocated without re-encrypting, in which case the metadata could be used to derive the keys that were used for encryption. However, this alone is not sufficient to handle conditional requests and to decrypt container listings where objects have been relocated, and further work will be required to solve those issues.