Object Encryption¶

Keymaster middleware¶

The keymaster middleware must be configured with a root secret before it is used. By default the keymaster middleware will use the root secret configured using the encryption_root_secret option in the middleware filter section of the proxy-server.conf file, for example:

[filter:keymaster]
use = egg:swift#keymaster
encryption_root_secret = your_secret

Root secret values MUST be at least 44 valid base-64 characters and should be consistent across all proxy servers. The minimum length of 44 has been chosen because it is the length of a base-64 encoded 32 byte value.

One method for generating a suitable value for encryption_root_secret is to use the openssl command line tool:

openssl rand -base64 32

[filter:keymaster]
use = egg:swift#keymaster
keymaster_config_path = /etc/swift/keymaster.conf

[keymaster]
encryption_root_secret = your_secret

encryption_root_secret_<secret_id> = <secret value>

[keymaster]
active_root_secret_id = 2
encryption_root_secret = your_secret
encryption_root_secret_1 = your_secret_1
encryption_root_secret_2 = your_secret_2

Encryption middleware¶

Once deployed, the encryption filter will by default encrypt object data and metadata when handling PUT and POST requests and decrypt object data and metadata when handling GET and HEAD requests. COPY requests are transformed into GET and PUT requests by the Server Side Copy middleware before reaching the encryption middleware and as a result object data and metadata is decrypted and re-encrypted when copied.

Encryption Root Secret in External Key Management System¶

The benefits of using a dedicated system for storing the encryption root secret include the auditing and access control infrastructure that are already in place in such a system, and the fact that an encryption root secret stored in a key management system (KMS) may be backed by a hardware security module (HSM) for additional security. Another significant benefit of storing the root encryption secret in an external KMS is that it is in this case never stored on a disk in the Swift cluster.

Swift supports fetching encryption root secrets from a Barbican service or a KMIP service using the kms_keymaster or kmip_keymaster middleware respectively.

sudo pip install .[kms_keymaster]

cryptography>=1.6                       # BSD/Apache-2.0
castellan>=0.6.0

<other middleware> kms_keymaster encryption proxy-logging proxy-server

[filter:kms_keymaster]
use = egg:swift#kms_keymaster
keymaster_config_path = file_with_kms_keymaster_config

[kms_keymaster]
key_id = changeme
username = swift
password = password
project_name = swift
auth_endpoint = http://keystonehost:5000/v3

openstack secret store --name swift_root_secret \
--payload-content-type="application/octet-stream" \
--payload-content-encoding="base64" --algorithm aes --bit-length 256 \
--mode ctr --secret-type symmetric --payload <base64_encoded_root_secret>

openstack secret order create --name swift_root_secret \
--payload-content-type="application/octet-stream" --algorithm aes \
--bit-length 256 --mode ctr key

sudo pip install .[kmip_keymaster]

[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck proxy-logging \
    <other middleware> kmip_keymaster encryption proxy-logging proxy-server

[filter:kmip_keymaster]
use = egg:swift#kmip_keymaster
key_id = <unique id of secret to be fetched from the KMIP service>
host = <KMIP server host>
port = <KMIP server port>
certfile = /path/to/client/cert.pem
keyfile = /path/to/client/key.pem
ca_certs = /path/to/server/cert.pem
username = <KMIP username>
password = <KMIP password>

[filter:kmip_keymaster]
use = egg:swift#kmip_keymaster
keymaster_config_path = /etc/swift/kmip_keymaster.conf

[kmip_keymaster]
key_id = 1234567890
host = 127.0.0.1
port = 5696
certfile = /etc/swift/kmip_client.crt
keyfile = /etc/swift/kmip_client.key
ca_certs = /etc/swift/kmip_server.crt
username = swift
password = swift_password

...
key_id = 1234567890
key_id_foo = 0987654321
key_id_bar = 5432106789
active_root_secret_id = foo
...

Upgrade Considerations¶

When upgrading an existing cluster to deploy encryption, the following sequence of steps is recommended:

1. Upgrade all object servers
2. Upgrade all proxy servers
3. Add keymaster and encryption middlewares to every proxy server’s middleware pipeline with the encryption disable_encryption option set to True and the keymaster encryption_root_secret value set as described above.
4. If required, follow the steps for Container sync configuration.
5. Finally, change the encryption disable_encryption option to False

Objects that existed in the cluster prior to the keymaster and encryption middlewares being deployed are still readable with GET and HEAD requests. The content of those objects will not be encrypted unless they are written again by a PUT or COPY request. Any user metadata of those objects will not be encrypted unless it is written again by a PUT, POST or COPY request.

Disabling Encryption¶

Once deployed, the keymaster and encryption middlewares should not be removed from the pipeline. To do so will cause encrypted object data and/or metadata to be returned in response to GET or HEAD requests for objects that were previously encrypted.

Encryption of inbound object data may be disabled by setting the encryption disable_encryption option to True, in which case existing encrypted objects will remain encrypted but new data written with PUT, POST or COPY requests will not be encrypted. The keymaster and encryption middlewares should remain in the pipeline even when encryption of new objects is not required. The encryption middleware is needed to handle GET requests for objects that may have been previously encrypted. The keymaster is needed to provide keys for those requests.

Container sync configuration¶

If container sync is being used then the keymaster and encryption middlewares must be added to the container sync internal client pipeline. The following configuration steps are required:

1. Create a custom internal client configuration file for container sync (if one is not already in use) based on the sample file internal-client.conf-sample. For example, copy internal-client.conf-sample to /etc/swift/container-sync-client.conf.

2. Modify this file to include the middlewares in the pipeline in the same way as described above for the proxy server.

3. Modify the container-sync section of all container server config files to point to this internal client config file using the internal_client_conf_path option. For example:

   internal_client_conf_path = /etc/swift/container-sync-client.conf
   

Encryption scheme¶

Plaintext data is encrypted to ciphertext using the AES cipher with 256-bit keys implemented by the python cryptography package. The cipher is used in counter (CTR) mode so that any byte or range of bytes in the ciphertext may be decrypted independently of any other bytes in the ciphertext. This enables very simple handling of ranged GETs.

In general an item of unencrypted data, plaintext, is transformed to an item of encrypted data, ciphertext:

ciphertext = E(plaintext, k, iv)

where E is the encryption function, k is an encryption key and iv is a unique initialization vector (IV) chosen for each encryption context. For example, the object body is one encryption context with a randomly chosen IV. The IV is stored as metadata of the encrypted item so that it is available for decryption:

plaintext = D(ciphertext, k, iv)

where D is the decryption function.

The implementation of CTR mode follows NIST SP800-38A, and the full IV passed to the encryption or decryption function serves as the initial counter block.

In general any encrypted item has accompanying crypto-metadata that describes the IV and the cipher algorithm used for the encryption:

crypto_metadata = {"iv": <16 byte value>,
                   "cipher": "AES_CTR_256"}

This crypto-metadata is stored either with the ciphertext (for user metadata and etags) or as a separate header (for object bodies).

Key management¶

A keymaster middleware is responsible for providing the keys required for each encryption and decryption operation. Two keys are required when handling object requests: a container key that is uniquely associated with the container path and an object key that is uniquely associated with the object path. These keys are made available to the encryption middleware via a callback function that the keymaster installs in the WSGI request environ.

The current keymaster implementation derives container and object keys from the encryption_root_secret in a deterministic way by constructing a SHA256 HMAC using the encryption_root_secret as a key and the container or object path as a message, for example:

object_key = HMAC(encryption_root_secret, "/a/c/o")

Other strategies for providing object and container keys may be employed by future implementations of alternative keymaster middleware.

During each object PUT, a random key is generated to encrypt the object body. This random key is then encrypted using the object key provided by the keymaster. This makes it safe to store the encrypted random key alongside the encrypted object data and metadata.

This process of key wrapping enables more efficient re-keying events when the object key may need to be replaced and consequently any data encrypted using that key must be re-encrypted. Key wrapping minimizes the amount of data encrypted using those keys to just other randomly chosen keys which can be re-wrapped efficiently without needing to re-encrypt the larger amounts of data that were encrypted using the random keys.

Encryption middleware¶

The encryption middleware is composed of an encrypter component and a decrypter component.

X-Object-Meta-Private1: value1
X-Object-Meta-Private2: value2

X-Object-Transient-Sysmeta-Crypto-Meta-Private1:
  E(value1, object_key, header_iv_1); swift_meta={"iv": header_iv_1,
                                                  "cipher": "AES_CTR_256"}
X-Object-Transient-Sysmeta-Crypto-Meta-Private2:
  E(value2, object_key, header_iv_2); swift_meta={"iv": header_iv_2,
                                                  "cipher": "AES_CTR_256"}

body_ciphertext = E(body_plaintext, body_key, body_iv)

wrapped_body_key = E(body_key, object_key, body_key_iv)

X-Object-Sysmeta-Crypto-Body-Meta:
    {"iv": body_iv,
     "cipher": "AES_CTR_256",
     "body_key": {"key": wrapped_body_key,
                  "iv": body_key_iv}}

X-Object-Sysmeta-Crypto-Etag:
  E(md5(plaintext), object_key, etag_iv); swift_meta={"iv": etag_iv,
                                                      "cipher": "AES_CTR_256"}

X-Object-Sysmeta-Container-Update-Override-Etag:
    E(md5(plaintext), container_key, override_etag_iv);
    meta={"iv": override_etag_iv, "cipher": "AES_CTR_256"}

X-Object-Sysmeta-Crypto-Etag-Mac: HMAC(object_key, md5(plaintext))

If-Match: match_etag

If-Match: match_etag,HMAC(object_key, match_etag)
X-Backend-Etag-Is-At: X-Object-Sysmeta-Crypto-Etag-Mac

Impact on other Swift services and features¶

Encryption has no impact on Object Versioning other than that any previously unencrypted objects will be encrypted as they are copied to or from the versions container. Keymaster and encryption middlewares should be placed after versioned_writes in the proxy server pipeline, as described in Deployment and operation.

Container Sync uses an internal client to GET objects that are to be sync’d. This internal client must be configured to use the keymaster and encryption middlewares as described above.

Encryption has no impact on the object-auditor service. Since the ETag header saved with the object at rest is the md5 sum of the encrypted object body then the auditor will verify that encrypted data is valid.

Encryption has no impact on the object-expirer service. X-Delete-At and X-Delete-After headers are not encrypted.

Encryption has no impact on the object-replicator and object-reconstructor services. These services are unaware of the object or EC fragment data being encrypted.

Encryption has no impact on the container-reconciler service. The container-reconciler uses an internal client to move objects between different policy rings. The destination object has the same URL as the source object and the object is moved without re-encryption.

Considerations for developers¶

Developers should be aware that keymaster and encryption middlewares rely on the path of an object remaining unchanged. The included keymaster derives keys for containers and objects based on their paths and the encryption_root_secret. The keymaster does not rely on object metadata to inform its generation of keys for GET and HEAD requests because when handling Conditional Requests it is required to provide the object key before any metadata has been read from the object.

Developers should therefore give careful consideration to any new features that would relocate object data and metadata within a Swift cluster by means that do not cause the object data and metadata to pass through the encryption middlewares in the proxy pipeline and be re-encrypted.

The crypto-metadata associated with each encrypted item does include some key_id metadata that is provided by the keymaster and contains the path used to derive keys. This key_id metadata is persisted in anticipation of future scenarios when it may be necessary to decrypt an object that has been relocated without re-encrypting, in which case the metadata could be used to derive the keys that were used for encryption. However, this alone is not sufficient to handle conditional requests and to decrypt container listings where objects have been relocated, and further work will be required to solve those issues.