Storage Policies¶

Defining a policy¶

Each policy is defined by a section in the /etc/swift/swift.conf file. The section name must be of the form [storage-policy:<N>] where <N> is the policy index. There’s no reason other than readability that policy indexes be sequential but the following rules are enforced:

• If a policy with index 0 is not declared and no other policies are defined, Swift will create a default policy with index 0.
• The policy index must be a non-negative integer.
• Policy indexes must be unique.

Each policy section contains the following options:

• name = <policy_name> (required)

  • The primary name of the policy.
  • Policy names are case insensitive.
  • Policy names must contain only letters, digits or a dash.
  • Policy names must be unique.
  • Policy names can be changed.
  • The name Policy-0 can only be used for the policy with index 0.

• alias = <policy_name>[, <policy_name>, ...] (optional)

  • A comma-separated list of alternative names for the policy.
  • The default value is an empty list (i.e. no aliases).
  • All alias names must follow the rules for the name option.
  • Aliases can be added to and removed from the list.
  • Aliases can be useful to retain support for old primary names if the primary name is changed.

• default = [true|false] (optional)

  • If true then this policy will be used when the client does not specify a policy.
  • The default value is false.
  • The default policy can be changed at any time, by setting default = true in the desired policy section.
  • If no policy is declared as the default and no other policies are defined, the policy with index 0 is set as the default;
  • Otherwise, exactly one policy must be declared default.
  • Deprecated policies cannot be declared the default.
  • See Default versus ‘Policy-0’ for more information.

• deprecated = [true|false] (optional)

  • If true then new containers cannot be created using this policy.
  • The default value is false.
  • Any policy may be deprecated by adding the deprecated option to the desired policy section. However, a deprecated policy may not also be declared the default. Therefore, since there must always be a default policy, there must also always be at least one policy which is not deprecated.
  • See Deprecating Policies for more information.

• policy_type = [replication|erasure_coding] (optional)

  • The option policy_type is used to distinguish between different policy types.
  • The default value is replication.
  • When defining an EC policy use the value erasure_coding.

The EC policy type has additional required options. See Using an Erasure Code Policy for details.

The following is an example of a properly configured swift.conf file. See Adding Storage Policies to an Existing SAIO for full instructions on setting up an all-in-one with this example configuration.:

[swift-hash]
# random unique strings that can never change (DO NOT LOSE)
# Use only printable chars (python -c "import string; print(string.printable)")
swift_hash_path_prefix = changeme
swift_hash_path_suffix = changeme

[storage-policy:0]
name = gold
aliases = yellow, orange
policy_type = replication
default = yes

[storage-policy:1]
name = silver
policy_type = replication
deprecated = yes

Creating a ring¶

Once swift.conf is configured for a new policy, a new ring must be created. The ring tools are not policy name aware so it’s critical that the correct policy index be used when creating the new policy’s ring file. Additional object rings are created using swift-ring-builder in the same manner as the legacy ring except that -N is appended after the word object in the builder file name, where N matches the policy index used in swift.conf. So, to create the ring for policy index 1:

swift-ring-builder object-1.builder create 10 3 1

Continue to use the same naming convention when using swift-ring-builder to add devices, rebalance etc. This naming convention is also used in the pattern for per-policy storage node data directories.

Proxy server configuration (optional)¶

The Proxy Server configuration options related to read and write affinity may optionally be overridden for individual storage policies. See Per policy configuration for more details.

Parsing and Configuring¶

The module, Storage Policy, is responsible for parsing the swift.conf file, validating the input, and creating a global collection of configured policies via class StoragePolicyCollection. This collection is made up of policies of class StoragePolicy. The collection class includes handy functions for getting to a policy either by name or by index , getting info about the policies, etc. There’s also one very important function, get_object_ring(). Object rings are members of the StoragePolicy class and are actually not instantiated until the load_ring() method is called. Any caller anywhere in the code base that needs to access an object ring must use the POLICIES global singleton to access the get_object_ring() function and provide the policy index which will call load_ring() if needed; however, when starting request handling services such as the Proxy Server rings are proactively loaded to provide moderate protection against a mis-configuration resulting in a run time error. The global is instantiated when Swift starts and provides a mechanism to patch policies for the test code.

Middleware¶

Middleware can take advantage of policies through the POLICIES global and by importing get_container_info() to gain access to the policy index associated with the container in question. From the index it can then use the POLICIES singleton to grab the right ring. For example, List Endpoints is policy aware using the means just described. Another example is Recon which will report the md5 sums for all of the rings.

Proxy Server¶

The Proxy Server module’s role in Storage Policies is essentially to make sure the correct ring is used as its member element. Before policies, the one object ring would be instantiated when the Application class was instantiated and could be overridden by test code via init parameter. With policies, however, there is no init parameter and the Application class instead depends on the POLICIES global singleton to retrieve the ring which is instantiated the first time it’s needed. So, instead of an object ring member of the Application class, there is an accessor function, get_object_ring(), that gets the ring from POLICIES.

In general, when any module running on the proxy requires an object ring, it does so via first getting the policy index from the cached container info. The exception is during container creation where it uses the policy name from the request header to look up policy index from the POLICIES global. Once the proxy has determined the policy index, it can use the get_object_ring() method described earlier to gain access to the correct ring. It then has the responsibility of passing the index information, not the policy name, on to the back-end servers via the header X -Backend-Storage-Policy-Index. Going the other way, the proxy also strips the index out of headers that go back to clients, and makes sure they only see the friendly policy names.

On Disk Storage¶

Policies each have their own directories on the back-end servers and are identified by their storage policy indexes. Organizing the back-end directory structures by policy index helps keep track of things and also allows for sharing of disks between policies which may or may not make sense depending on the needs of the provider. More on this later, but for now be aware of the following directory naming convention:

• /objects maps to objects associated with Policy-0
• /objects-N maps to storage policy index #N
• /async_pending maps to async pending update for Policy-0
• /async_pending-N maps to async pending update for storage policy index #N
• /tmp maps to the DiskFile temporary directory for Policy-0
• /tmp-N maps to the DiskFile temporary directory for policy index #N
• /quarantined/objects maps to the quarantine directory for Policy-0
• /quarantined/objects-N maps to the quarantine directory for policy index #N

Note that these directory names are actually owned by the specific Diskfile implementation, the names shown above are used by the default Diskfile.

Object Server¶

The Object Server is not involved with selecting the storage policy placement directly. However, because of how back-end directory structures are setup for policies, as described earlier, the object server modules do play a role. When the object server gets a Diskfile, it passes in the policy index and leaves the actual directory naming/structure mechanisms to Diskfile. By passing in the index, the instance of Diskfile being used will assure that data is properly located in the tree based on its policy.

For the same reason, the Object Updater also is policy aware. As previously described, different policies use different async pending directories so the updater needs to know how to scan them appropriately.

The Object Replicator is policy aware in that, depending on the policy, it may have to do drastically different things, or maybe not. For example, the difference in handling a replication job for 2x versus 3x is trivial; however, the difference in handling replication between 3x and erasure code is most definitely not. In fact, the term ‘replication’ really isn’t appropriate for some policies like erasure code; however, the majority of the framework for collecting and processing jobs is common. Thus, those functions in the replicator are leveraged for all policies and then there is policy specific code required for each policy, added when the policy is defined if needed.

The ssync functionality is policy aware for the same reason. Some of the other modules may not obviously be affected, but the back-end directory structure owned by Diskfile requires the policy index parameter. Therefore ssync being policy aware really means passing the policy index along. See ssync_sender and ssync_receiver for more information on ssync.

For Diskfile itself, being policy aware is all about managing the back-end structure using the provided policy index. In other words, callers who get a Diskfile instance provide a policy index and Diskfile’s job is to keep data separated via this index (however it chooses) such that policies can share the same media/nodes if desired. The included implementation of Diskfile lays out the directory structure described earlier but that’s owned within Diskfile; external modules have no visibility into that detail. A common function is provided to map various directory names and/or strings based on their policy index. For example Diskfile defines get_data_dir() which builds off of a generic get_policy_string() to consistently build policy aware strings for various usage.

Container Server¶

The Container Server plays a very important role in Storage Policies, it is responsible for handling the assignment of a policy to a container and the prevention of bad things like changing policies or picking the wrong policy to use when nothing is specified (recall earlier discussion on Policy-0 versus default).

The Container Updater is policy aware, however its job is very simple, to pass the policy index along to the Account Server via a request header.

The Container Backend is responsible for both altering existing DB schema as well as assuring new DBs are created with a schema that supports storage policies. The “on-demand” migration of container schemas allows Swift to upgrade without downtime (sqlite’s alter statements are fast regardless of row count). To support rolling upgrades (and downgrades) the incompatible schema changes to the container_stat table are made to a container_info table, and the container_stat table is replaced with a view that includes an INSTEAD OF UPDATE trigger which makes it behave like the old table.

The policy index is stored here for use in reporting information about the container as well as managing split-brain scenario induced discrepancies between containers and their storage policies. Furthermore, during split-brain, containers must be prepared to track object updates from multiple policies so the object table also includes a storage_policy_index column. Per-policy object counts and bytes are updated in the policy_stat table using INSERT and DELETE triggers similar to the pre-policy triggers that updated container_stat directly.

The Container Replicator daemon will pro-actively migrate legacy schemas as part of its normal consistency checking process when it updates the reconciler_sync_point entry in the container_info table. This ensures that read heavy containers which do not encounter any writes will still get migrated to be fully compatible with the post-storage-policy queries without having to fall back and retry queries with the legacy schema to service container read requests.

The Container Sync functionality only needs to be policy aware in that it accesses the object rings. Therefore, it needs to pull the policy index out of the container information and use it to select the appropriate object ring from the POLICIES global.

Account Server¶

The Account Server’s role in Storage Policies is really limited to reporting. When a HEAD request is made on an account (see example provided earlier), the account server is provided with the storage policy index and builds the object_count and byte_count information for the client on a per policy basis.

The account servers are able to report per-storage-policy object and byte counts because of some policy specific DB schema changes. A policy specific table, policy_stat, maintains information on a per policy basis (one row per policy) in the same manner in which the account_stat table does. The account_stat table still serves the same purpose and is not replaced by policy_stat, it holds the total account stats whereas policy_stat just has the break downs. The backend is also responsible for migrating pre-storage-policy accounts by altering the DB schema and populating the policy_stat table for Policy-0 with current account_stat data at that point in time.

The per-storage-policy object and byte counts are not updated with each object PUT and DELETE request, instead container updates to the account server are performed asynchronously by the swift-container-updater.

Upgrading and Confirming Functionality¶

Upgrading to a version of Swift that has Storage Policy support is not difficult, in fact, the cluster administrator isn’t required to make any special configuration changes to get going. Swift will automatically begin using the existing object ring as both the default ring and the Policy-0 ring. Adding the declaration of policy 0 is totally optional and in its absence, the name given to the implicit policy 0 will be ‘Policy-0’. Let’s say for testing purposes that you wanted to take an existing cluster that already has lots of data on it and upgrade to Swift with Storage Policies. From there you want to go ahead and create a policy and test a few things out. All you need to do is:

1. Upgrade all of your Swift nodes to a policy-aware version of Swift
2. Define your policies in /etc/swift/swift.conf
3. Create the corresponding object rings
4. Create containers and objects and confirm their placement is as expected

For a specific example that takes you through these steps, please see Adding Storage Policies to an Existing SAIO