Large objects¶

Static large objects¶

To create a static large object, divide your content into pieces and create (upload) a segment object to contain each piece.

You must record the ETag response header value that the PUT operation returns. Alternatively, you can calculate the MD5 checksum of the segment before you perform the upload and include this value in the ETag request header. This action ensures that the upload cannot corrupt your data.

List the name of each segment object along with its size and MD5 checksum in order.

Create a manifest object. Include the ?multipart-manifest=put query string at the end of the manifest object name to indicate that this is a manifest object.

The body of the PUT request on the manifest object comprises a JSON list where each element contains these attributes:

path

The container and object name in the format: CONTAINER_NAME/OBJECT_NAME.

etag

The MD5 checksum of the content of the segment object. This value must match the ETag of that object.

size_bytes

The size of the segment object. This value must match the Content-Length of that object.

[
    {
        "path": "mycontainer/objseg1",
        "etag": "0228c7926b8b642dfb29554cd1f00963",
        "size_bytes": 1468006
    },
    {
        "path": "mycontainer/pseudodir/seg-obj2",
        "etag": "5bfc9ea51a00b790717eeb934fb77b9b",
        "size_bytes": 1572864
    },
    {
        "path": "other-container/seg-final",
        "etag": "b9c3da507d2557c1ddc51f27c54bae51",
        "size_bytes": 256
    }
]

Dynamic large objects¶

Before you can upload objects that are larger than 5 GB, you must segment them. You upload the segment objects like you do with any other object and create a dynamic large manifest object. The manifest object tells Object Storage how to find the segment objects that comprise the large object. You can still access each segment individually, but when you retrieve the manifest object, the API concatenates the segments. You can include any number of segments in a single large object.

To ensure the download works correctly, you must upload all the object segments to the same container and prefix each object name so that the segments sort in correct concatenation order.

You also create and upload a manifest file. The manifest file is a zero-byte file with the extra X-Object-Manifest CONTAINER/PREFIX header. The CONTAINER is the container the object segments are in and PREFIX is the common prefix for all the segments. You must UTF-8-encode and then URL-encode the container and common prefix in the X-Object-Manifest header.

It is best to upload all the segments first and then create or update the manifest. With this method, the full object is not available for downloading until the upload is complete. Also, you can upload a new set of segments to a second location and update the manifest to point to this new location. During the upload of the new segments, the original manifest is still available to download the first set of segments.

PUT /API_VERSION/ACCOUNT/CONTAINER/OBJECT HTTP/1.1
Host: storage.example.com
X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
ETag: 8a964ee2a5e88be344f36c22562a6486
Content-Length: 1
X-Object-Meta-PIN: 1234

PUT /API_VERSION/ACCOUNT/CONTAINER/OBJECT HTTP/1.1
Host: storage.example.com
X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
ETag: 8a964ee2a5e88be344f36c22562a6486
Content-Length: 1
X-Object-Meta-PIN: 1234

PUT /API_VERSION/ACCOUNT/CONTAINER/OBJECT HTTP/1.1
Host: storage.clouddrive.com
X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cbb
Content-Length: 0
X-Object-Meta-PIN: 1234
X-Object-Manifest: CONTAINER/PREFIX

[...]

Extra transaction information¶

You can use the X-Trans-Id-Extra request header to include extra information to help you debug any errors that might occur with large object upload and other Object Storage transactions.

The Object Storage API appends the first 32 characters of the X-Trans-Id-Extra request header value to the transaction ID value in the generated X-Trans-Id response header. You must UTF-8-encode and then URL-encode the extra transaction information before you include it in the X-Trans-Id-Extra request header.

For example, you can include extra transaction information when you upload large objects such as images.

When you upload each segment and the manifest, include the same value in the X-Trans-Id-Extra request header. If an error occurs, you can find all requests that are related to the large object upload in the Object Storage logs.

You can also use X-Trans-Id-Extra strings to help operators debug requests that fail to receive responses. The operator can search for the extra information in the logs.

Comparison of static and dynamic large objects¶

While static and dynamic objects have similar behavior, this table describes their differences:

Upload large objects with python-swiftclient¶

You can use python-swiftclient to easily upload large objects.

• Upload a large file by specifying the segment size with the --segment-size or -S arguments:

  $ swift upload CONTAINER OBJECT_FILENAME --segment-size <bytes>
  

  This will automatically break the file into the desired segment size and upload segments to a container named <container>_segments.

• After upload has completed, you can download the large object as a single file:

  $ swift download CONTAINER OBJECT_FILENAME
  

• Additional large object arguments can be found by using --help:

  $ swift upload --help