# Message Queuing API v1 Reference

Message Queuing API Reference
API v1
Message Queuing
2014-02-20
Copyright details are filled in by the build system.

This document is intended for software developers interested in developing applications using the Message Queuing Application Programming Interface (API).

2014-02-20

Initial document for OpenStack incubation.

## Preface

Message Queuing is an open source, scalable, and highly available message and notifications service. Message Queuing supports a variety of messaging patterns. Users of this service can create and manage a producer-consumer or a publisher-subscriber model () from one simple API. Unlimited queues and messages give Message Queuing users the flexibility they need to create powerful web applications in the cloud.

Message Queuing consists of a few basic components: queues, messages, claims, and statistics. In the producer-consumer model, users create queues in which producers, or servers, can post messages. Workers, or consumers, can then claim those messages and delete them after they complete the actions associated with the messages. A single claim can contain multiple messages, and administrators can query claims for status.

In the publisher-subscriber model, messages are posted to a queue as in the producer-consumer model, but messages are never claimed. Instead, subscribers, or watchers, send `GET` requests to pull all messages that have been posted since their last request. In this model, a message remains in the queue, unclaimed, until the message's time to live (TTL) has expired.

In both of these models, administrators can get queue statistics that display the most recent and oldest messages, the number of unclaimed messages, and more.

### Intended Audience

This document is intended for software developers who are interested in developing applications that use the Message Queuing API. It assumes that you have the following skills:

- RESTful web services

- HTTP/1.1 conventions

- JSON data serialization formats

### Document Change History

This version of the document replaces and obsoletes all earlier versions. The most recent changes are described in the following table.

## Concepts

To use the Message Queueing API effectively, you should understand the key concepts described in this section.

### How Message Queuing Works

Following is an overview of how Message Queuing works. For definitions of Message Queuing terms, see the .

1. You create a queue to which producers or publishers post messages.

2. Workers (consumers or subscribers) claim or get a message from the queue, complete the work in that message, and delete the message.

   If a worker will be off-line before it completes the work in a message, the worker can retire the claim's time to live (TTL), putting the message back into the queue for another worker to claim.

3. Subscribers monitor the claims from these queues to track activity and help troubleshoot errors.

For the majority of use cases, Message Queuing is not responsible for the ordering of messages. However, if there is only a single producer, Message Queuing ensures that messages are handled in a First In, First Out (FIFO) order.

### Messaging Patterns

The Message Queuing API supports a variety of messaging patterns including the following:

- Task distribution

- Event broadcasting

- Point-to-point messaging

#### Task distribution

The task distribution pattern has the following characteristics:

- A producer is programmed to send messages to a queue.

- Multiple workers (or consumers) are programmed to monitor a queue.

- Only one worker can claim a message so that no other worker can claim the message and duplicate the work.

- The worker must delete the message when work is done.

- TTL restores a message to an unclaimed state if the worker never finishes.

This pattern is ideal for dispatching jobs to multiple processors.

#### Event broadcasting

Characteristics of the event broadcasting pattern are:

- The publisher sends messages to a queue.

- Multiple observers get the messages in the queue.

- Multiple observers take action on each message.

- Observers send a marker to skip messages already seen.

- TTL eventually deletes messages.

This pattern is ideal for notification of events to multiple observers at once.

#### Point-to-point messaging

Characteristics of the point-to-point messaging pattern are:

- The publisher sends messages to a queue.

- The consumer gets the messages in the queue.

- The consumer can reply with the result of processing a message by sending another message to the same queue (queues are duplex by default).

- The publisher gets replies from the queue.

- The consumer sends a marker to skip messages already seen.

- TTL eventually deletes messages.

This pattern is ideal for communicating with a specific client, especially when a reply is desired from that client.

### Use Cases

Queuing systems are used to coordinate tasks within an application. Here are some examples:

- Backup: A backup application might use a queuing system to connect the actions that users do in the a control panel to the customer's backup agent on a server. When a customer wants to start a backup, they simply choose "start backup" on a panel. Doing so causes the producer to put a "startBackup" message into the queue. Every few minutes, the agent on the customers server (the worker) checks the queue to see if it has any new messages to act on. The agent claims the "startBackup" message and kicks off the backup on the customer's server.

- Storage: Gathering statistics for a large, distributed storage system can be a long process. The storage system can use a queuing system to ensure that jobs complete, even if one initially fails. Since messages are not deleted until after the worker has completed the job, the storage system can make sure that no job goes undone. If the worker fails to complete the job, the message stays in the queue to be completed by another server. In this case, a worker claims a message to perform a statistics job, but the claim's TTL expired and the message is put back into the queue when the job took too long to complete (meaning that it most likely failed). By giving the claim a TTL, applications can protect themselves from workers going off-line while processing a message. After a claim's TTL expires, the message is put back into the queue for another worker to claim.

- Email: The team for an email application is constantly migrating customer email from old versions to newer ones, so they develop a tool to let customers do it themselves. The migrations take a long time, so they cannot be done with single API calls, or by a single server. When a user starts a migration job from their portal, the migration tool sends messages to the queue with details of how to run the migration. A set of migration engines, the consumers in this case, periodically check the queues for new migration tasks, claim the messages, perform the migration, and update a database with the migration details. This process allows a set of servers to work together to accomplish large migrations in a timely manner.

Following are some generic use cases for Message Queuing:

- Distribute tasks among multiple workers (transactional job queues)

- Forward events to data collectors (transactional event queues)

- Publish events to any number of subscribers (event broadcasting)

- Send commands to one or more agents (point-to-point messaging or event broadcasting)

- Request an action or get information from a Remote Procedure Call (RPC) agent (point-to-point messaging)

## General API Information

The Message Queuing API is implemented using a RESTful web service interface. Like other OpenStack projects, Message Queuing shares a common token-based authentication system that allows seamless access between products and services.

All requests to authenticate against and operate the service are performed using SSL over HTTP (HTTPS) on TCP port 443.

### Authentication

You can use cURL to try the authentication process in two steps: get a token, and send the token to a service.

1. Get an authentication token by providing your user name and either your API key or your password. Here are examples of both approaches:

   You can request a token by providing your user name and your password.

       $ curl -X POST https://localhost:5000/v2.0/tokens -d '{"auth":{"passwordCredentials":{"username": "joecool", "password":"coolword"}, "tenantId":"5"}}' -H 'Content-type: application/json'

   Successful authentication returns a token which you can use as evidence that your identity has already been authenticated. To use the token, pass it to other services as an `X-Auth-Token` header.

   Authentication also returns a service catalog, listing the endpoints you can use for Cloud services.

2. Use the authentication token to send a `GET` to a service you would like to use.

Authentication tokens are typically valid for 24 hours. Applications should be designed to re-authenticate after receiving a 401 (Unauthorized) response from a service endpoint.

If you programmatically parse an authentication response, be aware that service names are stable for the life of the particular service and can be used as keys. You should also be aware that a user's service catalog can include multiple uniquely-named services that perform similar functions.

### Request and Response Types

The Message Queuing API supports JSON data serialization formats. You specify the request format by using the `Content-Type` header. The request format is required for calls that have a request body. You can specify the response format in requests either by using the Accept header or by adding a `.json` extension to the request URI. JSON is also used for the response format.

Response Format
Format | Accept Header | Query Extension | Default
JSON | application/json | .json | Yes

### Absolute Limits

Accounts are configured with default thresholds, or limits, that manage capacity and prevent abuse of the system.

The following table provides details about the absolute limits for the Message Queuing API.

These limits are configurable.

Absolute Limits
Type | Description | Limit | Default
--- | --- | --- | ---
Limits on Messages and Queues per Request | Number of queue records per page of results when listing queues | 1 - 1000 | 20
Number of messages per page of results when listing messages | 1 - 25 | 20
Number of messages that can be posted in a single request | 1 - 25 | 20
Number of messages that can be claimed at one time | 1 - 25 | 20
Number of messages that can be deleted in a single bulk delete request | 1 - 25 | 20
Number of messages that can be requested in a bulk get by ID | 1 - 25 | 20
Time Limits | Valid range for a message TTL | 60 - 1209600 seconds (1 minute to 14 days) | 1209600 seconds
Valid range for a claim TTL | 60 - 43200 seconds (1 minute to 12 hours) | 43200 seconds
Valid range for claim grace period | 60 - 43200 seconds (1 minute to 12 hours) | 43200 seconds
Data Limits | Queue metadata JSON | Cannot exceed 262144 bytes, including whitespace (256 KB) | 65536 bytes
Message body JSON | Cannot exceed 262144 bytes, including whitespace (256 KB) | 262144 bytes

For the 256 KB data limit for the message body JSON, if multiple messages are included in the request, this limit also applies to the sum of the message bodies added together. For example, you may have one message body of 256 KB, or up to 10 messages for which the sum of the message bodies is 256 KB. A 400 error is triggered if a single message body or the sum of multiple message bodies exceed 256 KB in a single request.

### Common Headers

Each request to the Message Queuing API must include certain standard and extended HTTP headers. These headers provide host, agent, authentication, and other pertinent information to the server. The following table provides the common headers used by the API.

Common Headers
Header | Description
--- | ---
Host | Host name of the API
Date | Current date and time
Accept | Media type desired; initially, only `application/json` is supported. Note: The `Accept` header is required.
Accept-Encoding | Specifies that the agent accepts gzip-encoded response bodies
Content-Type | `application/json`
Content-Length | For `POST` or `PUT` requests, the length in bytes of the message document being submitted
X-Auth-Token | Auth token
X-Project-Id | An ID for a project to which the value of X-Auth-Token grants access. Queues are created under this project.
Client-ID | A UUID for each client instance. The UUID must be submitted in its canonical form (for example, 3381af92-2b9e-11e3-b191-71861300734c). The client generates the Client-ID once. Client-ID persists between restarts of the client so the client should reuse that same Client-ID. Note: All message-related operations require the use of `Client-ID` in the headers to ensure that messages are not echoed back to the client that posted them, unless the client explicitly requests this.

### Date and Time Format

For the display and consumption of date and time values, the Message Queuing service uses a date format that complies with ISO 8601.

Service Date and Time Format

    yyyy-MM-dd'T'HH:mm:ss.SSSZ

For example, May 19, 2013 at 8:07:08 A.M., GMT-5 would have the following format:

    2013-05-19T08:07:08-05:00

The following table describes the date and time format codes.

Explanation of Date and Time Format Codes
Code | Description
yyyy | Four-digit year
MM | Two-digit month
dd | Two-digit day of at the month
T | Separator for date and time
HH | Two-digit hour of the day (00-23)
mm | Two-digit minutes of the hour
ss | Two-digit seconds of the minute
SSS | Three-digit milliseconds of the second
Z | RFC 822 time zone

### Faults

If any Message Queuing request results in an error, the queuing service returns an appropriate 4xx or 5xx HTTP status code, and the following information in the body:

- Title

- Description

- Internal code

- Link to more information

An example of an error message follows.

Error Message Example

    HTTP/1.1 400 Bad Request
    Content-Type: application/json

    {
      "title": "Unsupported limit",
      "description": "The given limit cannot be negative, and cannot be greater than 50.",
      "code": 1092,
      "link": {
        "rel": "help",
        "href": "http://docs.example.com/messages#limit",
        "text": "API documentation for the limit parameter"
      }
    }

Detailed error information for each API operation is included in the following sections. Additionally, brief error information is included in each operation description in .

#### Response Codes and Explanations

This section includes response codes and explanations for each of the Message Queuing API operations. The operations are described in .

##### Create Queue

The following table describes the response codes for the `Create Queue` operation.

Descriptions of Create Queue Response Codes
HTTP Response Code | Description
201 | Success (created)
204 | Success (already exists)
400 | Queue has a long name (greater than 64 bytes)
400 | Request header has missing fields
401 | Request header has invalid auth token
429 | Too many requests

##### List Queues

The following table describes the response codes for the `List Queues` operation.

Descriptions of List Queues Response Codes
HTTP Response Code | Description
200 | Success
200 | URI has invalid parameter (something other than limit, marker, or metadata)
204 | Request with non-existing value for marker
400 | Request with negative value for limit
400 | Request with zero value for limit
400 | Request with non-boolean value for metadata
400 | Header has missing fields
401 | Request header has invalid auth token
404 | Request a non-existing queue
406 | Header has Accept != "application/json"
429 | Too many requests

##### Delete Queue

The following table describes the response codes for the `Delete Queue` operation.

Descriptions of Delete Queue Response Codes
HTTP Response Code | Description
200 | URI has invalid parameters
204 | Success
204 | Delete a non-existing queue
400 | Header has missing fields
401 | Request header has invalid auth token
406 | Header has Accept != "application/json"
429 | Too many requests

##### Check Queue Existence

The following table describes the response codes for the `Check Queue Existence` operation.

Descriptions of Check Queue Existence Response Codes
HTTP Response Code | Description
204 | Check a non-existing queue
404 | Request a non-existing queue

##### Set Queue Metadata

The following table describes the response codes for the `Set Queue Metadata` operation.

Descriptions of Set Queue Metadata Response Codes
HTTP Response Code | Description
204 | Success
400 | Request body is empty
400 | Request body is greater than 64 KB
400 | Request body has malformed JSON
400 | Request with non-JSON body
406 | Request header has Accept != "application/json"
429 | Too many requests
| Request with UTF-16 char JSON body

##### Get Queue Metadata

The following table describes the response codes for the `Get Queue Metadata` operation.

Descriptions of Get Queue Metadata Response Codes
HTTP Response Code | Description
200 | Success
200 | No metadata exists for the queue
200 | URI has invalid parameters
400 | Header has missing fields
401 | Request header has invalid auth token
404 | Request metadata for a non-existing queue
406 | Header has Accept != "application/json"
429 | Too many requests

##### Get Queue Stats

The following table describes the response codes for the `Get Queue Stats` operation.

Descriptions of Get Queue Stats Response Codes
HTTP Response Code | Description
200 | Success
200 | URI has invalid parameters
400 | Header has missing fields
401 | Request header has invalid auth token
404 | Request stats for a non-existing queue
406 | Header has Accept != "application/json"
429 | Too many requests

##### Post Message

The following table describes the response codes for the `Post Message` operation.

Descriptions of Post Message Response Codes
HTTP Response Code | Description
200 | URI has invalid parameters
201 | Success
201 | Post multiple messages in a single request, few of them succeed (with "partial": true)
400 | Post greater than 100 messages with a single request
400 | Post with request body greater than 4 KB
400 | Post with invalid JSON request body
400 | Post with non-JSON request body
400 | Post with non-JSON message
400 | Post with TTL less than 60
400 | Post with TTL greater than 1209600
400 | Post with non-integer value of TTL
400 | Post with negative value of TTL
400 | Post without TTL in request body
400 | Post without "body" in request body
400 | Post with no request body
400 | Post with messages encapsulated in multiple arrays
400 | Header has missing fields
401 | Header has invalid auth token
404 | Post message to a non-existing queue
406 | Header has Accept != "application/json"
429 | Too many requests
503 | Post multiple messages in a single request and all of them fail

##### Get Messages

The following table describes the response codes for the `Get Messages` operation.

Descriptions of Get Messages Response Codes
HTTP Response Code | Description
200 | Success (query matched 1 or more messages)
200 | Request message with invalid parameter (something other than marker, limit, or echo)
204 | Success (query has no match)
204 | Request message with non-existing marker
400 | Request message with limit greater than 100
400 | Post with request body greater than 4 KB
400 | Request message with limit less than or equal to 0
400 | Request message with non-boolean value for echo
401 | Header has invalid auth token
404 | Request message from a non-existing queue
406 | Header has Accept != "application/json"
429 | Too many requests

##### Get a Set of Messages by ID

The following table describes the response codes for the `Get a Set of Messages by ID` operation.

Descriptions of Get a Set of Messages by ID Response Codes
HTTP Response Code | Description
200 | Success
200 | URI has invalid parameters
400 | Request message with limit greater than 100
401 | Post with request body greater than 4 KB
404 | Request message with limit less than or equal to 0
404 | Request message with non-boolean value for echo
406 | Header has invalid auth token
429 | Too many requests

##### Delete a Set of Messages by ID

The following table describes the response codes for the `Delete a Set of Messages by ID` operation.

Descriptions of Delete a Set of Messages by ID Response Codes
HTTP Response Code | Description
204 | Success
204 | Delete message from a non-existing queue
204 | Delete a non-existing message
400 | Header has missing fields
401 | Header has invalid auth token
403 | Delete with expired claim ID
403 | Delete with non-existing claim ID
403 | Delete claimed message without providing a claim ID
406 | Header has Accept != "application/json"
429 | Too many requests

##### Get a Specific Message

The following table describes the response codes for the `Get a Specific Message` operation.

Descriptions of Get a Specific Message Response Codes
HTTP Response Code | Description
200 | Success(query matched the message)
200 | URI has invalid parameters
400 | Header has missing fields
401 | Header has invalid auth token
404 | Request message from a non-existing queue
404 | Request a non-existing message
404 | Request an expired message
406 | Header has Accept != "application/json"
429 | Too many requests

##### Delete Message

The following table describes the response codes for the `Delete Message` operation.

Descriptions of Delete Message Response Codes
HTTP Response Code | Description
200 | URI has invalid parameters
204 | Success
204 | Delete message from a non-existing queue
204 | Delete a non-existing message
400 | Header has missing fields
401 | Header has invalid auth token
403 | Delete with expired claim ID
403 | Delete with non-existing claim ID
403 | Delete claimed message without providing a claim ID
406 | Header has Accept != "application/json"
429 | Too many requests

##### Claim Messages

The following table describes the response codes for the `Claim Messages` operation.

Descriptions of Claim Messages Response Codes
HTTP Response Code | Description
201 | Success
204 | Success (empty queue with no messages to claim)
204 | Claim message from a non-existing queue
400 | Claim message with invalid value for limit (non-integer, or less than or equal to 0)
400 | Claim message with limit greater than 100)
400 | Claim message with no request body
400 | Claim message with no TTL field
400 | Claim message with invalid TTL value (non-integer, or less than or equal to 0)
400 | Claim message with invalid JSON
400 | Claim message with non-JSON request body
400 | Claim message with missing header fields
401 | Claim message with invalid auth token
404 | Item not found
406 | Header has Accept != "application/json"
429 | Too many requests

##### Query Claim

The following table describes the response codes for the `Query Claim` operation.

Descriptions of Query Claim Response Codes
HTTP Response Code | Description
200 | Success
200 | Query claim with invalid URI parameters
400 | Query claim with missing header fields
401 | Query claim with invalid auth token
404 | Query expired claim
404 | Query claim from a non-existing queue
404 | Query claim with non-existing claim ID
406 | Header has Accept != "application/json"
429 | Too many requests

##### Update Claim

The following table describes the response codes for the `Update Claim` operation.

Descriptions of Update Claim Response Codes
HTTP Response Code | Description
200 | Update claim with invalid URI parameters
204 | Success
400 | Update claim with no request body
400 | Update claim with invalid JSON in request body
400 | Update claim with non-JSON request body
400 | Update claim with missing header fields
401 | Update claim with invalid auth token
404 | Update expired claim
404 | Update claim on a non-existing queue
404 | Update a non-existing claim
406 | Update claim with content-type != application/json-patch
429 | Too many requests

##### Release Claim

The following table describes the response codes for the `Release Claim` operation.

Descriptions of Release Claim Response Codes
HTTP Response Code | Description
200 | Release claim with invalid URI parameters
204 | Success
204 | Release expired claim
204 | Release a non-existing claim
400 | Release claim with missing header fields
401 | Release claim with invalid auth token
404 | Release claim from a non-existing queue
406 | Header has Accept != "application/json"
429 | Too many requests

## API Operations

This chapter describes each of the Message Queuing operations, which are summarized in the following table.

### Home Document

The section describes how to get the home document for Message Queuing.

The home document schema is currently not ratified in OpenStack and may change.

### Queues

This section describes queue operations that are supported by the Message Queuing API.

### Messages

This section describes message operations that are supported by the Message Queuing API.

All message-related operations require that Client-ID be included in the headers. This is to ensure that messages are not echoed back to the client that posted them unless the client explicitly requests this. For more information, see Client-ID in .

### Claims

This section describes claim operations that are supported by the Message Queuing API.

## Glossary

Claim

  The process of a worker checking out a message to perform a task. Claiming a message prevents other workers from attempting to process the same messages.

Claim TTL

  Defines how long a message will be in claimed state. A message can be claimed by one worker at a time.

Consumer

  A server that claims messages from the queue.

Message

  A task, a notification, or any meaningful data that a producer or publisher sends to the queue. A message exists until it is deleted by a recipient or automatically by the system based on a TTL (time-to-live) value.

Message TTL

  Defines how long a message will be accessible.

Producer

  A server or application that sends messages to the queue.

Producer - Consumer

  A pattern where each worker application that reads the queue has to claim the message in order to prevent duplicate processing. Later, when work is done, the worker is responsible for deleting the message. If message is not deleted in a predefined time, it can be claimed by other workers.

Publisher

  A server or application that posts messages to the queue with the intent to distribute information or updates to multiple subscribers.

Publisher - Subscriber

  A pattern where all worker applications have access to all messages in the queue. Workers cannot delete or update messages.

Queue

  The entity that holds messages. Ideally, a queue is created per work type. For example, if you want to compress files, you would create a queue dedicated to this job. Any application that reads from this queue would only compress files.

Subscriber

  An observer that watches messages like an RSS feed but does not claim any messages.

TTL

  Time-to-live value.

Worker

  A client that claims messages from the queue and performs actions based on those messages.