========================================
Writing stages for the storage pipelines
========================================
Introduction
~~~~~~~~~~~~
A pipeline is a set of stages needed to process a request. When a new request
comes to Zaqar, first the message goes through the transport layer pipeline and
then through one of the storage layer pipelines depending on the type of
operation of each particular request. For example, if Zaqar receives a
request to make a queue-related operation, the storage layer pipeline will be
``queue pipeline``. Zaqar always has the actual storage controller as the
final storage layer pipeline stage.
By setting the options in the ``[storage]`` section of ``zaqar.conf``
you can add additional stages to these storage layer pipelines:
* **Claim pipeline**
* **Message pipeline** with built-in stage available to use:
* ``zaqar.notification.notifier`` - sends notifications to the queue
subscribers on each incoming message to the queue, i.e. enables
notifications functionality.
* **Queue pipeline**
* **Subscription pipeline**
The storage layer pipelines options are empty by default, because additional
stages can affect the performance of Zaqar. Depending on the stages, the
sequence in which the option values are listed does matter or not.
You can add your own external stages to the storage layer pipelines.
Things to know before writing the stage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Stages in the pipeline must implement storage controller methods they need
to hook. You can find all available to hook methods in the abstract classes in
``zaqar/storage/base.py``. For example, if you're looking for all methods
available to hook for the queue storage layer pipeline, see ``Queue``
class in ``zaqar/storage/base.py``. As you can see, Zaqar's built-in stage
``zaqar.notification.notifier`` implements ``post`` method of
``zaqar.storage.base.Message`` abstract class.
A stage can halt the pipeline immediate by returning a value that is not
None; otherwise, processing will continue to the next stage, ending with the
actual storage controller.
.. warning::
For the most of the cases it does not matter what non-None value the storage
pipeline returns, but sometimes the returned value is used by the transport
layer and you have to be careful. For example, during queue creation
request, if the storage driver returns ``True``, the transport layer
responds to the client with the ``201`` http response code, if ``False``,
it responds with ``204`` http response code. See:
``zaqar.transport.wsgi.v2_0.queues.ItemResource#on_put``.
Zaqar finds stages with their source codes through the Python entry points
mechanism. All Python packages containing stages for Zaqar must register
their stages under ``zaqar.storage.stages`` entry point group during their
install either by ``setup.py`` or by ``setup.cfg``. If the stage is registered,
and the name of the stage's entry point is specified by the user in the one of
``zaqar.conf`` storage layer pipeline options, the stage will be loaded to
the particular storage layer pipeline. Zaqar imports stages as plugins. See
``zaqar.storage.pipeline#_get_storage_pipeline``.
For additional information about plugins see: `Stevedore - Creating Plugins`_
and `Stevedore - Loading the Plugins`_.
Example of external stage (written outside Zaqar package)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is an example of small package with a stage that can process queue-related
requests in Zaqar. The stage does not do anything useful, but is good as
example.
File tree structure of the package:
.. code-block:: none
.
├── setup.py
└── ubershystages
├── __init__.py
└── queues
├── __init__.py
└── lovely.py
2 directories, 4 files
``setup.py``:
.. code-block:: python
from setuptools import setup, find_packages
setup(
name='ubershystages',
version='1.0',
description='Demonstration package for Zaqar with plugin pipeline stage',
author='Ubershy',
author_email='ubershy@gmail.com',
url='',
classifiers=['Development Status :: 3 - Alpha',
'License :: OSI Approved :: Apache Software License',
'Programming Language :: Python',
'Programming Language :: Python :: 2',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3',
'Programming Language :: Python :: 3.4',
'Intended Audience :: Developers',
'Environment :: Console',
],
platforms=['Any'],
scripts=[],
packages=find_packages(),
include_package_data=True,
entry_points={
'zaqar.storage.stages': [
'ubershy.lovelyplugin = ubershystages.queues.lovely:LovelyStage',
],
},
zip_safe=False,
)
``lovely.py``:
.. code-block:: python
class LovelyStage(object):
"""This stage:
1. Prints 'Lovely stage is processing request...' on each queue creation or
deletion request.
2. Prints 'Oh, what a lovely day!' on each creation request of a queue
named 'lovely'.
3. Prevents deletion of a queue named 'lovely' and prints 'Secretly keeping
lovely queue' on such attempt.
"""
def __init__(self, *args, **kwargs):
print("Lovely stage is loaded!")
def create(self, name, metadata=None, project=None):
"""Stage's method which processes queue creation request.
:param name: The queue name
:param project: Project id
"""
self.printprocessing()
if name == 'lovely':
print("Oh, what a lovely day!")
def delete(self, name, project=None):
"""Stage's method which processes queue deletion request.
:param name: The queue name
:param project: Project id
:returns: Something non-None, if the queue has a name 'lovely'. It will
stop further processing through the other stages of the pipeline, and
the request will not reach the storage controller driver, preventing
queue deletion from the database.
"""
self.printprocessing()
if name == 'lovely':
print('Secretly keeping lovely queue')
something = "shhh... it's a bad practice"
return something
def printprocessing(self):
print('Lovely stage is processing request...')
To install the package to the system in the root directory of the package run:
.. code-block:: console
# pip install -e .
In ``zaqar.conf`` add ``ubershy.lovelyplugin`` to the ``queue_pipeline``
option:
.. code-block:: ini
[storage]
queue_pipeline = ubershy.lovelyplugin
Start Zaqar:
.. code-block:: console
$ zaqar-server
If the stage has successfully loaded to Zaqar you will see amongst terminal
output lines the ``Lovely stage is loaded!`` line. Then you can try to perform
queue create and queue delete operations with the queue 'lovely' and see what
will happen in Zaqar's database.
.. note::
You can hold multiple stages in one package, just be sure that all stages
will be registered as entry points. For example, in the ``setup.py`` you
can register additional ``ubershy.nastyplugin`` stage:
.. code-block:: python
entry_points={
'zaqar.storage.stages': [
'ubershy.lovelyplugin = ubershystages.queues.lovely:LovelyStage',
'ubershy.nastyplugin = ubershystages.messages.nasty:NastyStage',
],
},
.. _`Stevedore - Creating Plugins`: http://docs.openstack.org/developer/stevedore/tutorial/creating_plugins.html
.. _`Stevedore - Loading the Plugins`: http://docs.openstack.org/developer/stevedore/tutorial/loading.html