panko.storage package

Submodules

panko.storage.base module

Base classes for storage engines

class panko.storage.base.Connection(conf)

Bases: object

Base class for event storage system connections.

CAPABILITIES = {'events': {'query': {'simple': False}}}
STORAGE_CAPABILITIES = {'storage': {'production_ready': False}}
static clear()

Clear database.

static clear_expired_data(ttl, max_count=None)

Clear expired data from the backend storage system.

Clearing occurs according to the time-to-live. :param ttl: Number of seconds to keep records for. :param max_count: Number of records to delete.

classmethod get_capabilities()

Return an dictionary with the capabilities of each driver.

static get_event_types()

Return all event types as an iterable of strings.

static get_events(event_filter, pagination=None)

Return an iterable of model.Event objects.

classmethod get_storage_capabilities()

Return a dictionary representing the performance capabilities.

This is needed to evaluate the performance of each driver.

static get_trait_types(event_type)

Return a dictionary containing the name and data type of the trait.

Only trait types for the provided event_type are returned. :param event_type: the type of the Event

static get_traits(event_type, trait_type=None)

Return all trait instances associated with an event_type.

If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by

static record_events(events)

Write the events to the backend storage system.

Parameters

events – a list of model.Event objects.

static upgrade()

Migrate the database to version or the most recent version.

class panko.storage.base.Model(**kwds)

Bases: object

Base class for storage API models.

as_dict()

panko.storage.impl_elasticsearch module

class panko.storage.impl_elasticsearch.Connection(url, conf)

Bases: panko.storage.base.Connection

Put the event data into an ElasticSearch db.

Events in ElasticSearch are indexed by day and stored by event_type. An example document:

{"_index":"events_2014-10-21",
 "_type":"event_type0",
 "_id":"dc90e464-65ab-4a5d-bf66-ecb956b5d779",
 "_score":1.0,
 "_source":{"timestamp": "2014-10-21T20:02:09.274797"
            "traits": {"id4_0": "2014-10-21T20:02:09.274797",
                       "id3_0": 0.7510790937279408,
                       "id2_0": 5,
                       "id1_0": "18c97ba1-3b74-441a-b948-a702a30cbce2"}
           }
}
CAPABILITIES = {'events': {'query': {'simple': True}}}
STORAGE_CAPABILITIES = {'storage': {'production_ready': True}}
get_event_types()

Return all event types as an iterable of strings.

get_events(event_filter, pagination=None)

Return an iterable of model.Event objects.

get_trait_types(event_type)

Return a dictionary containing the name and data type of the trait.

Only trait types for the provided event_type are returned. :param event_type: the type of the Event

get_traits(event_type, trait_type=None)

Return all trait instances associated with an event_type.

If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by

record_events(events)

Write the events to the backend storage system.

Parameters

events – a list of model.Event objects.

upgrade()

Migrate the database to version or the most recent version.

panko.storage.impl_log module

class panko.storage.impl_log.Connection(conf)

Bases: panko.storage.base.Connection

Log event data.

static clear_expired_data(ttl, max_count)

Clear expired data from the backend storage system.

Clearing occurs according to the time-to-live.

Parameters

  • ttl – Number of seconds to keep records for.

  • max_count – Number of records to delete.

panko.storage.impl_mongodb module

MongoDB storage backend

class panko.storage.impl_mongodb.Connection(url, conf)

Bases: panko.storage.pymongo_base.Connection

Put the event data into a MongoDB database.

CONNECTION_POOL = <panko.storage.mongo.utils.ConnectionPool object>
clear()

Clear database.

clear_expired_data(ttl, max_count=None)

Clear expired data from the backend storage system.

Clearing occurs according to the time-to-live.

Parameters

  • ttl – Number of seconds to keep records for.

  • max_count – Number of records to delete (not used for MongoDB).

static update_ttl(ttl, ttl_index_name, index_field, coll)

Update or create time_to_live indexes.

Parameters

  • ttl – time to live in seconds.

  • ttl_index_name – name of the index we want to update or create.

  • index_field – field with the index that we need to update.

  • coll – collection which indexes need to be updated.

upgrade()

Migrate the database to version or the most recent version.

panko.storage.impl_sqlalchemy module

SQLAlchemy storage backend.

class panko.storage.impl_sqlalchemy.Connection(url, conf)

Bases: panko.storage.base.Connection

Put the event data into a SQLAlchemy database.

Tables:

- EventType
  - event definition
  - { id: event type id
      desc: description of event
      }
- Event
  - event data
  - { id: event id
      message_id: message id
      generated = timestamp of event
      event_type_id = event type -> eventtype.id
      }
- TraitInt
  - int trait value
  - { event_id: event -> event.id
      key: trait name
      value: integer value
      }
- TraitDatetime
  - datetime trait value
  - { event_id: event -> event.id
      key: trait name
      value: datetime value
      }
- TraitText
  - text trait value
  - { event_id: event -> event.id
      key: trait name
      value: text value
      }
- TraitFloat
  - float trait value
  - { event_id: event -> event.id
      key: trait name
      value: float value
      }
CAPABILITIES = {'events': {'query': {'simple': True}}}
STORAGE_CAPABILITIES = {'storage': {'production_ready': True}}
clear()

Clear database.

clear_expired_data(ttl, max_count)

Clear expired data from the backend storage system.

Clearing occurs according to the time-to-live.

Parameters

  • ttl – Number of seconds to keep records for.

  • max_count – Number of records to delete.

static dress_url(url)
get_event_types()

Return all event types as an iterable of strings.

get_events(event_filter, pagination=None)

Return an iterable of model.Event objects.

Parameters

  • event_filter – EventFilter instance

  • pagination – Pagination parameters.

get_trait_types(event_type)

Return a dictionary containing the name and data type of the trait.

Only trait types for the provided event_type are returned. :param event_type: the type of the Event

get_traits(event_type, trait_type=None)

Return all trait instances associated with an event_type.

If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by

record_events(event_models)

Write the events to SQL database via sqlalchemy.

Parameters

event_models – a list of model.Event objects.

upgrade()

Migrate the database to version or the most recent version.

panko.storage.models module

Model classes for use in the events storage API.

class panko.storage.models.Event(message_id, event_type, generated, traits, raw)

Bases: panko.storage.base.Model

A raw event from the source system. Events have Traits.

Metrics will be derived from one or more Events.

DEFAULT_DIR = 'asc'
DEFAULT_SORT = [('generated', 'asc'), ('message_id', 'asc')]
DUPLICATE = 1
INCOMPATIBLE_TRAIT = 3
PRIMARY_KEY = 'message_id'
SUPPORT_DIRS = ('asc', 'desc')
SUPPORT_SORT_KEYS = ('message_id', 'generated')
UNKNOWN_PROBLEM = 2
append_trait(trait_model)
serialize()
class panko.storage.models.Trait(name, dtype, value)

Bases: panko.storage.base.Model

A Trait is a key/value pair of data on an Event.

The value is variant record of basic data types (int, date, float, etc).

DATETIME_TYPE = 4
FLOAT_TYPE = 3
INT_TYPE = 2
NONE_TYPE = 0
TEXT_TYPE = 1
classmethod convert_value(trait_type, value)
classmethod get_name_by_type(type_id)
classmethod get_type_by_name(type_name)
get_type_name()
classmethod get_type_names()
serialize()
type_names = {0: 'none', 1: 'string', 2: 'integer', 3: 'float', 4: 'datetime'}
panko.storage.models.serialize_dt(value)

Serializes parameter if it is datetime.

panko.storage.pymongo_base module

Common functions for MongoDB backend

class panko.storage.pymongo_base.Connection(conf)

Bases: panko.storage.base.Connection

Base event Connection class for MongoDB driver.

CAPABILITIES = {'events': {'query': {'simple': True}}}
STORAGE_CAPABILITIES = {'storage': {'production_ready': True}}
get_event_types()

Return all event types as an iter of strings.

get_events(event_filter, pagination=None)

Return an iter of models.Event objects.

Parameters

  • event_filter – storage.EventFilter object, consists of filters for events that are stored in database.

  • pagination – Pagination parameters.

get_trait_types(event_type)

Return a dictionary containing the name and data type of the trait.

Only trait types for the provided event_type are returned.

Parameters

event_type – the type of the Event.

get_traits(event_type, trait_name=None)

Return all trait instances associated with an event_type.

If trait_type is specified, only return instances of that trait type.

Parameters

  • event_type – the type of the Event to filter by

  • trait_name – the name of the Trait to filter by

record_events(event_models)

Write the events to database.

Parameters

event_models – a list of models.Event objects.

Module contents

Storage backend management

class panko.storage.EventFilter(start_timestamp=None, end_timestamp=None, event_type=None, message_id=None, traits_filter=None, admin_proj=None)

Bases: object

Properties for building an Event query.

Parameters

  • start_timestamp – UTC start datetime (mandatory)

  • end_timestamp – UTC end datetime (mandatory)

  • event_type – the name of the event. None for all.

  • message_id – the message_id of the event. None for all.

  • admin_proj – the project_id of admin role. None if non-admin user.

  • traits_filter – the trait filter dicts, all of which are optional. This parameter is a list of dictionaries that specify trait values:

{'key': <key>,
'string': <value>,
'integer': <value>,
'datetime': <value>,
'float': <value>,
'op': <eq, lt, le, ne, gt or ge> }
exception panko.storage.InvalidMarker

Bases: Exception

Invalid pagination marker parameters

exception panko.storage.StorageBadAggregate

Bases: Exception

Error raised when an aggregate is unacceptable to storage backend.

code = 400
exception panko.storage.StorageBadVersion

Bases: Exception

Error raised when the storage backend version is not good enough.

exception panko.storage.StorageUnknownWriteError

Bases: Exception

Error raised when an unknown error occurs while recording.

panko.storage.get_connection(url, conf)

Return an open connection to the database.

panko.storage.get_connection_from_config(conf)