panko.storage package¶
Subpackages¶
Submodules¶
panko.storage.base module¶
Base classes for storage engines
-
class
panko.storage.base.
Connection
(conf)¶ Bases:
object
Base class for event storage system connections.
-
CAPABILITIES
= {'events': {'query': {'simple': False}}}¶
-
STORAGE_CAPABILITIES
= {'storage': {'production_ready': False}}¶
-
static
clear
()¶ Clear database.
-
static
clear_expired_data
(ttl, max_count=None)¶ Clear expired data from the backend storage system.
Clearing occurs according to the time-to-live. :param ttl: Number of seconds to keep records for. :param max_count: Number of records to delete.
-
classmethod
get_capabilities
()¶ Return an dictionary with the capabilities of each driver.
-
static
get_event_types
()¶ Return all event types as an iterable of strings.
-
static
get_events
(event_filter, pagination=None)¶ Return an iterable of model.Event objects.
-
classmethod
get_storage_capabilities
()¶ Return a dictionary representing the performance capabilities.
This is needed to evaluate the performance of each driver.
-
static
get_trait_types
(event_type)¶ Return a dictionary containing the name and data type of the trait.
Only trait types for the provided event_type are returned. :param event_type: the type of the Event
-
static
get_traits
(event_type, trait_type=None)¶ Return all trait instances associated with an event_type.
If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by
-
static
record_events
(events)¶ Write the events to the backend storage system.
- Parameters
events – a list of model.Event objects.
-
static
upgrade
()¶ Migrate the database to version or the most recent version.
-
panko.storage.impl_elasticsearch module¶
-
class
panko.storage.impl_elasticsearch.
Connection
(url, conf)¶ Bases:
panko.storage.base.Connection
Put the event data into an ElasticSearch db.
Events in ElasticSearch are indexed by day and stored by event_type. An example document:
{"_index":"events_2014-10-21", "_type":"event_type0", "_id":"dc90e464-65ab-4a5d-bf66-ecb956b5d779", "_score":1.0, "_source":{"timestamp": "2014-10-21T20:02:09.274797" "traits": {"id4_0": "2014-10-21T20:02:09.274797", "id3_0": 0.7510790937279408, "id2_0": 5, "id1_0": "18c97ba1-3b74-441a-b948-a702a30cbce2"} } }
-
CAPABILITIES
= {'events': {'query': {'simple': True}}}¶
-
STORAGE_CAPABILITIES
= {'storage': {'production_ready': True}}¶
-
get_event_types
()¶ Return all event types as an iterable of strings.
-
get_events
(event_filter, pagination=None)¶ Return an iterable of model.Event objects.
-
get_trait_types
(event_type)¶ Return a dictionary containing the name and data type of the trait.
Only trait types for the provided event_type are returned. :param event_type: the type of the Event
-
get_traits
(event_type, trait_type=None)¶ Return all trait instances associated with an event_type.
If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by
-
record_events
(events)¶ Write the events to the backend storage system.
- Parameters
events – a list of model.Event objects.
-
upgrade
()¶ Migrate the database to version or the most recent version.
-
panko.storage.impl_log module¶
-
class
panko.storage.impl_log.
Connection
(conf)¶ Bases:
panko.storage.base.Connection
Log event data.
-
static
clear_expired_data
(ttl, max_count)¶ Clear expired data from the backend storage system.
Clearing occurs according to the time-to-live.
- Parameters
ttl – Number of seconds to keep records for.
max_count – Number of records to delete.
-
static
panko.storage.impl_mongodb module¶
MongoDB storage backend
-
class
panko.storage.impl_mongodb.
Connection
(url, conf)¶ Bases:
panko.storage.pymongo_base.Connection
Put the event data into a MongoDB database.
-
CONNECTION_POOL
= <panko.storage.mongo.utils.ConnectionPool object>¶
-
clear
()¶ Clear database.
-
clear_expired_data
(ttl, max_count=None)¶ Clear expired data from the backend storage system.
Clearing occurs according to the time-to-live.
- Parameters
ttl – Number of seconds to keep records for.
max_count – Number of records to delete (not used for MongoDB).
-
static
update_ttl
(ttl, ttl_index_name, index_field, coll)¶ Update or create time_to_live indexes.
- Parameters
ttl – time to live in seconds.
ttl_index_name – name of the index we want to update or create.
index_field – field with the index that we need to update.
coll – collection which indexes need to be updated.
-
upgrade
()¶ Migrate the database to version or the most recent version.
-
panko.storage.impl_sqlalchemy module¶
SQLAlchemy storage backend.
-
class
panko.storage.impl_sqlalchemy.
Connection
(url, conf)¶ Bases:
panko.storage.base.Connection
Put the event data into a SQLAlchemy database.
Tables:
- EventType - event definition - { id: event type id desc: description of event } - Event - event data - { id: event id message_id: message id generated = timestamp of event event_type_id = event type -> eventtype.id } - TraitInt - int trait value - { event_id: event -> event.id key: trait name value: integer value } - TraitDatetime - datetime trait value - { event_id: event -> event.id key: trait name value: datetime value } - TraitText - text trait value - { event_id: event -> event.id key: trait name value: text value } - TraitFloat - float trait value - { event_id: event -> event.id key: trait name value: float value }
-
CAPABILITIES
= {'events': {'query': {'simple': True}}}¶
-
STORAGE_CAPABILITIES
= {'storage': {'production_ready': True}}¶
-
clear
()¶ Clear database.
-
clear_expired_data
(ttl, max_count)¶ Clear expired data from the backend storage system.
Clearing occurs according to the time-to-live.
- Parameters
ttl – Number of seconds to keep records for.
max_count – Number of records to delete.
-
static
dress_url
(url)¶
-
get_event_types
()¶ Return all event types as an iterable of strings.
-
get_events
(event_filter, pagination=None)¶ Return an iterable of model.Event objects.
- Parameters
event_filter – EventFilter instance
pagination – Pagination parameters.
-
get_trait_types
(event_type)¶ Return a dictionary containing the name and data type of the trait.
Only trait types for the provided event_type are returned. :param event_type: the type of the Event
-
get_traits
(event_type, trait_type=None)¶ Return all trait instances associated with an event_type.
If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by
-
record_events
(event_models)¶ Write the events to SQL database via sqlalchemy.
- Parameters
event_models – a list of model.Event objects.
-
upgrade
()¶ Migrate the database to version or the most recent version.
-
panko.storage.models module¶
Model classes for use in the events storage API.
-
class
panko.storage.models.
Event
(message_id, event_type, generated, traits, raw)¶ Bases:
panko.storage.base.Model
A raw event from the source system. Events have Traits.
Metrics will be derived from one or more Events.
-
DEFAULT_DIR
= 'asc'¶
-
DEFAULT_SORT
= [('generated', 'asc'), ('message_id', 'asc')]¶
-
DUPLICATE
= 1¶
-
INCOMPATIBLE_TRAIT
= 3¶
-
PRIMARY_KEY
= 'message_id'¶
-
SUPPORT_DIRS
= ('asc', 'desc')¶
-
SUPPORT_SORT_KEYS
= ('message_id', 'generated')¶
-
UNKNOWN_PROBLEM
= 2¶
-
append_trait
(trait_model)¶
-
serialize
()¶
-
-
class
panko.storage.models.
Trait
(name, dtype, value)¶ Bases:
panko.storage.base.Model
A Trait is a key/value pair of data on an Event.
The value is variant record of basic data types (int, date, float, etc).
-
DATETIME_TYPE
= 4¶
-
FLOAT_TYPE
= 3¶
-
INT_TYPE
= 2¶
-
NONE_TYPE
= 0¶
-
TEXT_TYPE
= 1¶
-
classmethod
convert_value
(trait_type, value)¶
-
classmethod
get_name_by_type
(type_id)¶
-
classmethod
get_type_by_name
(type_name)¶
-
get_type_name
()¶
-
classmethod
get_type_names
()¶
-
serialize
()¶
-
type_names
= {0: 'none', 1: 'string', 2: 'integer', 3: 'float', 4: 'datetime'}¶
-
-
panko.storage.models.
serialize_dt
(value)¶ Serializes parameter if it is datetime.
panko.storage.pymongo_base module¶
Common functions for MongoDB backend
-
class
panko.storage.pymongo_base.
Connection
(conf)¶ Bases:
panko.storage.base.Connection
Base event Connection class for MongoDB driver.
-
CAPABILITIES
= {'events': {'query': {'simple': True}}}¶
-
STORAGE_CAPABILITIES
= {'storage': {'production_ready': True}}¶
-
get_event_types
()¶ Return all event types as an iter of strings.
-
get_events
(event_filter, pagination=None)¶ Return an iter of models.Event objects.
- Parameters
event_filter – storage.EventFilter object, consists of filters for events that are stored in database.
pagination – Pagination parameters.
-
get_trait_types
(event_type)¶ Return a dictionary containing the name and data type of the trait.
Only trait types for the provided event_type are returned.
- Parameters
event_type – the type of the Event.
-
get_traits
(event_type, trait_name=None)¶ Return all trait instances associated with an event_type.
If trait_type is specified, only return instances of that trait type.
- Parameters
event_type – the type of the Event to filter by
trait_name – the name of the Trait to filter by
-
record_events
(event_models)¶ Write the events to database.
- Parameters
event_models – a list of models.Event objects.
-
Module contents¶
Storage backend management
-
class
panko.storage.
EventFilter
(start_timestamp=None, end_timestamp=None, event_type=None, message_id=None, traits_filter=None, admin_proj=None)¶ Bases:
object
Properties for building an Event query.
- Parameters
start_timestamp – UTC start datetime (mandatory)
end_timestamp – UTC end datetime (mandatory)
event_type – the name of the event. None for all.
message_id – the message_id of the event. None for all.
admin_proj – the project_id of admin role. None if non-admin user.
traits_filter – the trait filter dicts, all of which are optional. This parameter is a list of dictionaries that specify trait values:
{'key': <key>, 'string': <value>, 'integer': <value>, 'datetime': <value>, 'float': <value>, 'op': <eq, lt, le, ne, gt or ge> }
-
exception
panko.storage.
InvalidMarker
¶ Bases:
Exception
Invalid pagination marker parameters
-
exception
panko.storage.
StorageBadAggregate
¶ Bases:
Exception
Error raised when an aggregate is unacceptable to storage backend.
-
code
= 400¶
-
-
exception
panko.storage.
StorageBadVersion
¶ Bases:
Exception
Error raised when the storage backend version is not good enough.
-
exception
panko.storage.
StorageUnknownWriteError
¶ Bases:
Exception
Error raised when an unknown error occurs while recording.
-
panko.storage.
get_connection
(url, conf)¶ Return an open connection to the database.
-
panko.storage.
get_connection_from_config
(conf)¶