Data processing and pipelines¶

Rate of change transformer¶

Transformer that computes the change in value between two data points in time. In the case of the transformer that creates the cpu_util meter, the definition looks like:

transformers:
    - name: "rate_of_change"
      parameters:
          target:
              name: "cpu_util"
              unit: "%"
              type: "gauge"
              scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"

The rate of change transformer generates the cpu_util meter from the sample values of the cpu counter, which represents cumulative CPU time in nanoseconds. The transformer definition above defines a scale factor (for nanoseconds and multiple CPUs), which is applied before the transformation derives a sequence of gauge samples with unit %, from sequential values of the cpu meter.

The definition for the disk I/O rate, which is also generated by the rate of change transformer:

transformers:
    - name: "rate_of_change"
      parameters:
          source:
              map_from:
                  name: "disk\\.(read|write)\\.(bytes|requests)"
                  unit: "(B|request)"
          target:
              map_to:
                  name: "disk.\\1.\\2.rate"
                  unit: "\\1/s"
              type: "gauge"

Unit conversion transformer¶

Transformer to apply a unit conversion. It takes the volume of the meter and multiplies it with the given scale expression. Also supports map_from and map_to like the rate of change transformer.

Sample configuration:

transformers:
    - name: "unit_conversion"
      parameters:
          target:
              name: "disk.kilobytes"
              unit: "KB"
              scale: "volume * 1.0 / 1024.0"

With map_from and map_to:

transformers:
    - name: "unit_conversion"
      parameters:
          source:
              map_from:
                  name: "disk\\.(read|write)\\.bytes"
          target:
              map_to:
                  name: "disk.\\1.kilobytes"
              scale: "volume * 1.0 / 1024.0"
              unit: "KB"

Aggregator transformer¶

A transformer that sums up the incoming samples until enough samples have come in or a timeout has been reached.

Timeout can be specified with the retention_time option. If you want to flush the aggregation, after a set number of samples have been aggregated, specify the size parameter.

The volume of the created sample is the sum of the volumes of samples that came into the transformer. Samples can be aggregated by the attributes project_id, user_id and resource_metadata. To aggregate by the chosen attributes, specify them in the configuration and set which value of the attribute to take for the new sample (first to take the first sample’s attribute, last to take the last sample’s attribute, and drop to discard the attribute).

To aggregate 60s worth of samples by resource_metadata and keep the resource_metadata of the latest received sample:

transformers:
    - name: "aggregator"
      parameters:
          retention_time: 60
          resource_metadata: last

To aggregate each 15 samples by user_id and resource_metadata and keep the user_id of the first received sample and drop the resource_metadata:

transformers:
    - name: "aggregator"
      parameters:
          size: 15
          user_id: first
          resource_metadata: drop

Accumulator transformer¶

This transformer simply caches the samples until enough samples have arrived and then flushes them all down the pipeline at once:

transformers:
    - name: "accumulator"
      parameters:
          size: 15

Multi meter arithmetic transformer¶

This transformer enables us to perform arithmetic calculations over one or more meters and/or their metadata, for example:

memory_util = 100 * memory.usage / memory

A new sample is created with the properties described in the target section of the transformer’s configuration. The sample’s volume is the result of the provided expression. The calculation is performed on samples from the same resource.

Example configuration:

transformers:
    - name: "arithmetic"
      parameters:
        target:
          name: "memory_util"
          unit: "%"
          type: "gauge"
          expr: "100 * $(memory.usage) / $(memory)"

To demonstrate the use of metadata, the following implementation of a novel meter shows average CPU time per core:

transformers:
    - name: "arithmetic"
      parameters:
        target:
          name: "avg_cpu_per_core"
          unit: "ns"
          type: "cumulative"
          expr: "$(cpu) / ($(cpu).resource_metadata.cpu_number or 1)"

Delta transformer¶

This transformer calculates the change between two sample datapoints of a resource. It can be configured to capture only the positive growth deltas.

Example configuration:

transformers:
    - name: "delta"
      parameters:
        target:
            name: "cpu.delta"
        growth_only: True

gnocchi (default)¶

When the gnocchi publisher is enabled, measurement and resource information is pushed to gnocchi for time-series optimized storage. Gnocchi must be registered in the Identity service as Ceilometer discovers the exact path via the Identity service.

More details on how to enable and configure gnocchi can be found on its official documentation page.

panko¶

Event data in Ceilometer can be stored in panko which provides an HTTP REST interface to query system events in OpenStack. To push data to panko, set the publisher to direct://?dispatcher=panko. Beginning in panko’s Pike release, the publisher can be set as panko://

notifier¶

The notifier publisher can be specified in the form of notifier://?option1=value1&option2=value2. It emits data over AMQP using oslo.messaging. Any consumer can then subscribe to the published topic for additional processing.

The following customization options are available:

per_meter_topic

The value of this parameter is 1. It is used for publishing the samples on additional metering_topic.sample_name topic queue besides the default metering_topic queue.

policy

Used for configuring the behavior for the case, when the publisher fails to send the samples, where the possible predefined values are:

default

Used for waiting and blocking until the samples have been sent.

drop

Used for dropping the samples which are failed to be sent.

queue

Used for creating an in-memory queue and retrying to send the samples on the queue in the next samples publishing period (the queue length can be configured with max_queue_length, where 1024 is the default value).

topic

The topic name of the queue to publish to. Setting this will override the default topic defined by metering_topic and event_topic options. This option can be used to support multiple consumers.

udp¶

This publisher can be specified in the form of udp://<host>:<port>/. It emits metering data over UDP.

file¶

The file publisher can be specified in the form of file://path?option1=value1&option2=value2. This publisher records metering data into a file.

The following options are available for the file publisher:

max_bytes

When this option is greater than zero, it will cause a rollover. When the specified size is about to be exceeded, the file is closed and a new file is silently opened for output. If its value is zero, rollover never occurs.

backup_count

If this value is non-zero, an extension will be appended to the filename of the old log, as ‘.1’, ‘.2’, and so forth until the specified value is reached. The file that is written and contains the newest data is always the one that is specified without any extensions.

http¶

The Telemetry service supports sending samples to an external HTTP target. The samples are sent without any modification. To set this option as the notification agents’ target, set http:// as a publisher endpoint in the pipeline definition files. The HTTP target should be set along with the publisher declaration. For example, additional configuration options can be passed in: http://localhost:80/?option1=value1&option2=value2

The following options are availble:

timeout

The number of seconds before HTTP request times out.

max_retries

The number of times to retry a request before failing.

batch

If false, the publisher will send each sample and event individually, whether or not the notification agent is configured to process in batches.

poolsize

The maximum number of open connections the publisher will maintain. Increasing value may improve performance but will also increase memory and socket consumption requirements.

The default publisher is gnocchi, without any additional options specified. A sample publishers section in the /etc/ceilometer/pipeline.yaml looks like the following:

publishers:
    - gnocchi://
    - panko://
    - udp://10.0.0.2:1234
    - notifier://?policy=drop&max_queue_length=512&topic=custom_target

direct¶

This publisher can be specified in the form of direct://?dispatcher=http. The dispatcher’s options include: database, file, http, and gnocchi. It emits data in the configured dispatcher directly, default configuration (the form is direct://) is database dispatcher. In the Mitaka release, this method can only emit data to the database dispatcher, and the form is direct://.

kafka¶

The kafka publisher can be specified in the form of: kafka://kafka_broker_ip: kafka_broker_port?topic=kafka_topic &option1=value1.

This publisher sends metering data to a kafka broker. The kafka publisher offers similar options as notifier publisher.

database¶

When the database dispatcher is configured as a data store, you have the option to set a time_to_live option (ttl) for samples. By default the ttl value for samples is set to -1, which means that they are kept in the database forever.

The time to live value is specified in seconds. Each sample has a time stamp, and the ttl value indicates that a sample will be deleted from the database when the number of seconds has elapsed since that sample reading was stamped. For example, if the time to live is set to 600, all samples older than 600 seconds will be purged from the database.

Certain databases support native TTL expiration. In cases where this is not possible, a command-line script, which you can use for this purpose is ceilometer-expirer. You can run it in a cron job, which helps to keep your database in a consistent state.

The level of support differs in case of the configured back end: