Build a new optimization strategy

Watcher Decision Engine has an external strategy plugin interface which gives anyone the ability to integrate an external strategy in order to make use of placement algorithms.

This section gives some guidelines on how to implement and integrate custom strategies with Watcher. If you wish to create a third-party package for your plugin, you can refer to our documentation for third-party package creation.

Pre-requisites

Before using any strategy, you should make sure you have your Telemetry service configured so that it would provide you all the metrics you need to be able to use your strategy.

Create a new strategy plugin

In order to create a new strategy, you have to:

• Extend the UnclassifiedStrategy class

• Implement its get_name() class method to return the unique ID of the new strategy you want to create. This unique ID should be the same as the name of the entry point we will declare later on.

• Implement its get_display_name() class method to return the translated display name of the strategy you want to create. Note: Do not use a variable to return the translated string so it can be automatically collected by the translation tool.

• Implement its get_translatable_display_name() class method to return the translation key (actually the English display name) of your new strategy. The value return should be the same as the string translated in get_display_name().

• Implement its execute() method to return the solution you computed within your strategy.

Here is an example showing how you can write a plugin called NewStrategy:

# filepath: thirdparty/new.py
# import path: thirdparty.new
import abc
from watcher._i18n import _
from watcher.decision_engine.strategy.strategies import base


class NewStrategy(base.UnclassifiedStrategy):

    def __init__(self, osc=None):
        super(NewStrategy, self).__init__(osc)

    def execute(self, original_model):
        self.solution.add_action(action_type="nop",
                                 input_parameters=parameters)
        # Do some more stuff here ...
        return self.solution

    @classmethod
    def get_name(cls):
        return "new_strategy"

    @classmethod
    def get_display_name(cls):
        return _("New strategy")

    @classmethod
    def get_translatable_display_name(cls):
        return "New strategy"

As you can see in the above example, the execute() method returns a BaseSolution instance as required. This solution is what wraps the abstract set of actions the strategy recommends to you. This solution is then processed by a planner to produce an action plan which contains the sequenced flow of actions to be executed by the Watcher Applier. This solution also contains the various efficacy indicators alongside its computed global efficacy.

Please note that your strategy class will expect to find the same constructor signature as BaseStrategy to instantiate you strategy. Therefore, you should ensure that your __init__ signature is identical to the BaseStrategy one.

Strategy efficacy

As stated before, the NewStrategy class extends a class called UnclassifiedStrategy. This class actually implements a set of abstract methods which are defined within the BaseStrategy parent class.

One thing this UnclassifiedStrategy class defines is that our NewStrategy achieves the unclassified goal. This goal is a peculiar one as it does not contain any indicator nor does it calculate a global efficacy. This proves itself to be quite useful during the development of a new strategy for which the goal has yet to be defined or in case a new goal has yet to be implemented.

Define Strategy Parameters

For each new added strategy, you can add parameters spec so that an operator can input strategy parameters when creating an audit to control the execute() behavior of strategy. This is useful to define some threshold for your strategy, and tune them at runtime.

To define parameters, just implements get_schema() to return parameters spec with jsonschema format. It is strongly encouraged that provide default value for each parameter, or else reference fails if operator specify no parameters.

Here is an example showing how you can define 2 parameters for DummyStrategy:

class DummyStrategy(base.DummyBaseStrategy):

    @classmethod
    def get_schema(cls):
        return {
            "properties": {
                "para1": {
                    "description": "number parameter example",
                    "type": "number",
                    "default": 3.2,
                    "minimum": 1.0,
                    "maximum": 10.2,
                },
                "para2": {
                    "description": "string parameter example",
                    "type": "string",
                    "default": "hello",
                },
            },
        }

You can reference parameters in execute():

class DummyStrategy(base.DummyBaseStrategy):

    def execute(self):
        para1 = self.input_parameters.para1
        para2 = self.input_parameters.para2

        if para1 > 5:
            ...

Operator can specify parameters with following commands:

$ watcher audit create -a <your_audit_template> -p para1=6.0 -p para2=hi

Pls. check user-guide for details.

Abstract Plugin Class

Here below is the abstract BaseStrategy class:

class watcher.decision_engine.strategy.strategies.base.BaseStrategy(config, osc=None)

A base class for all the strategies

A Strategy is an algorithm implementation which is able to find a Solution for a given Goal.

DATASOURCE_METRICS = []

Contains all metrics the strategy requires from a datasource to properly execute

__init__(config, osc=None)

Constructor: the signature should be identical within the subclasses

Parameters:

• config (Struct) – Configuration related to this plugin

• osc (OpenStackClients instance) – An OpenStackClients instance

property baremetal_model

Cluster data model

Returns:

Cluster data model the strategy is executed on

Rtype model:

ModelRoot instance

property compute_model

Cluster data model

Returns:

Cluster data model the strategy is executed on

Rtype model:

ModelRoot instance

abstract do_execute(audit=None)

Strategy execution phase

Parameters:

audit (Audit instance) – An Audit instance

This phase is where you should put the main logic of your strategy.

execute(audit=None)

Execute a strategy

Parameters:

audit (Audit instance) – An Audit instance

Returns:

A computed solution (via a placement algorithm)

Return type:

BaseSolution instance

classmethod get_config_opts()

Defines the configuration options to be associated to this loadable

Returns:

A list of configuration options relative to this Loadable

Return type:

list of oslo_config.cfg.Opt instances

abstract classmethod get_display_name()

The goal display name for the strategy

classmethod get_goal()

The goal the strategy achieves

abstract classmethod get_goal_name()

The goal name the strategy achieves

abstract classmethod get_name()

The name of the strategy

classmethod get_schema()

Defines a Schema that the input parameters shall comply to

Returns:

A jsonschema format (mandatory default setting)

Return type:

dict

abstract classmethod get_translatable_display_name()

The translatable msgid of the strategy

abstract post_execute()

Post-execution phase

This can be used to compute the global efficacy

abstract pre_execute()

Pre-execution phase

This can be used to fetch some pre-requisites or data.

property storage_model

Cluster data model

Returns:

Cluster data model the strategy is executed on

Rtype model:

ModelRoot instance

Add a new entry point

In order for the Watcher Decision Engine to load your new strategy, the strategy must be registered as a named entry point under the watcher_strategies entry point of your setup.py file. If you are using pbr, this entry point should be placed in your setup.cfg file.

The name you give to your entry point has to be unique and should be the same as the value returned by the get_name() class method of your strategy.

Here below is how you would proceed to register NewStrategy using pbr:

[entry_points]
watcher_strategies =
    new_strategy = thirdparty.new:NewStrategy

To get a better understanding on how to implement a more advanced strategy, have a look at the BasicConsolidation class.

Using strategy plugins

The Watcher Decision Engine service will automatically discover any installed plugins when it is restarted. If a Python package containing a custom plugin is installed within the same environment as Watcher, Watcher will automatically make that plugin available for use.

At this point, Watcher will scan and register inside the Watcher Database all the strategies (alongside the goals they should satisfy) you implemented upon restarting the Watcher Decision Engine.

You should take care when installing strategy plugins. By their very nature, there are no guarantees that utilizing them as is will be supported, as they may require a set of metrics which is not yet available within the Telemetry service. In such a case, please do make sure that you first check/configure the latter so your new strategy can be fully functional.

Querying metrics

A large set of metrics, generated by OpenStack modules, can be used in your strategy implementation. To collect these metrics, Watcher provides a DataSourceManager for two data sources which are Ceilometer (with Gnocchi as API) and Monasca. If you wish to query metrics from a different data source, you can implement your own and use it via DataSourceManager from within your new strategy. Indeed, strategies in Watcher have the cluster data models decoupled from the data sources which means that you may keep the former while changing the latter. The recommended way for you to support a new data source is to implement a new helper that would encapsulate within separate methods the queries you need to perform. To then use it, you would just have to add it to appropriate watcher_strategies.* section in config file.

If you want to use Ceilometer but with your own metrics database backend, please refer to the Ceilometer developer guide. The list of the available Ceilometer backends is located here. The Ceilosca project is a good example of how to create your own pluggable backend. Moreover, if your strategy requires new metrics not covered by Ceilometer, you can add them through a Ceilometer plugin.

Read usage metrics using the Watcher Datasource Helper

The following code snippet shows how datasource_backend is defined:

from watcher.datasource import manager as ds_manager

@property
def datasource_backend(self):
    if not self._datasource_backend:

        # Load the global preferred datasources order but override it
        # if the strategy has a specific datasources config
        datasources = CONF.watcher_datasources
        if self.config.datasources:
            datasources = self.config

        self._datasource_backend = ds_manager.DataSourceManager(
            config=datasources,
            osc=self.osc
        ).get_backend(self.DATASOURCE_METRICS)
    return self._datasource_backend

Using that you can now query the values for that specific metric:

avg_meter = self.datasource_backend.statistic_aggregation(
    instance.uuid, 'instance_cpu_usage', self.periods['instance'],
    self.granularity,
    aggregation=self.aggregation_method['instance'])