Verifiers

What is it?

Verifier Plugin is a compatibility layer between Rally and the specific tool (such as Tempest) which runs tests. It implements features like installation, configuration, upgrades, running, etc in terms of the tool. It is a driver in other words. It is a pluggable entity, which means that you can easily add support for whatever tool you want (see HowTo add support for new tool page for more information). Even more, you can deliver such plugin separately from Rally itself, but we firmly recommend to push a change to Rally upstream (see Contribute to Rally guide), so Rally core-team will able to review it and help to improve.

Verifier is an instance of the Verifier Plugin. It is an installed tool. For example, "Tempest" is a set of functional tests, it is Verifier Plugin (we have a plugin for it). Installed Tempest 12.0 from https://github.com/openstack/tempest in a virtual environment is the verifier.

Verifier is not aligned to any particular deployment like it was in the past, you can use one verifier for testing unlimited number of deployments (each deployment will have separate configuration files for the tool).

Verifier & Verifier Plugin are the main entities which Verification component operates with. Another one is the verifications results.

Verifier statuses

All verifiers can be in next statuses:

  • init - Initial state. It appears while you call rally verify create-verifier command and installation step is not yet started.
  • installing - Installation of the verifier is not a quick task. It is about cloning tool, checking packages or installing virtual environments with all required packages. This state indicates that this step is in the process.
  • installed - It should be one of your favourite states. It means that everything is ok and you can start verifying your cloud.
  • updating - This state identifies the process of updating verifier (version, source, packages, etc.).
  • extending - The process of extending a verifier by its plugins.
  • failed - Something went wrong while installation.

Verification statuses

  • init - Initial state. It appears instantly after calling rally verify start command before the actual run of verifier's tool.
  • running - Identifies the process of execution tool.
  • finished- Verification is finished without errors and failures.
  • failed - Verification is finished, but there are some failed tests.
  • crashed - Unexpected error had happened while running verification.

Known verifier types

Out of the box

You can execute command rally verify list-plugins locally to check available verifiers in your environment.

Cut down from Global Plugins Reference page:

tempest

Tempest verifier.

Description:

Quote from official documentation:

This is a set of integration tests to be run against a live OpenStack cluster. Tempest has batteries of tests for OpenStack API validation, Scenarios, and other specific tests useful in validating an OpenStack deployment.

Rally supports features listed below:

  • cloning Tempest: repository and version can be specified
  • installation: system-wide with checking existence of required packages or in virtual environment
  • configuration: options are discovered via OpenStack API, but you can override them if you need
  • running: pre-creating all required resources(i.e images, tenants, etc), prepare arguments, launching Tempest, live-progress output
  • results: all verifications are stored in db, you can built reports, compare verification at whatever you want time.

Appeared in Rally 0.8.0 (actually, it appeared long time ago with first revision of Verification Component, but 0.8.0 is mentioned since it is first release after Verification Component redesign)

Running arguments:
  • concurrency: Number of processes to be used for launching tests. In case of 0 value, number of processes will be equal to number of CPU cores.
  • load_list: a list of tests to launch.
  • pattern: a regular expression of tests to launch.
  • set: Name of predefined set of tests. Known names: full, smoke, baremetal, clustering, compute, database, data_processing, identity, image, messaging, network, object_storage, orchestration, telemetry, volume, scenario
  • skip_list: a list of tests to skip (actually, it is a dict where keys are names of tests, values are reasons).
  • xfail_list: a list of tests that are expected to fail (actually, it is a dict where keys are names of tests, values are reasons).
Installation arguments:
  • system_wide: Whether or not to use the system-wide environment for verifier instead of a virtual environment. Defaults to False.
  • source: Path or URL to the repo to clone verifier from. Defaults to https://git.openstack.org/openstack/tempest
  • version: Branch, tag or commit ID to checkout before verifier installation. Defaults to 'master'.

Platform: openstack

Module: rally.plugins.openstack.verification.tempest.manager

Third-party

Nothing here yet.