Senlin testing¶

Writing a Unit Test¶

The os-testr software (see: http://pypi.python.org/pypi/os-testr) is used to find and run tests, parallelize their runs, and record timing/results.

If new dependencies are introduced upon the development of a test, the test-requirements.txt file needs to be updated so that the virtual environment will be able to successfully execute all tests.

The test-requirements.txt file needs to be synchronized with the openstack/global-requirements project. Developers should try avoid introducing additional package dependencies unless forced to.

Running Unit Tests¶

Senlin uses tox for running unit tests, as practiced by many other OpenStack projects:

$ tox

This by default will run unit tests suite with Python 2.7 and PEP8/HACKING style checks. To run only one type of tests you can explicitly provide tox with the test environment to use:

$ tox -e py27 # test suite on python 2.7
$ tox -e pep8 # run full source code checker

To run only a subset of tests, you can provide tox with a regex argument:

$ tox -e py27 -- -r ClusterTest

To use debugger like pdb during test run, you have to run tests directly with other, non-concurrent test runner instead of testr. That also presumes that you have a virtual env with all senlin dependencies installed and configured.

A more convenient way to run specific test is to name the unit test directly, as shown below:

$ python -m testtools.run senlin.tests.unit.db.test_cluster_api

This command, however, is not using dependent packages in a particular virtual environment as the tox command does. It is using the system-wide Python package repository when running the tests.

Writing an API Test Case¶

When new APIs are added or existing APIs are changed, an API test case should be added to the senlin/tests/tempest/api sub-directory, based on the resources impacted by the change.

Each test case should derive from the class senlin.tests.tempest.api.base.BaseSenlinAPITest. Positive test cases should be separated from negative ones. We don’t encourage combining more than one test case into a single method, unless there is an obvious reason.

To improve the readability of the test cases, Senlin has provided a utility module which can be leveraged - senlin/tests/tempest/common/utils.py.

Running API Tests¶

Senlin API tests use fake OpenStack drivers to improve the throughput of test execution. This is because in API tests, we don’t care about the details in how senlin-engine is interacting with other services. We care more about the APIs succeeds in an expected way or fails in a predictable manner.

Although the senlin engine is talking to fake drivers, the test cases still need to communicate to the senlin API service as it would in a real deployment. That means you will have to export your OpenStack credentials before running the tests. For example, you will source the openrc file when using a devstack environment:

$ source $HOME/devstack/openrc

This will ensure you have environment variables such as OS_AUTH_URL, OS_USERNAME properly set and exported. The next step is to enter the tempest directory and run the tests there:

$ cd /opt/stack/tempest
$ ./run_tempest.sh -N -- senlin

To run a single test case, you can specify the test case name. For example:

$ cd /opt/stack/tempest
$ ./run_tempest.sh -N -- \
  senlin.tests.tempest.api.clusters.test_cluster_create

If you prefer running API tests in a virtual environment, you can simply use the following command:

$ cd /opt/stack/senlin
$ tox -e api

Writing Functional Tests¶

There are current a limited collection of functional test cases which can be found under senlin/tests/tempest/functional/ subdirectory. In future, we may add more test cases when needed. The above subdirectory will remain the home of newly added functional tests.

When writing functional tests, it is highly desirable that each test case is designed for a specific use case or story line.

Running Functional Tests¶

Similar to API tests, you will need to export your OpenStack credentials before running any functional tests.

The most straight forward way to run functional tests is to use the virtual environment defined in the tox.ini file, that is:

$ cd /opt/stack/senlin
$ tox -e functional

If you prefer running a particular functional test case, you can do the following as well:

$ cd /opt/stack/senlin
$ python -m testtools.run senlin.tests.tempest.functional.test_cluster_basic

Writing Integration Tests¶

Integration tests are designed to be run at Gerrit gate to ensure that changes to senlin code won’t break its interactions with other (backend) services. Since OpenStack gate infrastructure is a shared resource pool for all OpenStack projects, we are supposed to be very careful when adding new test cases. The test cases added are supposed to focus more on the interaction between senlin and other services than other things.

All integration test cases are to be placed under the subdirectory senlin/tests/tempest/integration. Test cases are expected to be organized into a small number of story lines that can exercise as many interactions between senlin and backend services as possible.

Each “story line” should be organized into a separate class module that inherits from the BaseSenlinIntegrationTest class which can be found at senlin/tests/tempest/integration/base.py file. Each test case should be annotated with a test.attr annotator and an idempotent ID as shown below:

from tempest.lib import decorators
from tempest import test

from senlin.tests.tempest.integration import base


class MyIntegrationTest(base.BaseSenlinIntegrationTest):

  @test.attr(type=['integration'])
  @decorators.idempotent_id('<A UUID for the test case>')
  def test_a_sad_story(self):
    # Test logic goes here
    # ...

Running Integration Tests¶

The integration tests are designed to be executed at Gerrit gate. However, you can still run them locally in your development environment, i.e. a devstack installation.

To run integration tests, you will need to configure tempest accounts by editing the /etc/tempest/accounts.yaml file. For each entry of the tempest account, you will need to provide values for username, tenant_name, password at least. For example:

- username: 'demo'
  tenant_name: 'demo'
  password: 'secretee'

After this is configured, you can run a specific test case using the following command:

$ cd /opt/stack/senlin
$ python -m testtools.run \
    senlin.tests.tempest.integration.test_nova_server_cluster

Writing Stress Test Cases¶

<TBD>

Running Stress Tests¶

<TBD>