Every proposed change to a charm is run through testing during the review verification process. If you want to contribute a change or fix to a charm, please take time to review the Unit Testing and Functional Testing sections of this document.
OpenStack Charm CI will verify your changes, but please execute at least unit tests locally before submitting patches to reduce load on the OpenStack CI infrastructure.
The OpenStack Charms are compliant with the OpenStack Consistent Testing Interface; take a read on how this works to understand in full.
You can verify the compliance of your code changes using flake8 and the charm proof tool using the pep8 tox environment:
tox -e pep8
Ensure that any non-compliance is corrected prior to raising/updating a review.
Execute the synthetic code checks (unit tests) for a charm using the tox py27 environment:
tox -e py27
Unit tests are stored in the unit_tests folder; when adding features or changing existing code, please ensure that appropriate unit tests are added or updated to cover the changes you are making.
Unit tests are written in Python using standard mocking techniques to isolate the unit tests from the underlying host operating system.
Functional tests for a charm are written using the Amulet test framework and should exercise the target charm with a subset of a full OpenStack deployment to ensure that the charm is able to correctly deploy and configure the service that is encapsulates.
The OpenStack charm helpers provide some Amulet deployment helpers to ease testing of different OpenStack release combinations; typically each charm will test the OpenStack and Ubuntu release combinations currently supported by Ubuntu.
The OpenStack Charms Amulet tests in their current form may be specific to execution within a tenant on an OpenStack cloud, via the Juju OpenStack provider, and that is how the third-party-CI executes them. Future functional test enhancements include the ability run the tests against the Juju OpenStack provider (a cloud) or the Juju LXD provider (all on one machine).
| Full Amulet: | Executes all Amulet gate tests (may take several hours). The full Amulet test set does not run automatically on each proposed change. After the lower-cost lint, unit, charm-single and Amulet-smoke tests have completed, reviewers can conduct code reviews then optionally trigger the full set of Amulet tests (see Rechecking). To manually trigger execution of all Amulet tests on your locally-defined cloud: |
|---|
tox -e func27
| Amulet Smoke: | Executes a subset (generaly one) of the Amulet deployment test sets. The Amulet smoke test set does run automatically on every proposed patchset. To manually trigger execution of the Amulet smoke test on your locally-defined cloud: |
|---|
tox -e func27-smoke
| No-Op: | Builds a Python virtualenv per definitions in tox.ini, which can be useful in test authoring. To manually trigger a build of the virtualenv on your local machine, but execute no tests: |
|---|
tox -e func27-noop
Test methods are called in lexical sort order, as with most test runners. However, each individual test method should be idempotent and expected to pass regardless of run order or Ubuntu:OpenStack combo. When writing or modifying tests, ensure that every individual test is not dependent on another test method.
Some tests may need to download files from the Internet, such as glance images. If a web proxy server is required in the environment, the AMULET_HTTP_PROXY environment variable must be set. This is unrelated to Juju’s http-proxy settings.
See tox.ini to determine specifically which test targets will be executed by each tox target. Amulet tests reside in the tests/ directory for classic charms, and in the src/tests/ directory for layered source charms.
BEFORE issuing a recheck of any kind, please inspect the CI results and log artifacts to understand the failure reason.
Rechecks should only be used in the event of a system failure (not for race conditions or problems introduced by the proposed code changes).
Developers are expected to have executed tests prior to submitting patches.
Tests can be retriggered, or additional tests can be requested, simply by replying on the Gerrit review with one of the recognized magic phrases below.