Tempest

The Designate team maintains a set of tempest tests to exercise the Designate service and APIs.

Intro and References

Quick Start

To run all tests from this plugin, install the plugin into your environment and from the tempest repo, run:

$ tox -e all-plugin -- designate

To run a single test case, run with the test case name, for example:

$ tox -e all-plugin -- designate_tempest_plugin.tests.api.v2.test_zones.ZonesAdminTest.test_get_other_tenant_zone

To run all tempest tests including this plugin, run:

$ tox -e all-plugin

Writing new tests

Writing new tests is easy, and we encourage contributors to write tests for any new or changed functionality. Most of the patterns you will find in the Designate tests will look familiar if you have contributed to tempest, so rather than re-type all their docs here, please have a read of the Tempest Docs.

Test Clients

In Tempest tests, it is forbidden to use a services python bindings or client, as doing so would allow API changes to go unnoticed when the server and client are updated. As such, each service is expected to have a minimal in-tree client. Designate’s client can be found in:

$ tree -P "*_client.py" designate_tempest_plugin/services/dns/

designate_tempest_plugin/services/dns/
├── json
│   └── versions_client.py
└── v2
    └── json
        ├── recordsets_client.py
        └── zones_client.py

An example client, in this case for a subset of /v2/zones is included below:

class ZonesClient(base.DnsClientV2Base):
"""API V2 Tempest REST client for Designate API"""

@base.handle_errors
def create_zone(self, name=None, email=None, ttl=None, description=None,
                wait_until=False, params=None):
    """Create a zone with the specified parameters.

    :param name: The name of the zone.
        Default: Random Value
    :param email: The email for the zone.
        Default: Random Value
    :param ttl: The ttl for the zone.
        Default: Random Value
    :param description: A description of the zone.
        Default: Random Value
    :param wait_until: Block until the zone reaches the desired status
    :param params: A Python dict that represents the query parameters to
                   include in the request URI.
    :return: A tuple with the server response and the created zone.
    """
    zone = {
        'name': name or dns_data_utils.rand_zone_name(),
        'email': email or dns_data_utils.rand_email(),
        'ttl': ttl or dns_data_utils.rand_ttl(),
        'description': description or data_utils.rand_name('test-zone'),
    }

    resp, body = self._create_request('zones', zone, params=params)

    if wait_until:
        waiters.wait_for_zone_status(self, body['id'], wait_until)

    return resp, body

Some items to note, client methods should be wrapped in the base.handle_errors decorator, which is used to allow for ignoring certain types of errors, in certain cases. Most commonly, this will be ignoring 404’s when cleaning up resources.

Test Cases

Designate’s tests can be found in:

$ tree -P "test_*.py" designate_tempest_plugin/tests/

designate_tempest_plugin/tests/
├── api
│   ├── test_versions.py
│   └── v2
│       ├── test_recordsets.py
│       └── test_zones.py
└── scenario
    └── v2
        ├── test_recordsets.py
        └── test_zones.py

There are two groupings of tests here “api” and “scenario”. API tests should be quick, and simple. Testing as small a surface area of the API as is possible while still getting the job done. Additionally, API tests should avoid waiting for resources to become ACTIVE etc, as this typically pushes test time out significantly, and would only duplicate scenario tests. Scenario tests should cover common real world uses cases. For example, creating a zone, waiting for it to become ACTIVE, adding some records, waiting for ACTIVE, querying the DNS servers themselves, and finally deleting the zone and waiting for it to 404.

An example test, in this case for a subset of /v2/zones functionality is included below:

class ZonesTest(BaseZonesTest):
    @classmethod
    def setup_clients(cls):
        super(ZonesTest, cls).setup_clients()

        cls.client = cls.os.zones_client

    @test.attr(type='smoke')
    @decorators.idempotent_id('fbabd6af-238a-462e-b923-de4d736b90a7')
    def test_create_zone(self):
        LOG.info('Create a zone')
        _, zone = self.client.create_zone()
        self.addCleanup(self.client.delete_zone, zone['id'])

        LOG.info('Ensure we respond with CREATE+PENDING')
        self.assertEqual('CREATE', zone['action'])
        self.assertEqual('PENDING', zone['status'])

        LOG.info('Ensure the fetched response matches the created zone')
        self._assertExpected(zone, body)

Test Cases - Alternative Credentials

Some tests require more than just a “standard” cloud user, e.g. those tests checking admin only functionality. We can ensure both user and admin credentials are available using the class level “credentials” property like so:

class ZonesAdminTest(BaseZonesTest):
    credentials = ['primary', 'admin']

    @classmethod
    def setup_clients(cls):
        super(ZonesAdminTest, cls).setup_clients()

        cls.client = cls.os.zones_client
        cls.adm_client = cls.os_adm.zones_client

    @decorators.idempotent_id('6477f92d-70ba-46eb-bd6c-fc50c405e222')
    def test_get_other_tenant_zone(self):
        LOG.info('Create a zone as a user')
        _, zone = self.client.create_zone()
        self.addCleanup(self.client.delete_zone, zone['id'])

        LOG.info('Fetch the zone as an admin')
        _, body = self.adm_client.show_zone(
            zone['id'], params={'all_tenants': True})

        LOG.info('Ensure the fetched response matches the created zone')
        self._assertExpected(zone, body)

Test Decorators

Several different test decorators are used within the test cases, this attempts to explain their purpose and correct usage.

@decorators.idempotent_id

The idempotent_id decorator allows for tracking of tests even after they have been renamed. The UUID should be randomly generated as the test is first written, e.g. with uuidgen on most linux hosts, and should not be changed when the test is renamed.

Every test should have a unique idempotent_id assigned.

Example:

class ZonesTest(BaseZonesTest):
    @decorators.idempotent_id('fbabd6af-238a-462e-b923-de4d736b90a7')
    def test_create_zone(self):
        pass

@test.attr

The attr decorator is used to set test attributes, this is most commonly used to set the test type. Currently, we use one test type “smoke”, which should be applied to any tests which test the most basic functionality Designate provides, allowing for the core functionality to be tested quickly, without having to run the entire suite. Another type we use is “slow”, which should be applied to tests which take on average 5 seconds or more.

Example:

class ZonesTest(BaseZonesTest):
    @test.attr(type='smoke')
    def test_create_zone(self):
        pass

    @test.attr(type='slow')
    def test_something_else(self):
        pass

@test.services

The services decorator is used to indicate which services are exercised by a given test. The services decorator may only be used on scenario tests, and (for now) should not include “dns” itself. For example, given a scenario test that interactions with Designate’s Reverse DNS APIs, which in turn talk to Neutron, we would use something like the below:

Example:

class ReverseTest(BaseDnsTest):
    @test.services('network')
    def test_reverse_dns_for_fips(self):
        pass