TripleO CI jobs primer

This primer aims to demonstrate where the Triple ci jobs are defined and illustrate the difference between the check and gate queues and how jobs are executed in them. Which queue a job is executed in also affects whether the job is defined as voting or not. Generally:

  • new jobs are run in check and are non voting

  • once a job is voting in check, it needs to be added to gate too.

  • once a job is voting in check and gate you should add it to the promotion jobs so that tripleo promotions (i.e. from tripleo-testing to current-tripleo) will depend on successful execution of that job.

Once a job becomes voting it must be added to the gate queue too. If it isn’t then we may end up with a situation where something passes the voting check job and merges without being run in the gate queue. It could be that for some reason it would have failed in the gate and thus not have merged. A common occurrence is the check jobs run on a particular submission and pass on one day but then not actually merge (and so run in the gate) until much later perhaps even after some days.In the meantime some unrelated change merges in another project which would cause the job to fail in the gate, but since we’re not running it there the code submission merges. This then means that the job is broken in subsequent check runs.

Non tripleo-projects are not gated in tripleo. The promotion jobs represent the point at which we take the latest built tripleo packages and the latest built non-tripleo projects packages (like nova, neutron etc) and test these together. For more information about promotions refer to Promotion Stages

Where do tripleo-ci jobs live

Note

If you ever need to search for a particular job to see which file it is defined in or which tripleo project repos it is running for you can search by name in the openstack-codesearch (e.g. that is a search for the tripleo-ci-centos-7-scenario003-standalone job).

Note

If you ever want to see the status for a particular job with respect to how often it is failing or passing, you can check the zuul_builds status and search by job name (again the linked example is for scenario003-standalone).

The tripleo ci jobs live in the tripleo-ci repo and specifically in various files defined under the zuul.d directory. As an example we can examine one of the scenario-standalone-jobs:

- job:
  name: tripleo-ci-centos-7-scenario001-standalone
  voting: true
  parent: tripleo-ci-base-standalone
  nodeset: single-centos-7-node
  branches: ^(?!stable/(newton|ocata|pike|queens|rocky)).*$
  vars:
    featureset: '052'
    standalone_ceph: true
    featureset_override:
      standalone_container_cli: docker
      standalone_environment_files:
        - 'ci/environments/scenario001-standalone.yaml'
        - 'environments/low-memory-usage.yaml'
      tempest_plugins:
        - python-telemetry-tests-tempest
        - python-heat-tests-tempest
      test_white_regex: ''
      tempest_workers: 1
      tempest_extra_config: {'telemetry.alarm_granularity': '60'}
      tempest_whitelist:
        - 'tempest.api.identity.v3'
        - 'tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern'
        - 'telemetry_tempest_plugin.scenario.test_telemetry_integration.TestTelemetryIntegration'

As you can see the job definition consists of the unique job name followed by the rest of the zuul variables, including whether the job is voting and which node layout (nodeset) should be used for that job. The unique job name is then used in the zuul layout (discussed in the next section) to determine if the job is run in check or gate or both. Since the job shown above is set as voting we can expect it to be defined in both gate and check.

Zuul queues - gate vs check

As with all OpenStack projects there are two zuul queues to which jobs are scheduled - the check jobs which are run each time a change is submitted and then the gate jobs which are run before a change is merged. There is also an experimental queue but that is invoked manually.

Which queue a given job is run in is determined by the zuul layout file for the given project - e.g. here is tripleo-heat-templates-zuul-layout. The layout file has the following general format:

- project:
   templates:
   .. list of templates
   check:
     jobs:
       .. list of job names and any options for each
   gate:
     queue: tripleo
     jobs:
       .. list of job names and any options for each

The templates: section in the outline above is significant because the layout can also be defined in one of the included templates. For example the scenario-standalone-layout defines the check/gate layout for the tripleo-standalone-scenarios-full template which is then included by the projects that want the jobs defined in that template to execute in the manner it specifies.

Where do tripleo promotion jobs live

Note

If you even need to find the definition for a particular promotion job you can search for it by name using the rdo-codesearch.

The tripleo promotions jobs are not defined in the tripleo-ci but instead live in the rdo-jobs repository. For more information about the promotion pipeline in TripleO refer to the Promotion Stages

Similar to the tripleo-ci jobs, they are defined in various files under the rdo-jobs-zuul.d directory and the job definitions look very similar to the tripleo-ci ones - for example the periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master:

- job:
  name: periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset010-master
  parent: tripleo-ci-base-multinode-periodic
  vars:
    nodes: 1ctlr
    featureset: '010'
    release: master

If you even need to find the definition for a particular promotion job you can search for it by name using the rdo-codesearch.

Contacting CI team

When in need you can contact the TripleO CI team members on one of the two irc channels on OFTC #tripleo by mentioning @oooq keyword in your message as team members get notified about such messages. It is good to remember that those nicknames with |ruck and |rover suffix are on duty to look for CI status.