Content Provider Jobs

This section gives an overview and some details about the ‘content provider’ zuul jobs. They are so called because they consist of a parent job that builds containers which are then consumed by any number of child jobs. Thus the parent jobs are the ‘content provider’ for the child jobs.

Why Do We Need Content Providers?

The content provider jobs were added by the Tripleo CI squad during the Victoria development cycle. Prior to this check and gate tripleo-ci jobs running on review.opendev.org code submissions were pulling the promoted ‘current-tripleo’ containers from docker.io.

Having all jobs pull directly from a remote registry obviously puts a strain on resources; consider multiple jobs per code submission with tens of container pulls for each. We have over time been affected by a number of issues related to the container pulls (such as timeouts) that would cause jobs to fail and block the gates. Furthermore, docker has recently announced that requests will be rate limited to one or two hundred pull requests per six hours (without and with authentication respectively) on the free plan effective 01 November 2020.

In anticipation of this the TripleO CI squad has moved all jobs to the new content provider architecture.

The Content Provider

The main task executed by the content provider job is to build the containers needed to deploy TripleO. This is achieved with a collection of ansible plays defined in the multinode-standalone-pre.yml tripleo-quickstart-extras playbook.

Once built, the content provider then needs to make those containers available for use by the child jobs. The build-container role itself as invoked in multinode-standalone-pre.yml ensures containers are pushed to the a local registry on the content provider node. However the child job will need to know the IP address on which they can reach that registry.

To achieve this we use the zuul_return module that allows for a parent job to return data for consumption within child jobs. We set the required zuul_return data in the run-provider.yml playbook:

- name: Set registry IP address
  zuul_return:
    data:
      zuul:
        pause: true
      registry_ip_address: "{{ hostvars[groups.all[0]].ansible_host }}"
      provider_dlrn_hash: "{{ dlrn_hash|default('') }}"
      provider_dlrn_hash_tag: "{{ dlrn_hash_tag|default('') }}"
      provider_job_branch: "{{ provider_job_branch }}"
      registry_ip_address_branch: "{{ registry_ip_address_branch }}"
      provider_dlrn_hash_branch: "{{ provider_dlrn_hash_branch }}"
  tags:
    - skip_ansible_lint

Child jobs retrieve the IP address for the content provider container registry via the registry_ip_address_branch dictionary. This contains a mapping between the release (master, victoria, ussuri etc) and the IP address of the content provider container registry with images for that release. For example:

registry_ip_address_branch:
  master: 38.145.33.72

Most jobs will only ever have one release in this dictionary but upgrade jobs will require two (more on that later). Note that besides setting the zuul_return data the task above sets the zuul pause: true. As the name suggests, this allows the parent content provider job to be paused until all children have executed.

Given all the above, it should be of little surprise ;) that the content provider zuul job definition is as follows (at time of writing):

- job:
    name: tripleo-ci-centos-8-content-provider
    parent: tripleo-ci-base-standalone-centos-8
    branches: ^(?!stable/(newton|ocata|pike|queens|rocky|stein)).*$
    run:
      - playbooks/tripleo-ci/run-v3.yaml
      - playbooks/tripleo-ci/run-provider.yml
    vars:
      featureset: '052'
      provider_job: true
      build_container_images: true
      ib_create_web_repo: true
      playbooks:
        - quickstart.yml
        - multinode-standalone-pre.yml

It uses the same featureset as the standalone job. Notice the multinode-standalone-pre.yml passed to tripleo-quickstart for execution. The run-provider.yml playbook is executed as the last of the zuul run plays.

Finally, one other important task performed by the content provider job is to build any dependent changes (i.e. depends-on in the code submission). This is done with build-test-packages invoked in the multinode-standalone-pre.yml. We ensure that the built repo is available to child jobs by setting the ib_create_web_repo variable when built-test-packages is invoked by a provider job. This makes the repo availabe via a HTTP server on the content provider node that consumers then retrieve as described below.

The Content Consumers

The child jobs or content consumers must use the container registry available from the content provider. To do this we set the docker_registry_host variable using the job.registry_ip_address_branch zuul_data returned from the parent content provider.

Any dependent changes built by build-test-packages are installed into consumer jobs using the install-built-repo playbook. This has been added into the appropriate base job definitions as a pre-run: play.

Finally, in order to make a given zuul job a consumer job we must set the content provider as dependency and pass the relevant variables. For example in order to run tripleo-ci-centos-8-scenario001-standalone as a consumer job:

- tripleo-ci-centos-8-content-provider
- tripleo-ci-centos-8-scenario001-standalone:
    files: *scen1_files
    vars: &consumer_vars
      consumer_job: true
      build_container_images: false
      tags:
        - standalone
    dependencies:
      - tripleo-ci-centos-8-content-provider

Upgrade Jobs

Upgrade jobs are a special case because they require content from more than one release. For instance tripleo-ci-centos-8-standalone-upgrade-ussuri will deploy train containers and then upgrade to ussuri containers.

To achieve this we use two content provider jobs as dependencies for the upgrade jobs that require them (not all do):

- tripleo-ci-centos-8-standalone-upgrade:
    vars: *consumer_vars
    dependencies:
      - tripleo-ci-centos-8-content-provider
      - tripleo-ci-centos-8-content-provider-ussuri

As shown earlier in this document the registry_ip_address_branch dictionary maps release to the appropriate registry. This is set by each of the two parent jobs and once both have executed the dictionary will contain more than one entry. For example:

registry_ip_address_branch:
  master: 213.32.75.192
  ussuri: 158.69.75.154

The consumer upgrade jobs then use the appropriate registry for the deployment or upgrade part of the test.