.. _configuration:

.. default-domain:: zuul

Configuration
=============

Nodepool reads its configuration from ``/etc/nodepool/nodepool.yaml``
by default.  The configuration file follows the standard YAML syntax
with a number of sections defined with top level keys.  For example, a
full configuration file may have the ``diskimages``, ``labels``,
and ``providers`` sections::

  diskimages:
    ...
  labels:
    ...
  providers:
    ...

The following sections are available.  All are required unless
otherwise indicated.

.. attr-overview::
   :maxdepth: 1

Options
-------

.. attr:: webapp

   Define the webapp endpoint port and listen address

   .. attr:: port
      :default: 8005
      :type: int

      The port to provide basic status information

   .. attr:: listen_address
      :default: 0.0.0.0

      Listen address for web app

.. attr:: elements-dir
   :example: /path/to/elements/dir
   :type: str

   If an image is configured to use diskimage-builder and glance to locally
   create and upload images, then a collection of diskimage-builder elements
   must be present. The ``elements-dir`` parameter indicates a directory
   that holds one or more elements.

.. attr:: images-dir
   :example: /path/to/images/dir
   :type: str

   When we generate images using diskimage-builder they need to be
   written to somewhere. The ``images-dir`` parameter is the place to
   write them.

   .. note:: The builder daemon creates a UUID to uniquely identify
          itself and to mark image builds in ZooKeeper that it
          owns. This file will be named ``builder_id.txt`` and will
          live in the directory named by the :attr:`images-dir`
          option. If this file does not exist, it will be created on
          builder startup and a UUID will be created automatically.


.. attr:: build-log-dir
   :example: /path/to/log/dir
   :type: str

   The builder will store build logs in this directory.  It will create
   one file for each build, named `<image>-<build-id>.log`; for example,
   `fedora-0000000004.log`.  It defaults to ``/var/log/nodepool/builds``.

.. attr:: build-log-retention
   :default: 7
   :type: int

   At the start of each build, the builder will remove old build logs if
   they exceed this value.  This option specifies how many will be
   kept (usually you will see one more, as deletion happens before
   starting a new build).  By default, the last 7 old build logs are
   kept.

.. attr:: zookeeper-servers
   :type: list
   :required:

   Lists the ZooKeeper servers uses for coordinating information between
   nodepool workers.

   .. code-block:: yaml

      zookeeper-servers:
        - host: zk1.example.com
          port: 2181
          chroot: /nodepool

   Each entry is a dictionary with the following keys

   .. attr:: host
      :type: str
      :example: zk1.example.com
      :required:

      A zookeeper host

   .. attr:: port
      :default: 2181
      :type: int

      Port to talk to zookeeper

   .. attr:: chroot
      :type: int
      :example: /nodepool

      The ``chroot`` key, used for interpreting ZooKeeper paths
      relative to the supplied root path, is also optional and has no
      default.

.. attr:: labels
   :type: list

   Defines the types of nodes that should be created.  Jobs should be
   written to run on nodes of a certain label. Example

   .. code-block:: yaml

      labels:
        - name: my-precise
          max-ready-age: 3600
          min-ready: 2
        - name: multi-precise
          min-ready: 2

   Each entry is a dictionary with the following keys

   .. attr:: name
      :type: string
      :required:

      Unique name used to tie jobs to those instances.

   .. attr:: max-ready-age
      :type: int
      :default: 0

       Maximum number of seconds the node shall be in ready state. If
       this is exceeded the node will be deleted. A value of 0 disables
       this.

   .. attr:: min-ready
      :type: int
      :default: 0

      Minimum number of instances that should be in a ready
      state. Nodepool always creates more nodes as necessary in response
      to demand, but setting ``min-ready`` can speed processing by
      attempting to keep nodes on-hand and ready for immedate use.
      ``min-ready`` is best-effort based on available capacity and is
      not a guaranteed allocation.  The default of 0 means that nodepool
      will only create nodes of this label when there is demand.  Set
      to -1 to have the label considered disabled, so that no nodes will
      be created at all.

.. attr:: max-hold-age
   :type: int
   :default: 0

   Maximum number of seconds a node shall be in "hold" state. If this
   is exceeded the node will be deleted. A value of 0 disables this.

   This setting is applied to all nodes, regardless of label or
   provider.

.. attr:: diskimages
   :type: list

   This section lists the images to be built using
   diskimage-builder. The name of the diskimage is mapped to the
   :attr:`providers.[openstack].diskimages` section of the provider,
   to determine which providers should received uploads of each image.
   The diskimage will be built in every format required by the
   providers with which it is associated.  Because Nodepool needs to
   know which formats to build, if the diskimage will only be built if
   it appears in at least one provider.

   To remove a diskimage from the system entirely, remove all
   associated entries in :attr:`providers.[openstack].diskimages` and
   remove its entry from `diskimages`.  All uploads will be deleted as
   well as the files on disk.

   .. code-block:: yaml

      diskimages:
        - name: ubuntu-precise
          pause: False
          rebuild-age: 86400
          elements:
            - ubuntu-minimal
            - vm
            - simple-init
            - openstack-repos
            - nodepool-base
            - cache-devstack
            - cache-bindep
            - growroot
            - infra-package-needs
          release: precise
          username: zuul
          env-vars:
            TMPDIR: /opt/dib_tmp
            DIB_CHECKSUM: '1'
            DIB_IMAGE_CACHE: /opt/dib_cache
            DIB_APT_LOCAL_CACHE: '0'
            DIB_DISABLE_APT_CLEANUP: '1'
            FS_TYPE: ext3
        - name: ubuntu-xenial
          pause: True
          rebuild-age: 86400
          formats:
            - raw
            - tar
          elements:
            - ubuntu-minimal
            - vm
            - simple-init
            - openstack-repos
            - nodepool-base
            - cache-devstack
            - cache-bindep
            - growroot
            - infra-package-needs
          release: precise
          username: ubuntu
          env-vars:
            TMPDIR: /opt/dib_tmp
            DIB_CHECKSUM: '1'
            DIB_IMAGE_CACHE: /opt/dib_cache
            DIB_APT_LOCAL_CACHE: '0'
            DIB_DISABLE_APT_CLEANUP: '1'
            FS_TYPE: ext3

   Each entry is a dictionary with the following keys

   .. attr:: name
      :type: string
      :required:

      Identifier to reference the disk image in
      :attr:`providers.[openstack].diskimages` and :attr:`labels`.

   .. attr:: formats
      :type: list

      The list of formats to build is normally automatically created
      based on the needs of the providers to which the image is
      uploaded.  To build images even when no providers are configured
      or to build additional formats which you know you may need in the
      future, list those formats here.

   .. attr:: rebuild-age
      :type: int
      :default: 86400

      If the current diskimage is older than this value (in seconds),
      then nodepool will attempt to rebuild it.  Defaults to 86400 (24
      hours).

   .. attr:: release
      :type: string

      Specifies the distro to be used as a base image to build the image using
      diskimage-builder.

   .. attr:: build-timeout
      :type: int

      How long (in seconds) to wait for the diskimage build before giving up.
      The default is 8 hours.

   .. attr:: elements
      :type: list

      Enumerates all the elements that will be included when building the image,
      and will point to the :attr:`elements-dir` path referenced in the same
      config file.

   .. attr:: env-vars
      :type: dict

      Arbitrary environment variables that will be available in the spawned
      diskimage-builder child process.

   .. attr:: pause
      :type: bool
      :default: False

      When set to True, ``nodepool-builder`` will not build the diskimage.

   .. attr:: username
      :type: string
      :default: zuul

      The username that a consumer should use when connecting to the
      node.

   .. attr:: python-path
      :type: string
      :default: /usr/bin/python2

      The path of the default python interpreter.


.. attr:: providers
   :type: list

   Lists the providers Nodepool should use. Each provider is associated to
   a driver listed below.

   Each entry is a dictionary with the following keys

   .. attr:: name
      :type: string
      :required:

      Name of the provider

   .. attr:: max-concurrency
      :type: int
      :default: 0

      Maximum number of node requests that this provider is allowed to
      handle concurrently. The default, if not specified, is to have
      no maximum. Since each node request is handled by a separate
      thread, this can be useful for limiting the number of threads
      used by the nodepool-launcher daemon.

   .. attr:: driver
      :type: string
      :default: openstack

      The driver type.

      .. value:: openstack

         For details on the extra options required and provided by the
         OpenStack driver, see the separate section
         :attr:`providers.[openstack]`

      .. value:: static

         For details on the extra options required and provided by the
         static driver, see the separate section
         :attr:`providers.[static]`

      .. value:: kubernetes

         For details on the extra options required and provided by the
         kubernetes driver, see the separate section
         :attr:`providers.[kubernetes]`

      .. value:: openshift

         For details on the extra options required and provided by the
         openshift driver, see the separate section
         :attr:`providers.[openshift]`

      .. value:: aws

         For details on the extra options required and provided by the
         AWS driver, see the separate section
         :attr:`providers.[aws]`

      .. value:: openshiftpods
         For details on the extra options required and provided by the
         openshiftpods driver, see the separate section
         :attr:`providers.[openshiftpods]`


OpenStack Driver
----------------

Selecting the OpenStack driver adds the following options to the
:attr:`providers` section of the configuration.

.. attr-overview::
   :prefix: providers.[openstack]
   :maxdepth: 3

.. attr:: providers.[openstack]

   Specifying the ``openstack`` driver for a provider adds the
   following keys to the :attr:`providers` configuration.

   .. note:: For documentation purposes the option names are prefixed
             ``providers.[openstack]`` to disambiguate from other
             drivers, but ``[openstack]`` is not required in the
             configuration (e.g. below ``providers.[openstack].cloud``
             refers to the ``cloud`` key in the ``providers`` section
             when the ``openstack`` driver is selected).

   An OpenStack provider's resources are partitioned into groups
   called "pools" (see :attr:`providers.[openstack].pools` for details),
   and within a pool, the node types which are to be made available
   are listed (see :attr:`providers.[openstack].pools.labels` for
   details).

   Within each OpenStack provider the available Nodepool image types
   are defined (see :attr:`providers.[openstack].diskimages`).

   .. code-block:: yaml

      providers:
        - name: provider1
          driver: openstack
          cloud: example
          region-name: 'region1'
          rate: 1.0
          boot-timeout: 120
          launch-timeout: 900
          launch-retries: 3
          image-name-format: '{image_name}-{timestamp}'
          hostname-format: '{label.name}-{provider.name}-{node.id}'
          diskimages:
            - name: trusty
              meta:
                  key: value
                  key2: value
            - name: precise
            - name: devstack-trusty
          pools:
            - name: main
              max-servers: 96
              availability-zones:
                - az1
              networks:
                - some-network-name
              security-groups:
                - zuul-security-group
              labels:
                - name: trusty
                  min-ram: 8192
                  diskimage: trusty
                  console-log: True
                - name: precise
                  min-ram: 8192
                  diskimage: precise
                - name: devstack-trusty
                  min-ram: 8192
                  diskimage: devstack-trusty
        - name: provider2
          driver: openstack
          cloud: example2
          region-name: 'region1'
          rate: 1.0
          image-name-format: '{image_name}-{timestamp}'
          hostname-format: '{label.name}-{provider.name}-{node.id}'
          diskimages:
            - name: precise
              meta:
                  key: value
                  key2: value
          pools:
            - name: main
              max-servers: 96
              labels:
                - name: trusty
                  min-ram: 8192
                  diskimage: trusty
                - name: precise
                  min-ram: 8192
                  diskimage: precise
                - name: devstack-trusty
                  min-ram: 8192
                  diskimage: devstack-trusty

   .. attr:: cloud
      :type: string
      :required:

      Name of a cloud configured in ``clouds.yaml``.

      The instances spawned by nodepool will inherit the default
      security group of the project specified in the cloud definition
      in `clouds.yaml` (if other values not specified). This means
      that when working with Zuul, for example, SSH traffic (TCP/22)
      must be allowed in the project's default security group for Zuul
      to be able to reach instances.

      More information about the contents of `clouds.yaml` can be
      found in `the openstacksdk documentation
      <https://docs.openstack.org/openstacksdk/>`_.

   .. attr:: boot-timeout
      :type: int seconds
      :default: 60

      Once an instance is active, how long to try connecting to the
      image via SSH.  If the timeout is exceeded, the node launch is
      aborted and the instance deleted.

   .. attr:: launch-timeout
      :type: int seconds
      :default: 3600

      The time to wait from issuing the command to create a new instance
      until that instance is reported as "active".  If the timeout is
      exceeded, the node launch is aborted and the instance deleted.

   .. attr:: nodepool-id
      :type: string
      :default: None

      *Deprecated*

      A unique string to identify which nodepool instances is using a
      provider.  This is useful if you want to configure production
      and development instances of nodepool but share the same
      provider.

  .. attr:: launch-retries
     :type: int
     :default: 3

      The number of times to retry launching a server before
      considering the job failed.

  .. attr:: region-name
     :type: string

     The region name if the provider cloud has multiple regions.

  .. attr:: hostname-format
     :type: string
     :default: {label.name}-{provider.name}-{node.id}

     Hostname template to use for the spawned instance.

  .. attr:: image-name-format
     :type: string
     :default: {image_name}-{timestamp}

     Format for image names that are uploaded to providers.

  .. attr:: rate
     :type: int seconds
     :default: 1

     In seconds, amount to wait between operations on the provider.

  .. attr:: clean-floating-ips
     :type: bool
     :default: True

     If it is set to True, nodepool will assume it is the only user of
     the OpenStack project and will attempt to clean unattached
     floating ips that may have leaked around restarts.

  .. attr:: diskimages
     :type: list

     Each entry in a provider's `diskimages` section must correspond
     to an entry in :attr:`diskimages`.  Such an entry indicates that
     the corresponding diskimage should be uploaded for use in this
     provider.  Additionally, any nodes that are created using the
     uploaded image will have the associated attributes (such as
     flavor or metadata).

     If an image is removed from this section, any previously uploaded
     images will be deleted from the provider.

     .. code-block:: yaml

        diskimages:
          - name: precise
            pause: False
            meta:
                key: value
                key2: value
          - name: windows
            connection-type: winrm
            connection-port: 5986

     Each entry is a dictionary with the following keys

     .. attr:: name
        :type: string
        :required:

        Identifier to refer this image from
        :attr:`providers.[openstack].pools.labels` and
        :attr:`diskimages` sections.

     .. attr:: pause
        :type: bool
        :default: False

        When set to True, nodepool-builder will not upload the image
        to the provider.

     .. attr:: config-drive
        :type: bool
        :default: unset

        Whether config drive should be used for the image. Defaults to
        unset which will use the cloud's default behavior.

     .. attr:: meta
        :type: dict

        Arbitrary key/value metadata to store for this server using
        the Nova metadata service. A maximum of five entries is
        allowed, and both keys and values must be 255 characters or
        less.

     .. attr:: connection-type
        :type: string

        The connection type that a consumer should use when connecting
        to the node. For most diskimages this is not
        necessary. However when creating Windows images this could be
        ``winrm`` to enable access via ansible.

     .. attr:: connection-port
        :type: int
        :default: 22 / 5986

        The port that a consumer should use when connecting to the
        node. For most diskimages this is not necessary. This defaults
        to 22 for ssh and 5986 for winrm.

  .. attr:: cloud-images
     :type: list

     Each entry in this section must refer to an entry in the
     :attr:`labels` section.

     .. code-block:: yaml

        cloud-images:
          - name: trusty-external
            config-drive: False
          - name: windows-external
            connection-type: winrm
            connection-port: 5986

     Each entry is a dictionary with the following keys

     .. attr:: name
        :type: string
        :required:

        Identifier to refer this cloud-image from :attr:`labels`
        section.  Since this name appears elsewhere in the nodepool
        configuration file, you may want to use your own descriptive
        name here and use one of ``image-id`` or ``image-name`` to
        specify the cloud image so that if the image name or id
        changes on the cloud, the impact to your Nodepool
        configuration will be minimal.  However, if neither of those
        attributes are provided, this is also assumed to be the image
        name or ID in the cloud.

     .. attr:: config-drive
        :type: bool
        :default: unset

        Whether config drive should be used for the cloud
        image. Defaults to unset which will use the cloud's default
        behavior.

     .. attr:: image-id
        :type: str

        If this is provided, it is used to select the image from the
        cloud provider by ID, rather than name.  Mutually exclusive
        with :attr:`providers.[openstack].cloud-images.image-name`

     .. attr:: image-name
        :type: str

        If this is provided, it is used to select the image from the
        cloud provider by this name or ID.  Mutually exclusive with
        :attr:`providers.[openstack].cloud-images.image-id`

     .. attr:: username
        :type: str

        The username that a consumer should use when connecting to the
        node.

     .. attr:: python-path
        :type: str
        :default: /usr/bin/python2

        The path of the default python interpreter.

     .. attr:: connection-type
        :type: str

        The connection type that a consumer should use when connecting
        to the node. For most diskimages this is not
        necessary. However when creating Windows images this could be
        'winrm' to enable access via ansible.

     .. attr:: connection-port
        :type: int
        :default: 22/ 5986

        The port that a consumer should use when connecting to the
        node. For most diskimages this is not necessary. This defaults
        to 22 for ssh and 5986 for winrm.

  .. attr:: pools
     :type: list

     A pool defines a group of resources from an OpenStack
     provider. Each pool has a maximum number of nodes which can be
     launched from it, along with a number of cloud-related attributes
     used when launching nodes.

     .. code-block:: yaml

        pools:
          - name: main
            max-servers: 96
            availability-zones:
              - az1
            networks:
              - some-network-name
            security-groups:
              - zuul-security-group
            auto-floating-ip: False
            host-key-checking: True
            node-attributes:
              key1: value1
              key2: value2
            labels:
              - name: trusty
                min-ram: 8192
                diskimage: trusty
                console-log: True
              - name: precise
                min-ram: 8192
                diskimage: precise
              - name: devstack-trusty
                min-ram: 8192
                diskimage: devstack-trusty

     Each entry is a dictionary with the following keys

     .. attr:: name
        :type: string
        :required:

        Pool name

     .. attr:: node-attributes
        :type: dict

        A dictionary of key-value pairs that will be stored with the node data
        in ZooKeeper. The keys and values can be any arbitrary string.

     .. attr:: max-cores
        :type: int

        Maximum number of cores usable from this pool. This can be used
        to limit usage of the tenant. If not defined nodepool can use
        all cores up to the quota of the tenant.

     .. attr:: max-servers
        :type: int

        Maximum number of servers spawnable from this pool. This can
        be used to limit the number of servers. If not defined
        nodepool can create as many servers the tenant allows.

     .. attr:: max-ram
        :type: int

        Maximum ram usable from this pool. This can be used to limit
        the amount of ram allocated by nodepool. If not defined
        nodepool can use as much ram as the tenant allows.

     .. attr:: ignore-provider-quota
        :type: bool
        :default: False

        Ignore the provider quota for this pool. Instead, only check
        against the configured max values for this pool and the
        current usage based on stored data. This may be useful in
        circumstances where the provider is incorrectly calculating
        quota.

     .. attr:: availability-zones
        :type: list

        A list of availability zones to use.

        If this setting is omitted, nodepool will fetch the list of
        all availability zones from nova.  To restrict nodepool to a
        subset of availability zones, supply a list of availability
        zone names in this setting.

        Nodepool chooses an availability zone from the list at random
        when creating nodes but ensures that all nodes for a given
        request are placed in the same availability zone.

     .. attr:: networks
        :type: list

        Specify custom Neutron networks that get attached to each
        node. Specify the name or id of the network as a string.

     .. attr:: security-groups
        :type: list

        Specify custom Neutron security groups that get attached to
        each node. Specify the name or id of the security_group as a
        string.

     .. attr:: auto-floating-ip
        :type: bool
        :default: True

        Specify custom behavior of allocating floating ip for each
        node.  When set to False, ``nodepool-launcher`` will not apply
        floating ip for nodes. When zuul instances and nodes are
        deployed in the same internal private network, set the option
        to False to save floating ip for cloud provider.

     .. attr:: host-key-checking
        :type: bool
        :default: True

        Specify custom behavior of validation of SSH host keys.  When
        set to False, nodepool-launcher will not ssh-keyscan nodes
        after they are booted. This might be needed if
        nodepool-launcher and the nodes it launches are on different
        networks.  The default value is True.

     .. attr:: labels
        :type: list

        Each entry in a pool`s `labels` section indicates that the
        corresponding label is available for use in this pool.  When
        creating nodes for a label, the flavor-related attributes in that
        label's section will be used.

        .. code-block:: yaml

           labels:
             - name: precise
               min-ram: 8192
               flavor-name: 'something to match'
               console-log: True
             - name: trusty
               min-ram: 8192
               networks:
                 - public
                 - private

        Each entry is a dictionary with the following keys

        .. attr:: name
           :type: str
           :required:

           Identifier to refer this image; from :attr:`labels` and
           :attr:`diskimages` sections.

        .. attr:: diskimage
           :type: str
           :required:

           Refers to provider's diskimages, see
           :attr:`providers.[openstack].diskimages`.  Mutually exclusive
           with :attr:`providers.[openstack].pools.labels.cloud-image`

        .. attr:: cloud-image
           :type: str
           :required:

           Refers to the name of an externally managed image in the
           cloud that already exists on the provider. The value of
           ``cloud-image`` should match the ``name`` of a previously
           configured entry from the ``cloud-images`` section of the
           provider. See :attr:`providers.[openstack].cloud-images`.
           Mutually exclusive with
           :attr:`providers.[openstack].pools.labels.diskimage`

        .. attr:: key-name
           :type: str

           Name or id of the ssh key to use.

        .. attr:: flavor-name
           :type: str

           Name or id of the flavor to use. If
           :attr:`providers.[openstack].pools.labels.min-ram` is
           omitted, it must be an exact match. If
           :attr:`providers.[openstack].pools.labels.min-ram` is given,
           ``flavor-name`` will be used to find flavor names that meet
           :attr:`providers.[openstack].pools.labels.min-ram` and also
           contain ``flavor-name``.

           One of :attr:`providers.[openstack].pools.labels.min-ram` or
           :attr:`providers.[openstack].pools.labels.flavor-name` must
           be specified.

        .. attr:: min-ram
           :type: int

           Determine the flavor to use (e.g. ``m1.medium``,
           ``m1.large``, etc).  The smallest flavor that meets the
           ``min-ram`` requirements will be chosen.

           One of :attr:`providers.[openstack].pools.labels.min-ram` or
           :attr:`providers.[openstack].pools.labels.flavor-name` must
           be specified.

        .. attr:: boot-from-volume
           :type: bool
           :default: False

            If given, the label for use in this pool will create a
            volume from the image and boot the node from it.

        .. attr:: host-key-checking
           :type: bool
           :default: True

           Specify custom behavior of validation of SSH host keys.  When set to
           False, nodepool-launcher will not ssh-keyscan nodes after they are
           booted. This might be needed if nodepool-launcher and the nodes it
           launches are on different networks.  The default value is True.

           .. note:: This value will override the value for
                     :attr:`providers.[openstack].pools.host-key-checking`.

        .. attr:: networks
           :type: list

           Specify custom Neutron networks that get attached to each
           node. Specify the name or id of the network as a string.

           .. note:: This value will override the value for
                     :attr:`providers.[openstack].pools.networks`.

        .. attr:: key-name
           :type: string

           If given, is the name of a keypair that will be used when
           booting each server.

        .. attr:: console-log
           :type: bool
           :default: False

           On the failure of the ssh ready check, download the server
           console log to aid in debuging the problem.

        .. attr:: volume-size
           :type: int gigabytes
           :default: 50

           When booting an image from volume, how big should the
           created volume be.

        .. attr:: instance-properties
           :type: dict
           :default: None

           A dictionary of key/value properties to set when booting
           each server.  These properties become available via the
           ``meta-data`` on the active server (e.g. within
           ``config-drive:openstack/latest/meta_data.json``)

        .. attr:: userdata
           :type: str
           :default: None

           A string of userdata for a node. Example usage is to install
           cloud-init package on image which will apply the userdata.
           Additional info about options in cloud-config:
           https://cloudinit.readthedocs.io/en/latest/topics/examples.html


Static Driver
-------------

Selecting the static driver adds the following options to the
:attr:`providers` section of the configuration.

.. attr-overview::
   :prefix: providers.[static]
   :maxdepth: 3

.. attr:: providers.[static]
   :type: list

   The static provider driver is used to define static nodes.

   .. note:: For documentation purposes the option names are prefixed
             ``providers.[static]`` to disambiguate from other
             drivers, but ``[static]`` is not required in the
             configuration (e.g. below ``providers.[static].pools``
             refers to the ``pools`` key in the ``providers`` section
             when the ``static`` driver is selected).

   Example:

   .. code-block:: yaml

      providers:
        - name: static-rack
          driver: static
          pools:
            - name: main
              nodes:
                - name: trusty.example.com
                  labels: trusty-static
                  timeout: 13
                  connection-port: 22022
                  host-key: fake-key
                  username: zuul
                  max-parallel-jobs: 1

   .. attr:: pools
      :type: list

      A pool defines a group of statically declared nodes.

      .. note::

         Although you may define more than one pool, it is essentially
         useless to do so since a node's ``name`` must be unique
         across all pools.

      Each entry is a dictionary with entries as follows

      .. attr:: name
         :type: str
         :required:

         Pool name

      .. attr:: nodes
         :type: list
         :required:

         Each entry indicates a static node and it's attributes.

         .. attr:: name
            :type: str
            :required:

            The hostname or ip address of the static node. This must be
            unique across all nodes defined within the configuration file.

         .. attr:: labels
            :type: list
            :required:

            The list of labels associated with the node.

         .. attr:: timeout
            :type: int
            :default: 5

            The timeout in second before the ssh ping is considered failed.

         .. attr:: connection-type
            :type: string

            The connection type that a consumer should use when connecting
            to the node.

            .. value:: winrm

            .. value:: ssh

         .. attr:: connection-port
            :type: int
            :default: 22 / 5986

            The port that a consumer should use when connecting to the node.
            For most nodes this is not necessary. This defaults to 22 when
            ``connection-type`` is 'ssh' and 5986 when it is 'winrm'.

         .. attr:: host-key
            :type: str

            The ssh host key of the node.

         .. attr:: username
            :type: str
            :default: zuul

            The username nodepool will use to validate it can connect to the
            node.

         .. attr:: python-path
            :type: str
            :default: /usr/bin/python2

            The path of the default python interpreter.

         .. attr:: max-parallel-jobs
            :type: int
            :default: 1

            The number of jobs that can run in parallel on this node.


Kubernetes Driver
-----------------

Selecting the kubernetes driver adds the following options to the
:attr:`providers` section of the configuration.

.. attr-overview::
   :prefix: providers.[kubernetes]
   :maxdepth: 3

.. attr:: providers.[kubernetes]
   :type: list

   A Kubernetes provider's resources are partitioned into groups
   called `pools` (see :attr:`providers.[kubernetes].pools` for
   details), and within a pool, the node types which are to be made
   available are listed (see :attr:`providers.[kubernetes].labels` for
   details).

   .. note:: For documentation purposes the option names are prefixed
             ``providers.[kubernetes]`` to disambiguate from other
             drivers, but ``[kubernetes]`` is not required in the
             configuration (e.g. below
             ``providers.[kubernetes].pools`` refers to the ``pools``
             key in the ``providers`` section when the ``kubernetes``
             driver is selected).

   Example:

   .. code-block:: yaml

     providers:
       - name: kubespray
         driver: kubernetes
         context: admin-cluster.local
         pools:
           - name: main
             labels:
               - name: kubernetes-namespace
                 type: namespace
               - name: pod-fedora
                 type: pod
                 image: docker.io/fedora:28


   .. attr:: context
      :required:

      Name of the context configured in ``kube/config``.

      Before using the driver, Nodepool services need a
      ``kube/config`` file manually installed with cluster admin
      context.

   .. attr:: launch-retries
      :default: 3

      The number of times to retry launching a node before considering
      the job failed.


   .. attr:: pools
      :type: list

      A pool defines a group of resources from a Kubernetes
      provider.

      .. attr:: name
         :required:

         Namespaces are prefixed with the pool's name.

   .. attr:: labels
      :type: list

      Each entry in a pool`s `labels` section indicates that the
      corresponding label is available for use in this pool.

      Each entry is a dictionary with the following keys

      .. attr:: name
         :required:

         Identifier for this label; references an entry in the
         :attr:`labels` section.

      .. attr:: type

         The Kubernetes provider supports two types of labels:

         .. value:: namespace

            Namespace labels provide an empty namespace configured
            with a service account that can create pods, services,
            configmaps, etc.

         .. value:: pod

            Pod labels provide a dedicated namespace with a single pod
            created using the
            :attr:`providers.[kubernetes].labels.image` parameter and it
            is configured with a service account that can exec and get
            the logs of the pod.

      .. attr:: image

         Only used by the
         :value:`providers.[kubernetes].labels.type.pod` label type;
         specifies the image name used by the pod.

      .. attr:: python-path
         :type: str
         :default: /usr/bin/python2

         The path of the default python interpreter.


Openshift Driver
----------------

Selecting the openshift driver adds the following options to the
:attr:`providers` section of the configuration.

.. attr-overview::
   :prefix: providers.[openshift]
   :maxdepth: 3

.. attr:: providers.[openshift]
   :type: list

   An Openshift provider's resources are partitioned into groups called `pool`
   (see :attr:`providers.[openshift].pools` for details), and within a pool,
   the node types which are to be made available are listed
   (see :attr:`providers.[openshift].labels` for details).

   .. note:: For documentation purposes the option names are prefixed
             ``providers.[openshift]`` to disambiguate from other
             drivers, but ``[openshift]`` is not required in the
             configuration (e.g. below
             ``providers.[openshift].pools`` refers to the ``pools``
             key in the ``providers`` section when the ``openshift``
             driver is selected).

   Example:

   .. code-block:: yaml

     providers:
       - name: cluster
         driver: openshift
         context: context-name
         pools:
           - name: main
             labels:
               - name: openshift-project
                 type: project
               - name: openshift-pod
                 type: pod
                 image: docker.io/fedora:28

   .. attr:: context
      :required:

      Name of the context configured in ``kube/config``.

      Before using the driver, Nodepool services need a ``kube/config`` file
      manually installed with self-provisioner (the service account needs to
      be able to create projects) context.
      Make sure the context is present in ``oc config get-contexts`` command
      output.

   .. attr:: launch-retries
      :default: 3

      The number of times to retry launching a node before considering
      the job failed.

   .. attr:: max-projects
      :default: infinite
      :type: int

      Maximum number of projects that can be used.

   .. attr:: pools
      :type: list

      A pool defines a group of resources from an Openshift provider.

      .. attr:: name
         :required:

         Project's name are prefixed with the pool's name.

   .. attr:: labels
      :type: list

      Each entry in a pool`s `labels` section indicates that the
      corresponding label is available for use in this pool.

      Each entry is a dictionary with the following keys

      .. attr:: name
         :required:

         Identifier for this label; references an entry in the
         :attr:`labels` section.

      .. attr:: type

         The Openshift provider supports two types of labels:

         .. value:: project

            Project labels provide an empty project configured
            with a service account that can create pods, services,
            configmaps, etc.

         .. value:: pod

            Pod labels provide a new dedicated project with a single
            pod created using the
            :attr:`providers.[openshift].labels.image` parameter and it
            is configured with a service account that can exec and get
            the logs of the pod.

      .. attr:: image

         Only used by the
         :value:`providers.[openshift].labels.type.pod` label type;
         specifies the image name used by the pod.

      .. attr:: image-pull
         :default: IfNotPresent
         :type: str

         The ImagePullPolicy, can be IfNotPresent, Always or Never.

      .. attr:: python-path
         :type: str
         :default: /usr/bin/python2

         The path of the default python interpreter.

      .. attr:: cpu
         :type: int

         Only used by the
         :value:`providers.[openshift].labels.type.pod` label type;
         specifies the number of cpu to request for the pod.

      .. attr:: memory
         :type: int

         Only used by the
         :value:`providers.[openshift].labels.type.pod` label type;
         specifies the amount of memory in MB to request for the pod.


Openshift Pods Driver
---------------------

Selecting the openshift pods driver adds the following options to the
:attr:`providers` section of the configuration.

.. attr:: providers.[openshiftpods]
   :type: list

   The Openshift Pods driver is similar to the Openshift driver, but it
   only support pod label to be created in a single project. This enable
   using an unprivileged service account that doesn't requires the
   self-provisioner role.

   Example:

   .. code-block:: yaml

     providers:
       - name: cluster
         driver: openshiftpods
         context: unprivileged-context-name
         pools:
           - name: main
             labels:
               - name: openshift-pod
                 image: docker.io/fedora:28

   .. attr:: context
      :required:

      Name of the context configured in ``kube/config``.

      Before using the driver, Nodepool services need a ``kube/config`` file
      manually installed with self-provisioner (the service account needs to
      be able to create projects) context.
      Make sure the context is present in ``oc config get-contexts`` command
      output.

   .. attr:: launch-retries
      :default: 3

      The number of times to retry launching a pod before considering
      the job failed.

   .. attr:: max-pods
      :default: infinite
      :type: int

      Maximum number of pods that can be used.

   .. attr:: pools
      :type: list

      A pool defines a group of resources from an Openshift provider.

      .. attr:: name
         :required:

         The project's name that will be used to create the pods.

   .. attr:: labels
      :type: list

      Each entry in a pool`s `labels` section indicates that the
      corresponding label is available for use in this pool.

      Each entry is a dictionary with the following keys

      .. attr:: name
         :required:

         Identifier for this label; references an entry in the
         :attr:`labels` section.

      .. attr:: image

         The image name.

      .. attr:: image-pull
         :default: IfNotPresent
         :type: str

         The ImagePullPolicy, can be IfNotPresent, Always or Never.

      .. attr:: cpu
         :type: int

         The number of cpu to request for the pod.

      .. attr:: memory
         :type: int

         The amount of memory in MB to request for the pod.


AWS EC2 Driver
--------------

Selecting the aws driver adds the following options to the :attr:`providers`
section of the configuration.

.. attr-overview::
   :prefix: providers.[aws]
   :maxdepth: 3

.. attr:: providers.[aws]
   :type: list

   An AWS provider's resources are partitioned into groups called `pool`
   (see :attr:`providers.[aws].pools` for details), and within a pool,
   the node types which are to be made available are listed
   (see :attr:`providers.[aws].pools.labels` for details).

   See `Boto Configuration`_ for information on how to configure credentials
   and other settings for AWS access in Nodepool's runtime environment.

   .. note:: For documentation purposes the option names are prefixed
             ``providers.[aws]`` to disambiguate from other
             drivers, but ``[aws]`` is not required in the
             configuration (e.g. below
             ``providers.[aws].pools`` refers to the ``pools``
             key in the ``providers`` section when the ``aws``
             driver is selected).

   Example:

   .. code-block:: yaml

     providers:
       - name: ec2-us-west-2
         driver: aws
         region-name: us-west-2
         cloud-images:
           - name: debian9
             image-id: ami-09c308526d9534717
             username: admin
         pools:
           - name: main
             max-servers: 5
             subnet-id: subnet-0123456789abcdef0
             security-group-id: sg-01234567890abcdef
             labels:
               - name: debian9
                 cloud-image: debian9
                 instance-type: t3.medium
                 key-name: zuul
               - name: debian9-large
                 cloud-image: debian9
                 instance-type: t3.large
                 key-name: zuul

   .. attr:: name
      :required:

      A unique name for this provider configuration.

   .. attr:: region
      :required:

      Name of the `AWS region`_ to interact with.

   .. attr:: profile-name

      The AWS credentials profile to load for this provider. If unspecified the
      `boto3` library will select a profile.

      See `Boto Configuration`_ for more information.

   .. attr:: boot-timeout
      :type: int seconds
      :default: 60

      Once an instance is active, how long to try connecting to the
      image via SSH.  If the timeout is exceeded, the node launch is
      aborted and the instance deleted.

   .. attr:: launch-retries
      :default: 3

      The number of times to retry launching a node before considering
      the job failed.

   .. attr:: cloud-images
      :type: list

      Each entry in this section must refer to an entry in the
      :attr:`providers.[aws].cloud-images` section.

      .. code-block:: yaml

         cloud-images:
           - name: ubuntu1804
             image-id: ami-082fd9a18128c9e8c
             username: ubuntu
           - name: my-custom-win2k3
             connection-type: winrm
             username: admin

      Each entry is a dictionary with the following keys

      .. attr:: name
         :type: string
         :required:

         Identifier to refer this cloud-image from :attr:`labels` section.
         Since this name appears elsewhere in the nodepool configuration file,
         you may want to use your own descriptive name here and use
         ``image-id`` to specify the cloud image so that if
         the image id changes on the cloud, the impact to your Nodepool
         configuration will be minimal. However, if ``image-id`` is not
         provided, this is assumed to be the image id in the cloud.

      .. attr:: image-id
         :type: str

         If this is provided, it is used to select the image from the cloud
         provider by ID.

      .. attr:: username
         :type: str

         The username that a consumer should use when connecting to the node.

      .. attr:: python-path
         :type: str
         :default: /usr/bin/python2

         The path of the default python interpreter.

      .. attr:: connection-type
         :type: str

         The connection type that a consumer should use when connecting to the
         node. For most images this is not necessary. However when creating
         Windows images this could be 'winrm' to enable access via ansible.

      .. attr:: connection-port
         :type: int
         :default: 22/ 5986

         The port that a consumer should use when connecting to the node. For
         most diskimages this is not necessary. This defaults to 22 for ssh and
         5986 for winrm.

   .. attr:: pools
      :type: list

      A pool defines a group of resources from an AWS provider. Each pool has a
      maximum number of nodes which can be launched from it, along with a number
      of cloud-related attributes used when launching nodes.

      .. attr:: name
         :required:

         A unique name within the provider for this pool of resources.

      .. attr:: subnet-id

         If provided, specifies the subnet to assign to the primary network
         interface of nodes.

      .. attr:: security-group-id

         If provided, specifies the security group ID to assign to the primary
         network interface of nodes.

      .. attr:: host-key-checking
         :type: bool
         :default: True

         Specify custom behavior of validation of SSH host keys.  When set to
         False, nodepool-launcher will not ssh-keyscan nodes after they are
         booted. This might be needed if nodepool-launcher and the nodes it
         launches are on different networks.  The default value is True.

      .. attr:: labels
         :type: list

         Each entry in a pool's `labels` section indicates that the
         corresponding label is available for use in this pool.  When creating
         nodes for a label, the flavor-related attributes in that label's
         section will be used.

         .. code-block:: yaml

            labels:
              - name: bionic
                instance-type: m5a.large
                console-log: True

         Each entry is a dictionary with the following keys

           .. attr:: name
              :type: str
              :required:

              Identifier to refer this label.

           .. attr:: cloud-image
              :type: str
              :required:

              Refers to the name of an externally managed image in the
              cloud that already exists on the provider. The value of
              ``cloud-image`` should match the ``name`` of a previously
              configured entry from the ``cloud-images`` section of the
              provider. See :attr:`providers.[aws].cloud-images`.

           .. attr:: instance-type
              :type: str
              :required:

              Name of the flavor to use.

           .. attr:: key-name
              :type: string

              If given, the name of a keypair that will be used when
              booting each server.

           .. attr:: volume-type
              :type: string

              If given, the root `EBS volume type`_

           .. attr:: volume-size
              :type: int

              If given, the size of the root EBS volume, in GiB.


.. _`EBS volume type`: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html
.. _`AWS region`: https://docs.aws.amazon.com/general/latest/gr/rande.html
.. _`Boto configuration`: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
