Kubernetes Custom Configuration

Introduction

Kubernetes configuration can be customized during deployment by specifying bootstrap overrides in the localhost.yml file during the Ansible bootstrap process or during runtime via sysinv service-parameters CLI.

Custom configuration includes:

  • Configuring options on kube-apiserver such as feature gates and admission controllers,

  • Configuring options on kube-controller-manager such as node-monitor-period and pod-eviction-timeout,

  • Configuring options on kube-scheduler such as feature gates,

  • Configuring options on kubelet such as maximum pods and enabling unsafe sysctls.

kube-apiserver configuration

The Kubernetes API server validates and configures data for the API objects which include pods, services, replicationcontrollers, and others. The API Server services REST operations and provides the frontend to the cluster’s shared state through which all other components interact.

For a list of all configurable options of kube-apiserver, see kube-apiserver.

Bootstrap configuration

To set or override a kube-apiserver option, add the desired parameters to an apiserver_extra_args section in the localhost.yml.

Example usage:

apiserver_extra_args:
  admission-control-config-file: "/etc/kubernetes/admission-control-config-file.yml"
  audit-policy-file: "/etc/kubernetes/audit-policy-file.yml"
  default-not-ready-toleration-seconds: "35"
  default-unreachable-toleration-seconds: "35"
  feature-gates: "SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true,RemoveSelfLink=false,MemoryManager=true"
  enable-admission-plugins: "NodeRestriction,PodNodeSelector"
  event-ttl: "20h"

Runtime configuration

To set, modify or delete a kube-apiserver parameter use the service-parameter add, modify or delete CLI command.

Example usage:

  • Add new parameter

    system service-parameter-add kubernetes kube_apiserver default-not-ready-toleration-seconds=31

system service-parameter-apply kubernetes

    Note

    Parameter must not exist on service parameters, otherwise use modify command.

  • Modify existing parameter

    system service-parameter-modify kubernetes kube_apiserver default-not-ready-toleration-seconds=33

system service-parameter-apply kubernetes

  • Delete parameter

    system service-parameter-list

    Copy parameter uuid to be deleted:

    system service-parameter-delete <uuid>

kube-controller-manager configuration

The Kubernetes controller manager is a daemon that embeds the core control loops shipped with Kubernetes. A controller is a control loop that watches the shared state of the cluster through the apiserver and makes changes attempting to move the current state towards the desired state.

For a list of all configurable options of kube-controller-manager, see kube-controller-manager.

Bootstrap configuration

To set or override a kube-controller-manager option, add the desired parameters to an controllermanager_extra_args section in the localhost.yml.

Example usage:

controllermanager_extra_args:
  node-monitor-period: "4s"
  node-monitor-grace-period: "25s"
  pod-eviction-timeout: '35s'
  feature-gates: "TTLAfterFinished=true,MemoryManager=true"

Runtime configuration

To set, modify or delete a kube-controller-manager parameter use the service-parameter add, modify or delete CLI command.

Example usage:

  • Add new parameter

    system service-parameter-add kubernetes kube_controller_manager node-monitor-period=5s

system service-parameter-apply kubernetes

    Note

    Parameter must not exist on service parameters, otherwise use modify command.

  • Modify existing parameter

    system service-parameter-modify kubernetes kube_controller_manager node-monitor-period=7s

system service-parameter-apply kubernetes

  • Delete parameter

    system service-parameter-list

    Copy parameter uuid to be deleted:

    system service-parameter-delete <uuid>

kube-scheduler configuration

The Kubernetes scheduler is a control plane process which assigns Pods to Nodes. The scheduler determines which Nodes are valid placements for each Pod in the scheduling queue according to constraints and available resources. The scheduler then ranks each valid Node and binds the Pod to a suitable Node. Multiple different schedulers may be used within a cluster; kube-scheduler is the reference implementation.

For a list of all configurable options of kube-scheduler, see kube-scheduler.

Bootstrap configuration

To set or override a kube-scheduler option, add the desired parameters to an scheduler_extra_args section in the localhost.yml.

Example usage:

scheduler_extra_args:
  feature-gates: "TTLAfterFinished=false"

Runtime configuration

To set, modify or delete a kube-controller-manager parameter use the service-parameter add, modify or delete CLI command.

Example usage:

  • Add new parameter

    system service-parameter-add kubernetes kube_scheduler leader-elect-lease-duration=16s

system service-parameter-apply kubernetes

    Note

    Parameter must not exist on service parameters, otherwise use modify command.

  • Modify existing parameter

    system service-parameter-modify kubernetes kube_scheduler leader-elect-lease-duration=14s

system service-parameter-apply kubernetes

  • Delete parameter

    system service-parameter-list

    Copy parameter uuid to be deleted:

    system service-parameter-delete <uuid>

kubelet configurations

The kubelet is the primary “node agent” that runs on each node.

For a list of all configurable options, see Kubelet Configuration (v1beta1).

Bootstrap configuration

To set or override a kubelet option, add the desired parameters to a kubelet_configurations section in the localhost.yml.

Example usage:

kubelet_configurations:
  featureGates:
    MemoryManager: true
    HugePageStorageMediumSize: true

Runtime configuration

To set, modify or delete a kubelet parameter use the service-parameter add, modify or delete CLI command.

The expected structure for existing field types are:

  • String, bool, meta/v1.Duration:

    • No structure defined, quotes may or may not be used.

    • Example: cgroupDriver=cgroupfs or cgroupDriver="cgroupfs"

  • int32,int64:

    • No structure defined, quotes may or may not be used.

    • Example: imageGCLowThresholdPercent=70 or imageGCLowThresholdPercent="70"

  • Array of strings ([]string):

    • JSON-like format: '["string1","string2","stringN"]'

    • Example: clusterDNS='["10.96.0.10"]'

  • map[string]string: json format.

    • JSON-like format: '{"key_string1":"string1","key_string2":"string2","key_stringN":"stringN"}'

    • Example: evictionHard='{"memory.available": "100Mi", "nodefs.available": "10%","nodefs.inodesFree": "6%", "imagefs.available": "2Gi"}'

Example usage:

  • Add new parameter

    system service-parameter-add kubernetes kubelet clusterDNS='["10.96.0.10"]'

system service-parameter-apply kubernetes

    Note

    Parameter must not exist on service parameters, otherwise use modify` command.

  • Modify existent parameter

    system service-parameter-modify kubernetes kubelet nodeStatusUpdateFrequency="5s"

system service-parameter-apply kubernetes

  • Delete parameter

    system service-parameter-list

    Copy parameter uuid to be deleted:

    system service-parameter-delete <uuid>

kube-apiserver, kube-controller-manager and kube-scheduler extra-volumes configuration

Some options/parameters specified in apiserver_extra_args, controller-manager_extra_args and scheduler extra-args refer to configuration files or directories. Those referenced files or directories must be mounted as volumes on the corresponding control plane pod using the extra-volume parameters.

Bootstrap configuration

To set or override an extra-volumes option, add the desired parameters to the corresponding extra-args section in the localhost.yml, add the desired extra-volume including the volume details and file contents (if corresponds).

For instance, if admission plugins are configured and need additional configuration, that configuration should be set in a specific file referenced by the admission-control-config-file parameter.

See the example below where the admission-control-config-file option and the PodNodeSelector admission plugin is specified for kube-apiserver. Both of these options require the specification of a yaml file.

Example usage:

apiserver_extra_args:
  admission-control-config-file: "/etc/kubernetes/admission-control-config-file.yaml"
  enable-admission-plugins: "PodNodeSelector"

apiserver_extra_volumes:
  - name: admission-control-config-file
    mountPath: "/etc/kubernetes/admission-control-config-file.yaml"
    hostPath: "/etc/kubernetes/admission-control-config-file.yaml"
    readOnly: true
    pathType: "File"
    content: |
      apiVersion: apiserver.config.k8s.io/v1
      kind: AdmissionConfiguration
      plugins:
      - name: PodSecurity
        configuration:
          apiVersion: pod-security.admission.config.k8s.io/v1beta1
          kind: PodSecurityConfiguration
          # Defaults applied when a mode label is not set.
          #
          # Level label values must be one of:
          # - "privileged" (default)
          # - "baseline"
          # - "restricted"
          #
          # Version label values must be one of:
          # - "latest" (default)
          # - specific version like "v1.24"
          defaults:
            enforce: "privileged"
            enforce-version: "latest"
            audit: "privileged"
            audit-version: "latest"
            warn: "privileged"
            warn-version: "latest"
          exemptions:
            # Array of authenticated usernames to exempt.
            usernames: []
            # Array of runtime class names to exempt.
            runtimeClasses: []
            # Array of namespaces to exempt.
            namespaces: []
      - name: pod-node-selector
        mountPath: "/etc/kubernetes/pod-node-selector.yaml"
        hostPath: "/etc/kubernetes/pod-node-selector.yaml"
        readOnly: true
        pathType: "File"
        content: |
          podNodeSelectorPluginConfig:
          clusterDefaultNodeSelector: name-of-node-selector
          namespace1: name-of-node-selector
          namespace2: name-of-node-selector

The example below enables kubernetes auditing which requires an audit-policy-file.yaml file to specify the details of what events should be audited.

Example usage:

apiserver_extra_args:
  audit-policy-file: /etc/kubernetes/audit-policy-file.yaml
  audit-log-path: /var/log/kubernetes/audit/audit.log

apiserver_extra_volumes:
  - name: audit-policy-file
    mountPath: "/etc/kubernetes/audit-policy-file.yaml"
    hostPath: "/etc/kubernetes/audit-policy-file.yaml"
    readOnly: true
    pathType: "File"
    content: |
      # Log all requests at the Metadata level.
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
      - level: Metadata
  - name: audit-log-path
    mountPath: "/var/log/kubernetes/audit/"
    hostPath: "/var/log/kubernetes/audit/"
    readOnly: false
    pathType: "DirectoryOrCreate"

Runtime configuration

To set, modify or delete an extra-volume parameter use the service-parameter add, modify or delete CLI command.

Valid extra-volume sections:

  • kube_apiserver_volumes

  • kube_controller_manager_volumes

  • kube_scheduler_volumes

Valid extra-volume parameter fields:

  • name:

    • Volume name.

  • hostPath:

    • Absolute path in node file system where the file or directory to mount is located.

  • mounthPath (opc):

    • Absolute path in pod file system used to mount the file or directory.

    • Default value: same as hostPath.

  • pathType (opc):

    • The supported values are:

      • DirectoryOrCreate: If nothing exists at the given path, an empty directory will be created there as needed with permission set to 0755, having the same group and ownership with Kubelet.

      • File: A file must exist at the given path.

    • Default value: File.

  • readOnly (opc):

    • The supported values are: true or false.

    • Default value: true.

Valid input formats:

  • Pairs of strings separated by commas:

    • Example: audit-log-dir=hostPath:/var/log/kubernetes/audit,readOnly:false,pathType:DirectoryOrCreate

  • JSON format:

    • Example: encryption-config='{"name": "encryption-config", "hostPath": "/etc/kubernetes/encryption-provider.yaml", "mountPath": "/etc/kubernetes/encryption-provider.yaml", "readOnly": true, "pathType":"File"}'

Example usage with two linked configuration files:

An admission controller could be added to mitigates the problem when the API server gets flooded by requests to store new Events. The cluster admin can specify event rate limits by enabling the EventRateLimit admission controller and referencing an EventRateLimit configuration file.

  • Add new extra-args parameters

    system service-parameter-add kubernetes kube_apiserver admission-control-config-file=/etc/kubernetes/admission-control-config-file.yaml

system service-parameter-add kubernetes kube_apiserver enable-admission-plugins=EventRateLimit

    Note

    Parameter must not exist on service parameters, otherwise use modify command.

  • Add new extra-volume parameters

    system service-parameter-add kubernetes kube_apiserver_volumes admission-control-config-file=hostPath:/etc/kubernetes/admission-control-config-file.yaml

system service-parameter-add kubernetes kube_apiserver_volumes eventconfig=hostPath:/etc/kubernetes/eventconfig.yaml

system service-parameter-apply kubernetes

    Note

    Parameter must not exist on service parameters, otherwise use modify command.

  • Modify existent parameter

    The configuration file name, for example, can be changed. During this operation, the preloaded configuration file will be replaced.

    system service-parameter-modify kubernetes kube_apiserver_volumes admission-control-config-file=hostPath:/etc/kubernetes/new-admission-control-config-file.yaml

system service-parameter-apply kubernetes

  • Delete parameters

    system service-parameter-list

    Copy parameter uuid to be deleted:

    system service-parameter-delete <uuid>

    In the current example, if EventRateLimit is no longer needed, it should be removed from the kube_apiserver enable-admission-plugins parameter, either by changing its value or by removing the parameter. Then the extra-volume kube_apiserver_volumes eventconfig parameter can be deleted. If the configuration file is no longer needed, the kube_apserver admission-control-config-file parameter can also be removed. Then the-extra volume kube_apiserver_volumes connections-control-config-file can be deleted.

  • Configuration Files Examples:

    • admission-control-config-file.yaml

      apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
  - name: EventRateLimit
    path: /etc/kubernetes/eventconfig.yaml

    • eventconfig.yaml

      apiVersion: eventratelimit.admission.k8s.io/v1alpha1
kind: Configuration
limits:
  - type: Namespace
    qps: 50
    burst: 100
    cacheSize: 2000
  - type: User
    qps: 10
    burst: 50

Complex Example configuration

apiserver_extra_args:
  admission-control-config-file: "/etc/kubernetes/admission-control-config-file.yml"
  audit-policy-file: "/etc/kubernetes/audit-policy-file.yml"
  default-not-ready-toleration-seconds: "35"
  default-unreachable-toleration-seconds: "35"
  feature-gates: "SCTPSupport=true,TTLAfterFinished=true,HugePageStorageMediumSize=true,RemoveSelfLink=false,MemoryManager=true"
  enable-admission-plugins: "NodeRestriction,PodNodeSelector"
  event-ttl: "20h"
  audit-log-path: "/var/log/kubernetes/audit/audit.log"
  audit-log-maxage: "1"
  audit-log-maxbackup: "2"
  audit-log-maxsize: "1"

scheduler_extra_args:
  feature-gates: "TTLAfterFinished=false"

controllermanager_extra_args:
  node-monitor-period: "4s"
  node-monitor-grace-period: "25s"
  pod-eviction-timeout: '35s'
  feature-gates: "TTLAfterFinished=true,MemoryManager=true"

kubelet_configurations:
  featureGates:
    MemoryManager: true
    HugePageStorageMediumSize: true

apiserver_extra_volumes:
  - name: admission-control-config-file
    mountPath: "/etc/kubernetes/admission-control-config-file.yml"
    hostPath: "/etc/kubernetes/admission-control-config-file.yml"
    pathType: "File"
    readOnly: true
    content: |
      apiVersion: apiserver.config.k8s.io/v1
      kind: AdmissionConfiguration
      plugins:
      - name: PodNodeSelector
        path: /etc/kubernetes/podnodeselector.yaml
  - name: pod-nodes-selector-plugin-config
    mountPath: "/etc/kubernetes/podnodeselector.yaml"
    hostPath: "/etc/kubernetes/podnodeselector.yaml"
    pathType: "File"
    readOnly: true
    content: |
      podNodeSelecto+rPluginConfig:
      clusterDefaultNodeSelector: name-of-node-selector
      namespace1: name-of-node-selector
      namespace2: name-of-node-selector
  - name: audit-policy-file
    mountPath: "/etc/kubernetes/audit-policy-file.yml"
    hostPath: "/etc/kubernetes/audit-policy-file.yml"
    pathType: "File"
    readOnly: true
    content: |
      # Log all requests at the Metadata level.
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
      - level: Metadata
  - name: audit-log-path
    mountPath: "/var/log/kubernetes/audit/"
    hostPath: "/var/log/kubernetes/audit/"
    readOnly: false
    pathType: 'DirectoryOrCreate'

scheduler_extra_volumes:
  - name: sch-admission-control-config-file
    mountPath: "/etc/kubernetes/admission-control-config-file.yml"
    hostPath: "/etc/kubernetes/admission-control-config-file.yml"
    pathType: "File"
    readOnly: true
    content: |
      apiVersion:
      kind: AdmissionConfiguration
      plugins:
      - name: PodNodeSelector
        path: /etc/kubernetes/podnodeselector.yaml
  - name: sch-pod-nodes-selector-plugin-config
    mountPath: "/etc/kubernetes/podnodeselector.yaml"
    hostPath: "/etc/kubernetes/podnodeselector.yaml"
    pathType: "File"
    readOnly: true
    content: |
      podNodeSelectorPluginConfig:
      clusterDefaultNodeSelector: name-of-node-selector
      namespace1: name-of-node-selector
      namespace2: name-of-node-selector
  - name: sch-audit-policy-file
    mountPath: "/etc/kubernetes/audit-policy-file.yml"
    hostPath: "/etc/kubernetes/audit-policy-file.yml"
    pathType: "File"
    readOnly: true
    content: |
      # Log all requests at the Metadata level.
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
      - level: Metadata

controllermanager_extra_volumes:
   - name: cm-admission-control-config-file
     mountPath: "/etc/kubernetes/admission-control-config-file.yml"
     hostPath: "/etc/kubernetes/admission-control-config-file.yml"
     pathType: "File"
     readOnly: true
     content: |
       apiVersion: apiserver.config.k8s.io/v1
       kind: AdmissionConfiguration
       plugins:
       - name: PodNodeSelector
         path: /etc/kubernetes/podnodeselector.yaml
   - name: cm-pod-nodes-selector-plugin-config
     mountPath: "/etc/kubernetes/podnodeselector.yaml"
     hostPath: "/etc/kubernetes/podnodeselector.yaml"
     pathType: "File"
     readOnly: true
     content: |
       podNodeSelectorPluginConfig:
       clusterDefaultNodeSelector: name-of-node-selector
       namespace1: name-of-node-selector
       namespace2: name-of-node-selector
   - name: cm-audit-policy-file
     mountPath: "/etc/kubernetes/audit-policy-file.yml"
     hostPath: "/etc/kubernetes/audit-policy-file.yml"
     pathType: "File"
     readOnly: true
     content: |
       # Log all requests at the Metadata level.
       apiVersion: audit.k8s.io/v1
       kind: Policy
       rules:
       - level: Metadata