Kubernetes Operator Command Logging

The Kubernetes auditing provides a set of records that document the sequence of actions in a cluster. For more details, see https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/.

You can configure which events should be logged through a set of rules written in a YAML file, see https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/#audit-policy. A default policy file is provided in StarlingX at /etc/kubernetes/default-audit-policy.yaml. This default policy file is a version of the audit profile for Google Container-Optimized OS.

Kubernetes API Logging can be enabled and configured in StarlingX, and can be fully configured and enabled at bootstrap time. Post-bootstrap, Kubernetes API Logging can only be enabled or disabled.

The default policy file provided, present at /etc/kubernetes/default-audit-policy.yaml, is a version of the audit profile for Google Container-Optimized OS extracted from https://github.com/kubernetes/kubernetes/blob/75e49ec824b183288e1dbaccfd7dbe77d89db381/cluster/gce/gci/configure-helper.sh#L1129. Different log levels are used for different Kubernetes components.

The reference for the kube-apiserver parameters associated with Kubernetes API Logging can be found at https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/.

audit-policy-file

This parameter contains the full path of the audit policy configuration file to be used (e.g. /etc/kubernetes/default-audit-policy.yaml). When this parameter is present, the feature is enabled. In StarlingX r9.0, by default, this parameter is absent and the feature is disabled.

audit-log-path

This parameter points to the log file where the logs will be written. In StarlingX r9.0, by default, this parameter is present with the value /var/log/kubernetes/audit/audit.log. It is recommended to use the default value.

audit-log-maxsize

This parameter indicates the maximum size in megabytes of the audit log file before it gets rotated. In StarlingX r9.0, by default, this parameter is present with the value “100”, that means 100MB.

audit-log-maxage

This parameter indicates the maximum number of days to retain old audit log files. In StarlingX r9.0, by default, this parameter is present with the value “3”, that means 3 days.

audit-log-maxbackup

This parameter indicates the maximum number of old audit log files to retain. In StarlingX r9.0, by default, this parameter is present with the value “10”, that means that 10 old files are kept.

Bootstrap configuration of audit parameters

At bootstrap, all five parameters are configurable. When the value of these parameters are not overridden, the deployed environment will have the feature disabled, as the parameter audit-policy-file will be absent, and the other parameters will be present with the default values.

You can see below a YAML example that configures, at bootstrap in /home/sysadmin/localhost.yml, all parameters and defines the contents of a custom policy file to be used with the apiserver_extra_volumes: {name: my-audit-policy-file ...} parameter. By configuring the parameter audit-policy-file the feature will be enabled.

apiserver_extra_args:
  audit-log-maxage: '2'
  audit-log-maxbackup: '3'
  audit-log-maxsize: '40'
  audit-log-path: '/var/log/kubernetes/audit/audit.log'
  audit-policy-file: '/etc/kubernetes/my-audit-policy-file.yaml'
apiserver_extra_volumes:
  - name: my-audit-policy-file
    mountPath: '/etc/kubernetes/my-audit-policy-file.yaml'
    hostPath: '/etc/kubernetes/my-audit-policy-file.yaml'
    pathType: File
    readOnly: true
    content: |
      # Log all requests at the Metadata level.
      apiVersion: audit.k8s.io/v1
      kind: Policy
      rules:
      - level: Metadata

Runtime Configuration of audit parameters

After deploy, only the parameter audit-policy-file is configurable as a system service parameter, allowing the user to enable/disable the feature.

You can find below an example of how to add this parameter. The feature is disabled when the parameter is removed (i.e. system service-parameter-delete).

~(keystone_admin)$ system service-parameter-add kubernetes kube_apiserver audit-policy-file=/etc/kubernetes/default-audit-policy.yaml
~(keystone_admin)$ system service-parameter-apply kubernetes

Limitations

In StarlingX r9.0, a custom policy file can only be created at bootstrap time in apiserver_extra_volumes section. If a custom policy file was configured at bootstrap, then after bootstrap the user has the option to configure the parameter audit-policy-file to either this custom policy file (/etc/kubernetes/my-audit-policy-file.yaml in the example above) or the default policy file /etc/kubernetes/default-audit-policy.yaml. If no custom policy file was configured at bootstrap, then the user can only configure the parameter audit-policy-file to the default policy file.

Only the parameter audit-policy-file is configurable after bootstrap, so the others (audit-log-path, audit-log-maxsize, audit-log-maxage and audit-log-maxbackup) cannot be changed at runtime.