Fluent-logging¶
The fluent-logging chart in openstack-helm-infra provides the base for a centralized logging platform for OpenStack-Helm. The chart combines two services, Fluentbit and Fluentd, to gather logs generated by the services, filter on or add metadata to logged events, then forward them to Elasticsearch for indexing.
Fluentbit¶
Fluentbit runs as a log-collecting component on each host in the cluster, and can be configured to target specific log locations on the host. The Fluentbit configuration schema can be found on the official Fluentbit website.
Fluentbit provides a set of plug-ins for ingesting and filtering various log types. These plug-ins include:
Tail: Tails a defined file for logged events
Kube: Adds Kubernetes metadata to a logged event
Systemd: Provides ability to collect logs from the journald daemon
Syslog: Provides the ability to collect logs from a Unix socket (TCP or UDP)
The complete list of plugins can be found in the configuration section of the Fluentbit documentation.
Fluentbit uses parsers to turn unstructured log entries into structured entries to make processing and filtering events easier. The two formats supported are JSON maps and regular expressions. More information about Fluentbit’s parsing abilities can be found in the parsers section of Fluentbit’s documentation.
Fluentbit’s service and parser configurations are defined via the values.yaml file, which allows for custom definitions of inputs, filters and outputs for your logging needs. Fluentbit’s configuration can be found under the following key:
conf:
fluentbit:
- service:
header: service
Flush: 1
Daemon: Off
Log_Level: info
Parsers_File: parsers.conf
- containers_tail:
header: input
Name: tail
Tag: kube.*
Path: /var/log/containers/*.log
Parser: docker
DB: /var/log/flb_kube.db
Mem_Buf_Limit: 5MB
- kube_filter:
header: filter
Name: kubernetes
Match: kube.*
Merge_JSON_Log: On
- fluentd_output:
header: output
Name: forward
Match: "*"
Host: ${FLUENTD_HOST}
Port: ${FLUENTD_PORT}
Fluentbit is configured by default to capture logs at the info log level. To change this, override the Log_Level key with the appropriate levels, which are documented in Fluentbit’s configuration.
Fluentbit’s parser configuration can be found under the following key:
conf:
parsers:
- docker:
header: parser
Name: docker
Format: json
Time_Key: time
Time_Format: "%Y-%m-%dT%H:%M:%S.%L"
Time_Keep: On
The values for the fluentbit and parsers keys are consumed by a fluent-logging helper template that produces the appropriate configurations for the relevant sections. Each list item (keys prefixed with a ‘-’) represents a section in the configuration files, and the arbitrary name of the list item should represent a logical description of the section defined. The header key represents the type of definition (filter, input, output, service or parser), and the remaining entries will be rendered as space delimited configuration keys and values. For example, the definitions above would result in the following:
[SERVICE]
Daemon false
Flush 1
Log_Level info
Parsers_File parsers.conf
[INPUT]
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Name tail
Parser docker
Path /var/log/containers/*.log
Tag kube.*
[FILTER]
Match kube.*
Merge_JSON_Log true
Name kubernetes
[OUTPUT]
Host ${FLUENTD_HOST}
Match *
Name forward
Port ${FLUENTD_PORT}
[PARSER]
Format json
Name docker
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep true
Time_Key time
Fluentd¶
Fluentd runs as a forwarding service that receives event entries from Fluentbit and routes them to the appropriate destination. By default, Fluentd will route all entries received from Fluentbit to Elasticsearch for indexing. The Fluentd configuration schema can be found at the official Fluentd website.
Fluentd’s configuration is handled in the values.yaml file in fluent-logging. Similar to Fluentbit, configuration overrides provide flexibility in defining custom routes for tagged log events. The configuration can be found under the following key:
conf:
fluentd:
- fluentbit_forward:
header: source
type: forward
port: "#{ENV['FLUENTD_PORT']}"
bind: 0.0.0.0
- elasticsearch:
header: match
type: elasticsearch
expression: "**"
include_tag_key: true
host: "#{ENV['ELASTICSEARCH_HOST']}"
port: "#{ENV['ELASTICSEARCH_PORT']}"
logstash_format: true
buffer_chunk_limit: 10M
buffer_queue_limit: 32
flush_interval: "20"
max_retry_wait: 300
disable_retry_limit: ""
The values for the fluentd keys are consumed by a fluent-logging helper template that produces appropriate configurations for each directive desired. The list items (keys prefixed with a ‘-’) represent sections in the configuration file, and the name of each list item should represent a logical description of the section defined. The header key represents the type of definition (name of the fluentd plug-in used), and the expression key is used when the plug-in requires a pattern to match against (example: matches on certain input patterns). The remaining entries will be rendered as space delimited configuration keys and values. For example, the definition above would result in the following:
<source>
bind 0.0.0.0
port "#{ENV['FLUENTD_PORT']}"
@type forward
</source>
<match **>
buffer_chunk_limit 10M
buffer_queue_limit 32
disable_retry_limit
flush_interval 20s
host "#{ENV['ELASTICSEARCH_HOST']}"
include_tag_key true
logstash_format true
max_retry_wait 300
port "#{ENV['ELASTICSEARCH_PORT']}"
@type elasticsearch
</match>
Some fluentd plug-ins require nested definitions. The fluentd helper template can handle these definitions with the following structure:
conf:
td_agent:
- fluentbit_forward:
header: source
type: forward
port: "#{ENV['FLUENTD_PORT']}"
bind: 0.0.0.0
- log_transformer:
header: filter
type: record_transformer
expression: "foo.bar"
inner_def:
- record_transformer:
header: record
hostname: my_host
tag: my_tag
In this example, the my_transformer list will generate a nested configuration entry in the log_transformer section. The nested definitions are handled by supplying a list as the value for an arbitrary key, and the list value will indicate the entry should be handled as a nested definition. The helper template will render the above example key/value pairs as the following:
<source>
bind 0.0.0.0
port "#{ENV['FLUENTD_PORT']}"
@type forward
</source>
<filter foo.bar>
<record>
hostname my_host
tag my_tag
</record>
@type record_transformer
</filter>
Fluentd Exporter¶
The fluent-logging chart contains templates for an exporter to provide metrics for Fluentd. These metrics provide insight into Fluentd’s performance. Please note monitoring for Fluentd is disabled by default, and must be enabled with the following override:
monitoring:
prometheus:
enabled: true
The Fluentd exporter uses the same service annotations as the other exporters, and no additional configuration is required for Prometheus to target the Fluentd exporter for scraping. The Fluentd exporter is configured with command line flags, and the flags’ default values can be found under the following key in the values.yaml file:
conf:
fluentd_exporter:
log:
format: "logger:stdout?json=true"
level: "info"
The configuration keys configure the following behaviors:
log.format: Define the logger used and format of the output
log.level: Log level for the exporter to use
More information about the Fluentd exporter can be found on the exporter’s GitHub page.