First of all, enable ICMP (ping) to Photon OS
Also allow HTTP connections on port 9323, where docker Prometheus node exporter exposes metrics.
iptables -A INPUT -p ICMP -j ACCEPT
iptables -A OUTPUT -p ICMP -j ACCEPT
iptables -A INPUT -p tcp --dport 9323 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp --sport 9323 -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables-save > /etc/systemd/scripts/ip4save
We can continue with Fluentd agent installation.
Installation of Fluentd agent in Photon OS
# this will install Fluentd agent along with Ruby package manager (aka gem) used for other Ruby package installations
tdnf install rubygem-fluentd
# this will install wget to Photon OS to download some other required software components
tdnf install wget
# this will download VMware fluent-plugin-vmware-loginsight output plugin to do forward logs to VMware Log Insight
wget https://github.com/vmware/fluent-plugin-vmware-loginsight/releases/download/v1.0.0/fluent-plugin-vmware-loginsight-1.0.0.gem
# This will install VMware fluent-plugin-vmware-loginsight
gem install fluent-plugin-vmware-loginsight-1.0.0.gem
# This will install Docker fluent-plugin-docker
gem install fluent-plugin-docker
The Fluentd gem does not come with /etc/init.d/ scripts. You should use Process Management tools such as:
- daemontools
- runit
- supervisord
- upstart
- systemd
Let's use systemd to manage fluentd as a linux service
See. https://medium.com/@benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6
Fluentd is located at /usr/lib/ruby/gems/2.7.0/bin/fluentd
Let's create a Linux service (fluentd) with systemd.
vi /etc/systemd/system/fluentd.service with following content
[Unit]
Description=Fluentd service
After=
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=5
User=root
ExecStart= /usr/lib/ruby/gems/2.7.0/bin/fluentd
[Install]
WantedBy=multi-user.target
Now we can use standard systemd (systemctl) procedures to work with service.
systemctl enable fluentd
systemctl start fluentd
systemctl status fluentd
Configuration of Fluentd agent in Photon OS
Setup Fluentd configuration directory
/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.3/bin/fluentd --setup /etc/fluent
Navigate to Fluentd configuration file (i.e. at /etc/fluent/fluent.conf).
Create the test config file manually into /etc/fluent/test_docker.conf
## built-in TCP input
## $ echo <json> | fluent-cat <tag>
<source>
@type forward
@id forward_input
</source>
<match docker>
@type stdout
@id stdout_output
</match>
# run fluentd with test config
/usr/lib/ruby/gems/2.7.0/bin/fluentd -c /etc/fluent/test_docker.conf
# We can test logging by following command
docker run -it --log-driver=fluentd --log-opt tag="docker" alpine ash
and you can see log events on standard output
Default log driver and log options can be configured in docker configuration file /etc/docker/daemon.json
{
"log-driver": "fluentd",
"log-opts": {
"tag": "docker",
"mode": "non-blocking"
},
"metrics-addr" : "127.0.0.1:9323",
"experimental" : true
}
Metrics-addr is the Prometheus node exporter of Docker.
Restart docker to activate new configuration
systemctl restart docker
Now you can run docker without --log parameters and still use fluentd log routing.
docker run -it alpine ash
fluent-plugin-docker can be used to check and convert quoted JSON log messages into real JSON format
gem install fluent-plugin-docker
Fluentd configuration file is located in /etc/fluent/fluent.conf and below is the fluentd configuration example:
<source>
@type forward
@id forward_input
</source>
<filter docker>
@type docker
</filter>
# Match everything else
<match **>
@type copy
<store>
@type vmware_loginsight
@id out_vmw_li
scheme https
#ssl_verify true
ssl_verify false
# Loginsight host: One may use IP address or cname
host syslog.home.uw.cz
port 9543
#agent_id XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
# Keys from log event whose values should be added as log message/text to
# Loginsight. Note these key/value pairs won't be added as metadata/fields
log_text_keys ["log","msg","message","source"]
# Use this flag if you want to enable http debug logs
http_conn_debug true
#http_conn_debug false
</store>
# copy plugin supports sending/copying logs to multiple plugins
# One may choose to send them to multiple LIs
# Or one may want send a copy to stdout for debugging
# Please note, if you use stdout along with LI, catch the logger's log to make
# sure they're not cyclic
#<store>
# @type stdout
#</store>
</match>
TODO: I still have to find a way how to merge multiline log messages into a single event.
If we want to send logs to two log servers, we can do so to use by two stores.
Let's install Fluentd plugin for Grafana Loki
gem install fluent-plugin-grafana-loki
and add additional <store>...</store> into /etc/fluent/fluent.conf
Here is the additional <store> snippet for loki ...
<store>
@type loki
url "https://logs-prod-eu-west-0.grafana.net"
username "This is the loki user name"
password "For Grafana Cloud ... here should be the API key"
flush_interval 10s
flush_at_shutdown true
buffer_chunk_limit 1m
tenant dpasek
extra_labels {"worker":"fluentd"}
<label>
fluentd
</label>
</store>
For more info about these topics, read the following articles ...
Docker Logging (with runbook how to test it)
https://www.fluentd.org/guides/recipes/docker-logging
