Pages

Monday, April 4, 2022

Photon OS - install fluentd agent for LogInsight

First of all, enable ICMP (ping) to Photon OS
Also allow HTTP connections on port 9323, where docker Prometheus node exporter exposes metrics. 

iptables -A INPUT -p ICMP -j ACCEPT
iptables -A OUTPUT -p ICMP -j ACCEPT
iptables -A INPUT -p tcp --dport 9323 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
iptables -A OUTPUT -p tcp --sport 9323 -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables-save > /etc/systemd/scripts/ip4save

We can continue with Fluentd agent installation.

Installation of Fluentd agent in Photon OS

# this will install Fluentd agent along with Ruby package manager (aka gem) used for other Ruby package installations

tdnf install rubygem-fluentd

# this will install wget to Photon OS to download some other required software components

tdnf install wget

# this will download VMware fluent-plugin-vmware-loginsight output plugin to do forward logs to VMware Log Insight

wget https://github.com/vmware/fluent-plugin-vmware-loginsight/releases/download/v1.0.0/fluent-plugin-vmware-loginsight-1.0.0.gem

# This will install VMware fluent-plugin-vmware-loginsight

gem install fluent-plugin-vmware-loginsight-1.0.0.gem

# This will install Docker fluent-plugin-docker

gem install fluent-plugin-docker

The Fluentd gem does not come with /etc/init.d/ scripts. You should use Process Management tools such as:

  • daemontools
  • runit
  • supervisord
  • upstart
  • systemd
Let's use systemd to manage fluentd as a linux service
See. https://medium.com/@benmorel/creating-a-linux-service-with-systemd-611b5c8b91d6

Fluentd is located at /usr/lib/ruby/gems/2.7.0/bin/fluentd 
Let's create a Linux service (fluentd) with systemd.

vi /etc/systemd/system/fluentd.service with following content
[Unit]
Description=Fluentd service
After=
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=5
User=root
ExecStart= /usr/lib/ruby/gems/2.7.0/bin/fluentd

[Install]
WantedBy=multi-user.target

Now we can use standard systemd (systemctl) procedures to work with service.

systemctl enable fluentd
systemctl start fluentd
systemctl status fluentd

Configuration of Fluentd agent in Photon OS

Setup Fluentd configuration directory

/usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.3/bin/fluentd --setup /etc/fluent

Navigate to Fluentd configuration file (i.e. at /etc/fluent/fluent.conf).

Create the test config file manually into /etc/fluent/test_docker.conf

## built-in TCP input
## $ echo <json> | fluent-cat <tag>
<source>
  @type forward
  @id forward_input
</source> 
 
<match docker>
  @type stdout
  @id stdout_output
</match>

# run fluentd with test config

/usr/lib/ruby/gems/2.7.0/bin/fluentd -c /etc/fluent/test_docker.conf

# We can test logging by following command

docker run -it --log-driver=fluentd --log-opt tag="docker" alpine ash

and you can see log events on standard output

Default log driver and log options can be configured in docker configuration file /etc/docker/daemon.json

{
  "log-driver": "fluentd",
  "log-opts": {
    "tag": "docker",
    "mode": "non-blocking"
  },
  "metrics-addr" : "127.0.0.1:9323",
  "experimental" : true
} 

Metrics-addr is the Prometheus node exporter of Docker.

Restart docker to activate new configuration 

systemctl restart docker

Now you can run docker without --log parameters and still use fluentd log routing.

docker run -it alpine ash

fluent-plugin-docker can be used to check and convert quoted JSON log messages into real JSON format

gem install fluent-plugin-docker

Fluentd configuration file is located in /etc/fluent/fluent.conf and below is the fluentd configuration example:

<source>  
  @type forward  
  @id forward_input  
</source>  
<filter docker>
  @type docker
</filter>
# Match everything else  
<match **>  
  @type copy  
  <store>  
   @type vmware_loginsight  
   @id out_vmw_li  
   scheme https  
   #ssl_verify true  
   ssl_verify false  
   # Loginsight host: One may use IP address or cname  
   host syslog.home.uw.cz  
   port 9543  
   #agent_id XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX  
   # Keys from log event whose values should be added as log message/text to  
   # Loginsight. Note these key/value pairs won't be added as metadata/fields  
   log_text_keys ["log","msg","message","source"]  
   # Use this flag if you want to enable http debug logs  
   http_conn_debug true  
   #http_conn_debug false  
  </store>  
  # copy plugin supports sending/copying logs to multiple plugins  
  # One may choose to send them to multiple LIs  
  # Or one may want send a copy to stdout for debugging  
  # Please note, if you use stdout along with LI, catch the logger's log to make  
  # sure they're not cyclic  
  #<store>  
  # @type stdout  
  #</store>  
</match>

TODO: I still have to find a way how to merge multiline log messages into a single event.

If we want to send logs to two log servers, we can do so to use by two stores.

Let's install Fluentd plugin for Grafana Loki

gem install fluent-plugin-grafana-loki

and add additional <store>...</store> into /etc/fluent/fluent.conf

Here is the additional <store> snippet for loki ...

  <store>  
   @type loki  
   url "https://logs-prod-eu-west-0.grafana.net"  
   username "This is the loki user name"  
   password "For Grafana Cloud ... here should be the API key"  
   flush_interval 10s  
   flush_at_shutdown true  
   buffer_chunk_limit 1m  
   tenant dpasek  
   extra_labels {"worker":"fluentd"}  
   <label>  
    fluentd  
   </label>  
  </store>

For more info about these topics, read the following articles ... 

Docker Logging (with runbook how to test it)
https://www.fluentd.org/guides/recipes/docker-logging



Configure Docker logging drivers
https://docs.docker.com/config/containers/logging/configure/

fluent-plugin-vmware-loginsight
https://github.com/vmware/fluent-plugin-vmware-loginsight

How to produce Prometheus metrics out of Logs using Fluentd

https://www.youtube.com/watch?v=fiqnLA2Qr98



Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)