HAProxy community

Grafana / local telegraf agent: Could not connect to socket '/var/lib/haproxy/stats'

Hi

I am using haproxy 2.1.4 on CentOS 7 and would like to get observability through grafana.

I’m using a local telegraf agent that’s supposed to collect haproxy stats and haproxy logs.
haproxy is configured to run in a chroot jail, and it creates a stats socket file in /var/lib/haproxy/stats.
Grafana’s local telegraf agent runs as user “telegraf” and is configured to get haproxy stats from the socket file in /var/lib/haproxy/stats.
In the messages log, we can see that it fails to connect to the socket:

May 31 03:12:51 haproxyhost telegraf: 2020-05-31T08:12:51Z E! [inputs.haproxy] Error in plugin: Could not connect to socket '/var/lib/haproxy/stats': dial unix /var/lib/haproxy/stats: connect: permission denied

It seems that telegraf tries to send commands to the haproxy stats socket in order to actively fetch stats, hence the telegraf agent requires “write” access to the socket file.

haproxy seems to create the stats socket file with the following permissions:
# ls -l /var/lib/haproxy/stats srwxr-xr-x. 1 root root 0 May 31 23:40 /var/lib/haproxy/stats

When I manually change the socket owenership/permissions and add “telegraf” to the group “haproxy”, the error goes away:
# ls -l /var/lib/haproxy/stats srwxrwxr-x. 1 haproxy haproxy 0 May 31 23:40 /var/lib/haproxy/stats

However, after the next reboot, the modified ownership/permissions are restored to the old values.

How can I get telegraf working with haproxy, without running telegraf as root?

Here is an excerpt from my haproxy.cfg:
global
log /dev/log local0 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats level admin

I figured out how to control ownership and chmod on the stats socket file.
The configuration needs to be changed to the following:

stats socket /var/lib/haproxy/stats mode 660 group haproxy level admin

On the file system, the stats socket file will then look like this:
# ls -l /var/lib/haproxy/stats
srw-rw----. 1 root haproxy 0 Jun 4 17:07 /var/lib/haproxy/stats

Telegraf does not need to run as root anymore after this change.
However, “telegraf” needs to be added to the group “haproxy”.