Grafana / local telegraf agent: Could not connect to socket '/var/lib/haproxy/stats'

Help!
1

Hi

I am using haproxy 2.1.4 on CentOS 7 and would like to get observability through grafana.

I’m using a local telegraf agent that’s supposed to collect haproxy stats and haproxy logs.
haproxy is configured to run in a chroot jail, and it creates a stats socket file in /var/lib/haproxy/stats.
Grafana’s local telegraf agent runs as user “telegraf” and is configured to get haproxy stats from the socket file in /var/lib/haproxy/stats.
In the messages log, we can see that it fails to connect to the socket:

May 31 03:12:51 haproxyhost telegraf: 2020-05-31T08:12:51Z E! [inputs.haproxy] Error in plugin: Could not connect to socket '/var/lib/haproxy/stats': dial unix /var/lib/haproxy/stats: connect: permission denied

It seems that telegraf tries to send commands to the haproxy stats socket in order to actively fetch stats, hence the telegraf agent requires “write” access to the socket file.

haproxy seems to create the stats socket file with the following permissions:
# ls -l /var/lib/haproxy/stats srwxr-xr-x. 1 root root 0 May 31 23:40 /var/lib/haproxy/stats

When I manually change the socket owenership/permissions and add “telegraf” to the group “haproxy”, the error goes away:
# ls -l /var/lib/haproxy/stats srwxrwxr-x. 1 haproxy haproxy 0 May 31 23:40 /var/lib/haproxy/stats

However, after the next reboot, the modified ownership/permissions are restored to the old values.

How can I get telegraf working with haproxy, without running telegraf as root?

Here is an excerpt from my haproxy.cfg:
global
log /dev/log local0 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats level admin

2

I figured out how to control ownership and chmod on the stats socket file.
The configuration needs to be changed to the following:

stats socket /var/lib/haproxy/stats mode 660 group haproxy level admin

On the file system, the stats socket file will then look like this:
# ls -l /var/lib/haproxy/stats
srw-rw----. 1 root haproxy 0 Jun 4 17:07 /var/lib/haproxy/stats

Telegraf does not need to run as root anymore after this change.
However, “telegraf” needs to be added to the group “haproxy”.

3

hi i got the same error and i tried follow your steps but without any results some help please !

4
  1. add user “telegraf” to group “haproxy” (in /etc/group)
  2. use the line “stats socket ...” as mentioned above in the haproxy.cfg
  3. restart haproxy

As a result, you should see the above permissions when you run ls -l on the sockets file.

But there are also other ways to solve it, e.g. you could also configure the telegraf agent to run as root.

If it doesn’t work, you may want to first try run telegraf as root. As soon as it works, you could then as a next step try to enhance the security by using this solution.

Hope this helps?

5

i already tried the first solution but it does’nt worked !
how can i configure telegraf agent to run as root on centos7?

6

There is a configuration file for the telegraf agent:
/etc/telegraf/telegraf.conf

In that configuration file, you need to configure the telegraf agent to bind to the haproxy socket file:

[[inputs.haproxy]]
  servers = ["socket:/var/lib/haproxy/stats"]

Or whatever your haproxy socket file name is.

Remember, in more recent versions of haproxy, you can configure haproxy to bind to multiple sockets files simultaneously, and each can have different permissions configured.
So, you could create a dedicated haproxy stats socket just for telegraf, if you wanted to.

You can learn more about the haproxy telegraf plugin by running:

telegraf --usage haproxy

or more generally about telegraf:

telegraf --help

If you have questions about the basic configuration of telegraf, you should try to reach out to the telegraf community, as this is the haproxy community.

1 Like