HAProxy multiproc metrics

Help!
1

This post was originally a reply to this, but I feel this needs a bit more attention that it has received until then:

I’m trying to do some aggregation between processes

If you run HAProxy in multiprocess mode, your HTTP requests are probably load-balanced between the different processes. The stats are not shared between processes. So the value will differ between processes. Try to read the value of haproxy_process_relative_process_id to know where a request was processed.

So far, I always used this pattern to collect and measure:

    cpu-map 1 0
    cpu-map 2 1
    cpu-map 3 2
    cpu-map 4 3
    cpu-map 5 4
    cpu-map 6 5
    cpu-map 7 6
    cpu-map 8 7
    stats socket /run/p1.sock mode 660 process 1
    stats socket /run/p2.sock mode 660 process 2
    stats socket /run/p3.sock mode 660 process 3
    stats socket /run/p4.sock mode 660 process 4
    stats socket /run/p5.sock mode 660 process 5
    stats socket /run/p6.sock mode 660 process 6
    stats socket /run/p7.sock mode 660 process 7
    stats socket /run/p8.sock mode 660 process 8

It is quite clear that each proc has a stats socket.

In a simplified example regarding only proc 1:

frontend stats_prom_proc_1
   bind *:8404 interface eth1
   option http-use-htx
   http-request use-service prometheus-exporter if { path /metrics }
   stats enable
   stats uri /stats level admin process 1
   stats refresh 10s

I can bind my /stats uri to a proc but not my prometheus-exporter service.

I can’t see how I could handle an aggregation on those time series, and use-service is not documented. My current exporter scrapes all the sockets and exposes an aggregated value to prometheus’ scrape.

2

Hi,

In you example, the stats applet is not bound in the process 1. The option process is not supported here. options for the stats socket and the stats applet are not the same. The option process is only supported for the stats socket. So, in your example, a request on the uri /stats always returns the stats page, regardless the process processing it. There is no error during the startup because the config parser is not enough strict on this case. Everything after the uri is ignored. stats uri /stats foo bar will pass.

By the way, binding the stats page or prometheus metrics on a subset of processes with no dedicated port is risky and a bit strange IMHO. Because regularly some requests will not be handled as expected (an error will be returned or the request will be sent to your backends, depending on your configuration). And no aggregation is performed, so only stats of the current process will be returned. The problem is the same than for the stats socket. To aggregate the stats, you must do it on your own. You must bind X ports, each one on a process. For instance:

frontend stats_prom
   bind *:PORT1 interface eth1 process 1
   bind *:PORT2 interface eth1 process 2
   ...
   option http-use-htx
   http-request use-service prometheus-exporter if { path /metrics }
   stats enable
   stats uri /stats
   stats refresh 10s

But it is probably easier to use threads instead of multiproc.

3

Thanks for the hint, I’ll try it out :slight_smile: