I am using haproxy in as a loadbalancer in front of an http application that uses active-passive failover with active health checking. It works very well. Now I want to improve my internal monitoring dashboard, by showing which server is currently active. The dashboard is based on prometheus queries (actually it is grafana).
A few more details on the current setup.
So basically I have a proxy definition that some how looks like this:
balance first
option httpchk GET /health
retries 1
option redispatch
# backend server
default-server check fall 6 rise 4 inter 15s
server backend-001 <ip1>:8080
server backend-002 <ip2>:8080 backup
server backend-003 <ip3>:8080 backup
Sometimes (mainly during deployment) it is changed by an external process to look like this:
balance first
option httpchk GET /health
retries 1
option redispatch
# backend server
default-server check fall 6 rise 4 inter 15s
server backend-002 <ip1>:8080
server backend-001 <ip2>:8080 backup
server backend-003 <ip3>:8080 backup
So we are routing to the new version deployed on the second node first, but in case of failure failover to the old version is an option. As a result relying on a fixed order of server names is not really possible.
The information that I am now looking for is: Which server is the one that haproxy consideres healthy and hence will be served with the next request. The stats page seems somehow to convey that information, because it displays which server is backup and which not and it also knows about the order of the backup servers. But somehow this is all lost when exporting to prometheus or atleast I kind find it.
The best workaround so far is to measure which server actually is getting the requests, but that only shows me the information that I want in the presence of requests.
PS: After writing it down it kind of seems that I only need to know which server is the backup server.