Stats are reset to 0 after haproxy reload

Help!
1

Sorry to bring this topic back.

We are using haproxy for load balancer. We’d like to auto-provision servers based on the average TCP connections to the servers. Unfortunately, we found that stats are reset to 0 when haproxy config is changed and haproxy is reloaded. The TCP connections are still alive.

I have read some threads in the past (see below). It seems the issue still exists. I understand the stats are reset to 0 if haproxy is restarted. But why stats are reset when haproxy is reload and connections are still alive ?

Anyone has good suggestions ? restart might not be a good solution. We are using HAProxy version 2.2.9-2+deb11u3

2

A restart means that a new haproxy instance is started and serves new requests, while the old instance is closed immediately, even though old connections are still connected to that process.

A reload means that a new haproxy instance is started and serves new requests, while the old instance close is delayed, until old connections are naturally closed (or due to hard-stop-after).

So for the stats nothing changes between a restart or a reload: its a new process/instance that starts from zero.

3

@lukastribus
thanks for your quick response.

The only thing we can do on the connections to old haproxy instance, is to wait for them to be closed (by client, server or timeout), right ?

4

I don’t understand the question. What is it that you want to achieve exactly?

5

@lukastribus

what I want to get is, the current sessions to backend servers.

As I understand, when haproxy is reload, there are
(1) alive connections to old haproxy instance.
(2) new requests to new haproxy instance.

The haproxy stats show the counters of (2).
There is no way to get the counters of (1). Those connections will be closed by client, server or timeout, or hard-stop of old haproxy instance.

Please correct me if I am wrong.

6

This is correct.

When you really need to extract data and stats from the old process, then you can access it through the master CLI:

http://docs.haproxy.org/2.6/management.html#9.4.1

But this is for troubleshooting purposes really.

1 Like
7

Thanks a lot @lukastribus

I had a quick testing, it looks show stat in master CLI returns empty result.

# socat /run/haproxy-master.sock -
prompt
master> show proc
#<PID>          <type>          <relative PID>  <reloads>       <uptime>        <version>      
851             master          0               6               4d08h16m26s     2.2.9-2+deb11u3
# workers
1461917         worker          1               0               0d00h10m25s     2.2.9-2+deb11u3
# old workers
1460355         worker          [was: 1]        1               0d00h19m02s     2.2.9-2+deb11u3
# programs

master> @!1460355
1460355> show stat
# pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,chkdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,check_status,check_code,check_duration,hrsp_1xx,hrsp_2xx,hrsp_3xx,hrsp_4xx,hrsp_5xx,hrsp_other,hanafail,req_rate,req_rate_max,req_tot,cli_abrt,srv_abrt,comp_in,comp_out,comp_byp,comp_rsp,lastsess,last_chk,last_agt,qtime,ctime,rtime,ttime,agent_status,agent_code,agent_duration,check_desc,agent_desc,check_rise,check_fall,check_health,agent_rise,agent_fall,agent_health,addr,cookie,mode,algo,conn_rate,conn_rate_max,conn_tot,intercepted,dcon,dses,wrew,connect,reuse,cache_lookups,cache_hits,srv_icur,src_ilim,qtime_max,ctime_max,rtime_max,ttime_max,eint,idle_conn_cur,safe_conn_cur,used_conn_cur,need_conn_est,

1460355>
8

Well, version 2.2.9 contains known 391 bugs, so my first suggestion would be to upgrade to 2.6 LTS:

https://haproxy.debian.net/#distribution=Debian&release=bullseye&version=2.6

If the list is still empty on 2.6.6, file a bug at:

9

@lukastribus

I upgraded to haproxy 2.6.6-1, it looks good

# echo 'show proc' | socat -  /run/haproxy-master.sock
#<PID>          <type>          <reloads>       <uptime>        <version>      
1590727         master          1 [failed: 0]   0d00h08m11s     2.6.6-1~bpo11+1
# workers
1590983         worker          0               0d00h06m29s     2.6.6-1~bpo11+1
# programs


# echo '@!1590983 show stat' | socat -  /run/haproxy-master.sock | grep ^10_0_55_192-22,10_0_55_192-22 | cut -d "," -f5
2


# echo 'reload' | socat -  /run/haproxy-master.sock

# echo 'show proc' | socat -  /run/haproxy-master.sock
#<PID>          <type>          <reloads>       <uptime>        <version>      
1590727         master          2 [failed: 0]   0d00h08m28s     2.6.6-1~bpo11+1
# workers
1592078         worker          0               0d00h00m03s     2.6.6-1~bpo11+1
# old workers
1590983         worker          1               0d00h06m46s     2.6.6-1~bpo11+1
# programs


# echo '@!1590983 show stat' | socat -  /run/haproxy-master.sock | grep ^10_0_55_192-22,10_0_55_192-22 | cut -d "," -f5
2

Thanks @lukastribus

Hope this thread can help who want to get accurate counters after haproxy reload.

1 Like