We are seeing up to 10s values in %Tw field and +1 in %rc (retry) field.
How can we debug this?
By checking established connections to backend via netstat?
Any stats to look from from HAProxy?
%ac values vary from 400 to 1600 for such requests.
HAProxy version 2.0.14
HAProxy is configured to retry on empty response and connection failures and only for GETs
maxconn is set to 6000 in global and frontend section. Is it possible that HAProxy is retrying because it is not getting a connection from the pool rather than getting a connection error?
Connection timeout is 5s, but the wait times are sometimes more than 10s.
max retries are set to 2.
There is a backend that shows ~1400 active connections, but never shows any queue wait times.
Other backends show hardly any active connections but run into high wait times.
If %Tw is high, %Tc is zero and retry shows a redispatch “+1”, does it mean that request is stuck in HAProxy queue trying to get a connection slot because there are too many backend connections?
But backend connection count is hardly showing anything.
Or is it that connection could not be established hence HAProxy redispatched to another server hence +1 in “retries” column, but why high connection time is not reported?
maxconn in our setup is 6000 and nbproc is 4.
We don’t set minconn or fullconn
Output from “show info” command
Name: HAProxy
Version: 2.0.17
Release_date: 2020/07/31
Nbthread: 1
Nbproc: 4
Process_num: 4
Pid: 9023
Uptime: 18d 22h40m29s
Uptime_sec: 1636829
Memmax_MB: 0
PoolAlloc_MB: 37
PoolUsed_MB: 32
PoolFailed: 0
Ulimit-n: 12173
Maxsock: 12173
Maxconn: 6000
Hard_maxconn: 6000
CurrConns: 1678
CumConns: 323171
CumReq: 174181261
MaxSslConns: 0
CurrSslConns: 3361
CumSslConns: 64186823