In the HAProxy stats page, it displays on some of my backends a Queue Time of ~2s. I don’t understand why some requests are being queued despite maxconn being set to 60000 (it never reached that value).
Here is my conf:
Also, strangely, in the stats page, the session limit is set to 1 on all of my backends:
It is expected because maxconn is only relevant for servers and proxies with frontend capabilities.
For backend, this value (on the stats page) reflects the fullconn tunable instead.
Now as for why you sometime see a non-zero Queue time while you didn’t reached the maxconn value: Queue time doesn’t necessarily mean that the connection spent time in frontend or server queue, it is corresponds to the time between the connection was accepted and some input data is being processed. Due to IO activity, this could sometime show some delay. Value should be close to 0 though most of the time, except if the system/network is heavily loaded.
I don’t understand. My backends do have a frontend (nginx) with dynamic use_backend names.
In that case, does that mean there will be only one active session at the time and other requests will be queued ?
System/network is not overloaded at all. What I find strange is that the queue time is always about 2s, like my timeout connect. Are they related somehow ?
No, it means maxconn value will only be displayed for frontends as “limit” on the stats page. For backends this “limit” reflects the fullconn value which is a backend specific tunable.
System/network is not overloaded at all. What I find strange is that the queue time is always about 2s, like my timeout connect. Are they related somehow ?
Oh I missed that part, 2 seconds? Not even ms? In that case it is strange indeed because 2s is huge given that you don’t hit maxconn at all. Is it the case for all servers a just specific one?
So, it means that, like in my screenshot, fullconn is set to only 1 ? So there will be only 1 maximum request processed at a given time ?
2000ms yes. Sometimes 2001ms, always a little bit higher than 2000ms. I wonder why.
It’s impacts all of my servers but since the value “Queue time max” in the stats page is only for the last 1024 requests, it doesn’t appear everywhere everytime.
No, fullconn set to 1 doesn’t mean that 1 maximum request will be processed. In your case I think you can ignore it, as it is relevant when minconn != maxconn, please read the doc for fullconn carefully, it is well explained
2000ms yes. Sometimes 2001ms, always a little bit higher than 2000ms. I wonder why.
It’s impacts all of my servers but since the value “Queue time max” in the stats page is only for the last 1024 requests, it doesn’t appear everywhere everytime.
I don’t understand yet either. Which haproxy version are you using? Could you share a bit more of your config, namely frontend/backend sections? It could help to understand what’s going on
I use HAProxy v2.2.31.
Regarding my config, I already put it in my first post. The only difference is that I have about 30 backends (You can see in my frontend that I use_backend dynamically based on the host header). They are all parameterized the same, like this:
An explicitly odd value to see if it’s related to Queue time and indeed it is !
Now in my stat page I sometime see Queue times roughly equals to 1792ms:
Even if it doesn’t help with this issue, please consider upgrading to a newer LTS because 2.2 is EOL. If that’s not feasible at least update to the last 2.2.33 to benefit from last known bugfixes.
Now back to your issue, I think I was able to reproduce… It when the request is redispatched (due to “option redispatch”), redispatch time is counted under “Queue time” it seems. The maximum value I can see is roughly retries x timeout connect on my end. I assume it is a side-effect but not a bug in itself.
If requests are often redispatched, it may indicate that some servers are having trouble to handle the load, or that the network path between haproxy and some servers is not stable
They are not in the queue, the “Queue time” metric may be misleading but you are wrongly assuming thay they are put in the queue because the “Queue time” metric doesn’t show 0
As explained before, even when maxconn isn’t reached, Queue time may be != 0 if redispatched occurs, in which case queue time accounts for retry/redispatch time, not actual queue time (since the request wasn’t queued).
Try disabling redispatch and see if the metric still shows inconsistent results, or try solving the issue on servers side to prevent redispatch from occuring (they occur in case of failed connection attempts to the servers)
I still don’t get why the “Queue time” metric shows 2s, that’s a lot ! is there a delay between retries ?
You’re right, it might be related to redispatch. I have 9 servers per backend and I see on the stat page that the server which had the redispatch is not the same as the one who had 2s queue time:
