Understanding "show sess show-uri" output

Help!
1

Hello, haproxy users,

I am occasionally having problems with backend servers being busy up to their maxconn, with further requests waiting in the backend queue. But when I reload haproxy, everything goes back to normal for several hours again. So it is not an on-going DDoS attack or any other kind of ongoing traffic peak, just some kind of old “stuck” connections. I tried to display the opened sessions for that backend, but I am not sure how to interpret the show sess show-uri output. In which state the following sessions are, and what are they waiting for?

$ echo show sess show-uri | socat /run/haproxy/...sock stdio | grep be=mybackend | head -n 3
0x7f8e0e332c30: proto=tcpv4 src=162...:39215 fe=fe4 be=mybackend srv=myhost401 ts=00 epoch=0x9 age=1h37m calls=5 rate=0 cpu=0 lat=0 rq\[f=1840000h,i=0,an=8000h,ax=\] rp\[f=80040000h,i=0,an=4000000h,ax=\] scf=\[8,40082h,fd=6881,rex=,wex=22m13s\] scb=\[8,1211h,fd=5734,rex=,wex=\] exp=22m13s rc=0 c_exp= uri=“GET https://.../...pdf HTTP/2.0”
0x7f8e0d31a410: proto=tcpv4 src=31...:50988 fe=fe4 be=mybackend srv=myhost401 ts=00 epoch=0x1f age=17m23s calls=5 rate=0 cpu=0 lat=0 rq\[f=1840000h,i=0,an=8000h,ax=\] rp\[f=80070000h,i=0,an=4000000h,ax=\] scf=\[8,482h,fd=4860,rex=,wex=1h43m\] scb=\[8,1211h,fd=797,rex=,wex=\] exp=1h43m rc=0 c_exp= uri=“GET /pics/....png HTTP/1.1”
0x7f8e0e552d10: proto=tcpv4 src=170...:19493 fe=fe4 be=mybackend srv=myhost401 ts=00 epoch=0x1f age=16m59s calls=5 rate=0 cpu=0 lat=0 rq\[f=1840000h,i=0,an=8000h,ax=\] rp\[f=80070000h,i=0,an=4000000h,ax=\] scf=\[8,482h,fd=5936,rex=,wex=1h44m\] scb=\[8,1211h,fd=2233,rex=,wex=\] exp=1h44m rc=0 c_exp= uri=“GET /...pdf HTTP/1.1”

One possible issue is that for unrelated reasons, the frontend has timeout client 7200s. But still I would like to be able to interpret the above output in order to decide whether the session is waiting for a client, for a backend server response, HTTP keepalive, or whatever else. Is it documented somewhere?

Thanks!

-Yenya

2

Exploring it further: it seems that these requests are logged as finished on backend servers. I use Apache, and the URLs from the show sess show-uri for these long-running requests are not visible in Apache scoreboard (mod_status). So it seems they are waiting for clients, and backend Apache server does not know about them anymore. Such requests should probably not be considered as running on a backend server and definitely should not count against it’s maxconn, as it leads to idle backend servers even though the backend queue is filling up and further requests are being delayed.

It would be useful if it was possible to set timeout client … also in the backend section, overriding the default from the defaults or frontend sections only for connections directed to that backend.

It would also be nice to have some kind of scoreboard page for haproxy, similar to Apache scoredboard, to see what is going on in that particular haproxy instance - i.e. display all connections, their in-flight HTTP requests, and their status/timers/whatever, in a single HTML page.

Thanks!

-Yenya