We are facing one issue whenever client calls to our api via HaProxy then total response time is going beyond 1 sec which is against our SLO metrics.
When we debugged this issue and checked the HaProxy logs then we found out that Tq & Td are high.
Here is the sample log example of our HaProxy box
Feb 11 09:47:11 box-name haproxy: 18.104.22.168:35713 [11/Feb/2022:09:47:10.421] our_service_frontend~ our_service_name/aws_elb_url:port 426/0/0/338/973 200 1214 - - ---- 18/17/13/2/0 0/0 “POST api_url HTTP/1.1”
So, if you see above logs 426 is the Tq value which is high.
Similar way if we calculate Td = Tt - (Tq + Tw + Tc + Tr) = 209 which is again high.
Can you help us here ? What can be the possible issue ?