HAProxy 504 responses when backends sent 200 responses

Hello

Can someone offer some insight on this, I have been pulling my hear out for a week over it.

HAProxy v2.9.7 (and 2.3) appears to be losing some requests, resulting in 504 errors due to the timeout server limit.

Example:
ab -r -k -c 100 -n 10000 -H “Accept-Encoding: gzip,deflate” ‘http://mysite.com/some.file.js
apr_pollset_poll: The timeout specified has expired (70007)
Total of 9995 requests completed

The backend servers show 10,000 requests and responses with HTTP 200 status and a latency of ~0.2 seconds.
122 seconds after the test, haproxy logs show 9,995 requests and responses with HTTP 200 status and a latency of ~0.2 seconds, and 5 requests with HTTP 504 status and ~122 sec latency.

The same ab test sent directly to the backend server can hadnle 5,000 rps correctly.
During this test, the load on the servers is really low.

Any insights on what I should be looking for?