The amount it takes for the health check to complete. What that health check does, depends on how you configured it.
For example, if you access a specific URL on the server for the health check and to return the response the servers make some internal checks (like connecting to a database), then all those steps add to that value (as the health check waits for a response).
Make the same exact call externally, and it should yield the same results.
By externally I mean not with haproxy, but with an external programm like curl for example.
If you don’t have any httpchecks configured at all, haproxy will only establish the TCP connection (3-way handshake). If this already takes a long time, then there is probably something going on at TCP level. How do the Tc counters in your logs look like?