Is there a way to figure out how many requests are being refused due to the high traffic and exceeding number of requests being issued to a single HAProxy instance?
Refused at what level?
Great question. I’d like a way to figure out the level too (specially, for L4 and L7).
That depends on your configuration and what particular choke-points you will hit a the various levels.
Logs and stats are the first places to look at. The kernel can also drop connections if the limits are exceeded (syn backlog for example).
We want to determine the correct number of haproxy needed, for example, if we are receiving 1.5K requests per second and we are being able to respond to 1K and refusing 500 requests per second, so we need to increase the number of instances in 50%.
Is it possible to determine this number of refused requests otherwise than through trial and error?
Probably not, especially because you can never know if you really are considering all choke points, if you forgot some, or if some choke points are smaller than you thought.
There is no alternative for benchmarking end-to-end application performance, if you want to be sure of it.
I’m talking about everything here, not only choke points within haproxy userspace application. Your network, stateful firewalls, hosts, hypervisor, all the above on every level can drop packets or introduce unexpected latency or otherwise degrade performance.