I’ve been spending the better part of a few days trying to figure this out but have not made much progress and would greatly appreciate some help.
System: Ubuntu Server 16.04.3 LTS, HAProxy 1.7.9, Nginx 1.10.3
Issue: Under load or spike in traffic, around 10k/sec non-ssl or 4k/sec ssl using proxy protocol, I get random connection resets (see image as reported by haproxy).
Reproducing/Troubleshooting: I’m able to reproduce the issue every time load testing using the proxy protocol for example:
- Sending proxy protocol from HAProxy to Nginx, without SSL, I start getting connection resets at around 10k/sec, all the responses are under 1.5 seconds
- Without the proxy protocol and the same setup I get no errors at all, to give you an idea at 25k/sec I see no issues
- If I use “mode http” instead of “mode tcp” but still use the proxy protocol for the HAProxy backend then the issue disappears, however since I plan to terminate SSL at nginx that is not a solution.
- If I enable SSL (terminated at Nginx) I see the same issue but around 4k/sec and the number of connection resets is much higher around 10%.
- If I disable proxy protocol but still have SSL (terminated at Nginx) then the issue disappears
My connection timeouts are set to the following, however I don’t think it is related to this as the responses are all under 1.5 seconds.
timeout connect 10s
timeout client 50s
timeout server 50s
Let me know what could help and I will provide it.