HAProxy high connection resets using proxy protocol

Help!
1

I’ve been spending the better part of a few days trying to figure this out but have not made much progress and would greatly appreciate some help.

System: Ubuntu Server 16.04.3 LTS, HAProxy 1.7.9, Nginx 1.10.3

Issue: Under load or spike in traffic, around 10k/sec non-ssl or 4k/sec ssl using proxy protocol, I get random connection resets (see image as reported by haproxy).

2017-09-02_8-29-35
2017-09-02_8-29-35.png2259×206 37.1 KB

Reproducing/Troubleshooting: I’m able to reproduce the issue every time load testing using the proxy protocol for example:

  • Sending proxy protocol from HAProxy to Nginx, without SSL, I start getting connection resets at around 10k/sec, all the responses are under 1.5 seconds
  • Without the proxy protocol and the same setup I get no errors at all, to give you an idea at 25k/sec I see no issues
  • If I use “mode http” instead of “mode tcp” but still use the proxy protocol for the HAProxy backend then the issue disappears, however since I plan to terminate SSL at nginx that is not a solution.
  • If I enable SSL (terminated at Nginx) I see the same issue but around 4k/sec and the number of connection resets is much higher around 10%.
  • If I disable proxy protocol but still have SSL (terminated at Nginx) then the issue disappears

My connection timeouts are set to the following, however I don’t think it is related to this as the responses are all under 1.5 seconds.

    timeout connect 10s
    timeout client 50s
    timeout server 50s

Let me know what could help and I will provide it.

2

Can you share the log of those failed requests?

Also, can you try the current haproxy development tree or a development snapshot?