I have an application where I’m using HAProxy (1.5.14) to route traffic to different backends. HAProxy terminates the incoming SSL and then re-encrypts to the backend. There seems to be a large number of key exchanges, which is limiting the performance. In a short test (2 min), I sent 50k requests through HAProxy. I observed over 1k key exchanges. Looking further into the packets, it seems that HAProxy is closing the connections to the backends [RST, ACK]. When HAProxy attempts to reuse the session ID after this, the backend service replies with a new session ID. It appears that closing the connection invalidates the session ID on the backend. I believe that I have configured HAProxy correctly with option http-keep-alive.
When does HAProxy close a connection to the backend?
Is there a way to increase the SSL session reuse in the backend?
Haproxy closes when the keep-alive timeout strikes, or when an error is emitted (I believe there was some discussion about this on the mailing list some time ago).
My suggestion would be to upgrade to the latest haproxy 1.6 stable release and enabled http-reuse.
Thanks for responding! I looked into this further, and resets also occur when I go directly to the data service. However, the SSL sessions can be reused many times when going directly. Below is output from Wireshark. It is filtered by SSL session ID.
Wireshark - HAProxy
ServerHello_1 is where we first receive the SSL session ID. It is successfully reused in ClientHello_2. However, the backend service returns a new SSL session ID in response to ClientHello_3. Between 2 and 3, there is a reset of the connection where the SSL session ID was first established.
ServerHello_1: 9583 47.588186 IP_addr_backend IP_addr_haproxy TCP 2976 9466 → 46291 [ACK] Seq=1 Ack=134 Win=15544 Len=2920
ClientHello_2: 9611 47.599730 IP_addr_haproxy IP_addr_backend TCP 189 46295 → 9466 [PSH, ACK] Seq=1 Ack=1 Win=14600 Len=133
ServerHello_2: 9616 47.600332 IP_addr_backend IP_addr_haproxy TCP 142 9466 → 46295 [PSH, ACK] Seq=1 Ack=134 Win=15544 Len=86
Reset: 9668 47.606401 IP_addr_haproxy IP_addr_backend TCP 62 46295 → 9466 [RST, ACK] Seq=525 Ack=377 Win=15544 Len=0
Reset: 9674 47.609673 IP_addr_haproxy IP_addr_backend TCP 62 46291 → 9466 [RST, ACK] Seq=792 Ack=6025 Win=29200 Len=0
ClientHello_3: 9721 47.629867 IP_addr_haproxy IP_addr_backend TCP 189 46299 → 9466 [PSH, ACK] Seq=1 Ack=1 Win=14600 Len=133
=====
Wireshark - Direct
The SSL session ID is returned in ServerHello_1. It is successfully reused in ClientHello_2-4. There are resets between each request.
ServerHello_1: 1450 30.426777 IP_addr_backend IP_addr_testsrv TCP 2976 9466 → 34462 [ACK] Seq=1 Ack=205 Win=15544 Len=2920
Reset: 1626 30.503922 IP_addr_testsrv IP_addr_backend TCP 62 34462 → 9466 [RST] Seq=685 Win=0 Len=0
ClientHello_2: 1776 30.530303 IP_addr_testsrv IP_addr_backend TCP 292 34466 → 9466 [PSH, ACK] Seq=1 Ack=1 Win=14600 Len=236
ServerHello_2: 1780 30.530542 IP_addr_backend IP_addr_testsrv TCP 142 9466 → 34466 [PSH, ACK] Seq=1 Ack=237 Win=15544 Len=86
Reset: 1885 30.544821 IP_addr_testsrv IP_addr_backend TCP 62 34466 → 9466 [RST, ACK] Seq=642 Ack=433 Win=15544 Len=0
ClientHello_3: 1993 30.559608 IP_addr_testsrv IP_addr_backend TCP 292 34474 → 9466 [PSH, ACK] Seq=1 Ack=1 Win=14600 Len=236
ServerHello_3: 1996 30.559794 IP_addr_backend IP_addr_testsrv TCP 142 9466 → 34474 [PSH, ACK] Seq=1 Ack=237 Win=15544 Len=86
Reset: 2086 30.572950 IP_addr_testsrv IP_addr_backend TCP 62 34474 → 9466 [RST, ACK] Seq=642 Ack=433 Win=15544 Len=0
ClientHello_4: 2216 30.593426 IP_addr_testsrv IP_addr_backend TCP 292 34480 → 9466 [PSH, ACK] Seq=1 Ack=1 Win=14600 Len=236
ServerHello_4: 2218 30.593611 IP_addr_backend IP_addr_testsrv TCP 142 9466 → 34480 [PSH, ACK] Seq=1 Ack=237 Win=15544 Len=86
It appears that HAProxy does not send a close_notify alert. According to the TLS1.0 spec, if a connection is closed without a close_notify alert being sent, then the TLS session is not resumable. This requirement was removed in TLS1.1.
I configured HAProxy to use TLS1.2. However, this still did not work. I suspected that Jetty was also causing an issue, so I installed HAProxy in front of Jetty to terminate the TLS. This combination eliminated the key exchanges. The TLS sessions are successfully reused.
Terminating TLS on haproxy and not encrypting towards your backend is a better idea for a lot of reason anyway (like being able to intercept layer 7).