Many (tcp-retransmissions) when using http mode & ssl-offloading

Help!
1

Hello HAProxy community,

We are experiencing a high number of TCP retransmissions in an environment where HAProxy is used, and we are trying to understand the root cause.

Traffic Flow:

Client → MikroTik router (NAT + firewall)
MikroTik → HAProxy (SSL offloading)
HAProxy → IIS web server (HTTP)

HAProxy terminates TLS and forwards plain HTTP to IIS

We observe frequent TCP-Retransmissions (visible in packet captures and monitoring tools). These retransmissions occur regularly and give the impression that the client application is unstable or slow, even though no obvious application-level errors are logged.

Current HAProxy Timers:

timeout connect         5s
timeout client          1m
timeout server          1m
timeout check           5s
timeout http-request    5s
timeout http-keep-alive 5s

Current Firewall/NAT Timers:

tcp-syn-sent-timeout: 5s    
tcp-syn-received-timeout: 5s    
tcp-established-timeout: 1h    
tcp-fin-wait-timeout: 2m    
tcp-close-wait-timeout: 2m    
tcp-last-ack-timeout: 1m    
tcp-time-wait-timeout: 1m    
tcp-close-timeout: 1m    
tcp-max-retrans-timeout: 5m    
tcp-unacked-timeout: 5m

Wireshark Traffic Flow from MikroTik router:

Please sanitize data like external ip addresses and domains;
No. Time Source Destination Protocol Length Info
23 0.236448 10.31.1.220 EXTERNAL_IP_1 TLSv1.2 354 Application Data, Application Data
105 0.748442 10.31.1.220 EXTERNAL_IP_1 TLSv1.2 354 Application Data, Application Data
296 3.861400 10.31.1.220 EXTERNAL_IP_1 TLSv1.2 330 Application Data
297 3.861525 10.31.1.220 EXTERNAL_IP_1 TLSv1.2 137 Application Data
298 3.861525 10.31.1.220 EXTERNAL_IP_1 TCP 113 2443 → 65225 [FIN, ACK] Seq=242 Ack=1
299 3.880356 10.31.1.220 EXTERNAL_IP_1 TCP 113 [TCP Retransmission] 2443 → 65225 [FIN, ACK]
310 4.096361 10.31.1.220 EXTERNAL_IP_1 TCP 354 [TCP Retransmission] 2443 → 65225 [FIN, PSH, ACK]
339 4.524310 10.31.1.220 EXTERNAL_IP_1 TCP 354 [TCP Retransmission] 2443 → 65225 [FIN, PSH, ACK]
373 4.752099 EXTERNAL_IP_1 10.31.1.220 TLSv1.2 137 Application Data
374 4.752460 10.31.1.220 EXTERNAL_IP_1 TCP 113 2443 → 22247 [ACK]
375 4.753593 EXTERNAL_IP_1 10.31.1.220 TCP 113 22247 → 2443 [FIN, ACK]
376 4.786065 EXTERNAL_IP_1 10.31.1.220 TCP 121 60061 → 2443 [SYN]
377 4.786212 10.31.1.220 EXTERNAL_IP_1 TCP 121 2443 → 60061 [SYN, ACK]
378 4.791849 EXTERNAL_IP_1 10.31.1.220 TCP 113 60061 → 2443 [ACK]
379 4.796377 10.31.1.220 EXTERNAL_IP_1 TCP 113 2443 → 22247 [ACK]
380 4.802130 EXTERNAL_IP_1 10.31.1.220 TLSv1.3 630 Client Hello (SNI=example.com)
382 4.802821 10.31.1.220 EXTERNAL_IP_1 TLSv1.3 533 Server Hello, Change Cipher Spec, Application Data
384 4.802919 10.31.1.220 EXTERNAL_IP_1 TCP 533 [TCP Previous segment not captured]
387 4.809389 EXTERNAL_IP_1 10.31.1.220 TCP 113 [TCP ACKed unseen segment]
388 4.809427 EXTERNAL_IP_1 10.31.1.220 TCP 113 [TCP ACKed unseen segment]
389 4.809427 EXTERNAL_IP_1 10.31.1.220 TCP 113 [TCP ACKed unseen segment]
398 4.927203 EXTERNAL_IP_1 10.31.1.220 TLSv1.3 177 Change Cipher Spec, Application Data
399 4.927461 10.31.1.220 EXTERNAL_IP_1 TLSv1.3 400 Application Data
400 4.927547 10.31.1.220 EXTERNAL_IP_1 TLSv1.3 400 Application Data
401 4.933028 EXTERNAL_IP_1 10.31.1.220 TLSv1.3 457 Application Data
403 4.976293 10.31.1.220 EXTERNAL_IP_1 TCP 113 2443 → 60061 [ACK]
404 4.977098 EXTERNAL_IP_1 10.31.1.220 TCP 113 60061 → 2443 [ACK]
456 5.388278 10.31.1.220 EXTERNAL_IP_1 TCP 354 [TCP Retransmission]
603 6.896288 10.31.1.220 EXTERNAL_IP_1 TLSv1.2 354 Application Data, Application Data
611 7.088215 10.31.1.220 EXTERNAL_IP_1 TCP 354 [TCP Retransmission]
617 7.152301 10.31.1.220 EXTERNAL_IP_1 TCP 354 [TCP Retransmission]
895 10.476148 10.31.1.220 EXTERNAL_IP_1 TCP 354 [TCP Retransmission]
942 10.988228 10.31.1.220 EXTERNAL_IP_1 TLSv1.2 354 Application Data, Application Data
1112 14.029699 10.31.1.220 EXTERNAL_IP_1 TLSv1.2 330 Application Data
1113 14.029757 10.31.1.220 EXTERNAL_IP_1 TLSv1.2 137 Application Data
1114 14.029765 10.31.1.220 EXTERNAL_IP_1 TCP 113 2443 → 21707 [FIN, ACK]
1116 14.060066 10.31.1.220 EXTERNAL_IP_1 TCP 113 [TCP Retransmission]
1169 14.280069 10.31.1.220 EXTERNAL_IP_1 TCP 354 [TCP Retransmission]
1237 14.732047 10.31.1.220 EXTERNAL_IP_1 TCP 354 [TCP Retransmission]
1250 14.931297 EXTERNAL_IP_1 10.31.1.220 TLSv1.3 137 Application Data
1251 14.931452 10.31.1.220 EXTERNAL_IP_1 TCP 113 2443 → 60061 [ACK]
1252 14.934154 EXTERNAL_IP_1 10.31.1.220 TCP 113 60061 → 2443 [FIN, ACK]
1254 14.981241 10.31.1.220 EXTERNAL_IP_1 TCP 113 2443 → 60061 [ACK]
1255 14.981280 EXTERNAL_IP_1 10.31.1.220 TCP 121 53387 → 2443 [SYN]
1256 14.981315 10.31.1.220 EXTERNAL_IP_1 TCP 121 2443 → 53387 [SYN, ACK]
1257 14.989238 EXTERNAL_IP_1 10.31.1.220 TCP 113 53387 → 2443 [ACK]
1261 14.998012 EXTERNAL_IP_1 10.31.1.220 TLSv1.3 630 Client Hello (SNI=example.com)
1262 14.998464 10.31.1.220 EXTERNAL_IP_1 TCP 1329 [TCP Previous segment not captured]
1264 14.998478 10.31.1.220 EXTERNAL_IP_1 TLSv1.3 533 Server Hello, Change Cipher Spec, Application Data
1266 14.998797 10.31.1.220 EXTERNAL_IP_1 TCP 533 [PSH, ACK]
1267 14.998886 10.31.1.220 EXTERNAL_IP_1 TLSv1.3 207 Continuation Data
1269 15.006819 EXTERNAL_IP_1 10.31.1.220 TCP 125 [TCP Dup ACK]
1270 15.006819 EXTERNAL_IP_1 10.31.1.220 TCP 113 [ACK]
1271 15.006819 EXTERNAL_IP_1 10.31.1.220 TCP 113 [ACK]
1272 15.006819 EXTERNAL_IP_1 10.31.1.220 TCP 113 [TCP ACKed unseen segment]
1273 15.006819 EXTERNAL_IP_1 10.31.1.220 TCP 113 [ACK]
1281 15.139289 EXTERNAL_IP_1 10.31.1.220 TLSv1.3 177 Change Cipher Spec, Application Data
1282 15.139327 10.31.1.220 EXTERNAL_IP_1 TLSv1.3 400 Application Data
1283 15.139528 10.31.1.220 EXTERNAL_IP_1 TLSv1.3 400 Application Data
1284 15.143605 EXTERNAL_IP_1 10.31.1.220 TCP 113 [ACK]
1285 15.143605 EXTERNAL_IP_1 10.31.1.220 TCP 113 [ACK]
1286 15.149948 EXTERNAL_IP_1 10.31.1.220 TLSv1.3 457 Application Data
1287 15.196016 10.31.1.220 EXTERNAL_IP_1 TCP 113 [ACK]
1501 15.628015 10.31.1.220 EXTERNAL_IP_1 TCP 354 [TCP Retransmission]

Any guidance, best practices, or similar experiences would be greatly appreciated.

2

Retransmissions are pretty much normal, they also have no relation to haproxy. Haproxy is a user-space application, reading and writing to sockets. It is the kernel that handles everything TCP related including congestion control, packet loss and retransmissions.

3

lukastribus, thank you for your explanation! Its clear that retransmissions are handled by the kernel and not directly by HAProxy. However, we notice that these issues only occur when we enable HTTP mode and not in TCP mode. Could it be that HTTP mode introduces additional overhead or buffering that makes this effect?

4

TCP mode and HTTP mode are doing different things, they can’t be compared with each other.

Likely in TCP mode the connection to the browsers remains HTTP/1.1, while in HTTP mode it is upgraded to H2, meaning in H2 mode a single TCP connection will be able to serve multiple concurrent HTTP transactions will in H1 / TCP mode you will have one TCP connection per inflight transaction.

Browser and TCP stacks will react completely differently in those two configurations.

I have yet to hear an actual problem, other than the number of TCP retransmissions. Is there are an actual customer reported problem?

5

The application becomes unstable as soon as HAProxy is running in HTTP mode. That is the actual issue we are facing.To rule out HTTP/2-related behavior, we explicitly forced alpn http/1.1 on the frontend, but unfortunately this did not improve the situation — the problem still occurs.

Concretely, we observe abnormal behavior in Wireshark whenever HTTP mode is enabled, and in addition we see SSL handshake failures in the HAProxy logs. When running in TCP mode, this behavior does not occur and the application remains stable. Please let me know if you need specific logs, packet captures, or configuration details to investigate this further.