Unexpected Frontend Connection Closures by HAProxy

Help!
1

HAProxy is occasionally closing frontend connections prematurely, and I can’t determine why that is.

Frontend connections are configured with a timeout of 90 seconds.

However, in some cases HAProxy is terminating keep-alive connections immediately after received a request. It’s not sending back a well-formed response. Instead, HAProxy is simply closing the TCP connection with a FIN packet.

In the packet captures below, notice that HAProxy immediately responds with a FIN packet after the last request is received.

HAProxy version 2.1.3.

Configuration:

global
    daemon
    pidfile /opt/ec/services/iam-haproxy/bin/haproxy.pid
    log 127.0.0.1:514 syslog
    stats socket ipv4@*:9999 level admin
    stats timeout 30s

frontend http-in
    mode http
    maxconn 4000
    timeout client 90s
    option httplog
    monitor-uri /health
    log global
    stats enable
    stats hide-version
    stats refresh 5s
    stats show-node
    stats scope .
    stats uri /frontend
    bind *:8080
    acl routed_hdr hdr_cnt(X-IAM-Routed) gt 0
    http-request deny if routed_hdr
    http-request set-header X-IAM-Routed true
    default_backend servers

backend servers
    mode http
    timeout connect 5s
    timeout server 30s
    stats enable
    stats hide-version
    stats refresh 5s
    stats show-node
    stats scope .
    stats uri /backend
    option httpchk GET /albdefault
    http-check expect status 404
    server <name1> <ip1>:443 check ssl verify none
    server <name2> <ip2>:443 check ssl verify none

Sample packet capture 1:

23:52:46.765459 IP ip-10-39-143-101.ec2.internal.27034 > 2692ff359c5c.http-alt: Flags [S], seq 1559365195, win 29200, options [mss 1460,sackOK,TS val 212959171 ecr 0,nop,wscale 8], length 0
23:52:46.765476 IP 2692ff359c5c.http-alt > ip-10-39-143-101.ec2.internal.27034: Flags [S.], seq 2204833030, ack 1559365196, win 28960, options [mss 1460,sackOK,TS val 4075804078 ecr 212959171,nop,wscale 7], length 0
23:52:46.765840 IP ip-10-39-143-101.ec2.internal.27034 > 2692ff359c5c.http-alt: Flags [.], ack 1, win 115, options [nop,nop,TS val 212959171 ecr 4075804078], length 0
23:52:46.765870 IP ip-10-39-143-101.ec2.internal.27034 > 2692ff359c5c.http-alt: Flags [P.], seq 1:1030, ack 1, win 115, options [nop,nop,TS val 212959171 ecr 4075804078], length 1029: HTTP: GET /identity/v1/Users/me?clientTrackingId=aedzbyzaggbq HTTP/1.1
23:52:46.808995 IP 2692ff359c5c.http-alt > ip-10-39-143-101.ec2.internal.27034: Flags [.], ack 1030, win 243, options [nop,nop,TS val 4075804122 ecr 212959171], length 0
23:52:47.033165 IP 2692ff359c5c.http-alt > ip-10-39-143-101.ec2.internal.27034: Flags [P.], seq 1:1203, ack 1030, win 243, options [nop,nop,TS val 4075804346 ecr 212959171], length 1202: HTTP: HTTP/1.1 200 
23:52:47.033590 IP ip-10-39-143-101.ec2.internal.27034 > 2692ff359c5c.http-alt: Flags [.], ack 1203, win 124, options [nop,nop,TS val 212959238 ecr 4075804346], length 0
23:53:47.006260 IP ip-10-39-143-101.ec2.internal.27034 > 2692ff359c5c.http-alt: Flags [P.], seq 1030:2200, ack 1203, win 124, options [nop,nop,TS val 212974231 ecr 4075804346], length 1170: HTTP: GET /ext-admin/rest/accounts/3659355414452262666/users?filter=email+eq+%22iam-monitoring%40jedix.com%22&clientTrackingId=hsipwaforxnh HTTP/1.1
23:53:47.006274 IP 2692ff359c5c.http-alt > ip-10-39-143-101.ec2.internal.27034: Flags [.], ack 2200, win 261, options [nop,nop,TS val 4075864319 ecr 212974231], length 0
23:53:47.034015 IP 2692ff359c5c.http-alt > ip-10-39-143-101.ec2.internal.27034: Flags [F.], seq 1203, ack 2200, win 261, options [nop,nop,TS val 4075864347 ecr 212974231], length 0
23:53:47.034465 IP ip-10-39-143-101.ec2.internal.27034 > 2692ff359c5c.http-alt: Flags [F.], seq 2200, ack 1204, win 124, options [nop,nop,TS val 212974238 ecr 4075864347], length 0
23:53:47.034469 IP 2692ff359c5c.http-alt > ip-10-39-143-101.ec2.internal.27034: Flags [.], ack 2201, win 261, options [nop,nop,TS val 4075864347 ecr 212974238], length 0

Sample packet capture 2:

02:36:08.346717 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [S], seq 1517065129, win 29200, options [mss 1460,sackOK,TS val 242846745 ecr 0,nop,wscale 8], length 0
02:36:08.346733 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [S.], seq 3593117046, ack 1517065130, win 28960, options [mss 1460,sackOK,TS val 117475651 ecr 242846745,nop,wscale 7], length 0
02:36:08.347049 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [.], ack 1, win 115, options [nop,nop,TS val 242846745 ecr 117475651], length 0
02:36:08.347107 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [P.], seq 1:1030, ack 1, win 115, options [nop,nop,TS val 242846745 ecr 117475651], length 1029: HTTP: GET /identity/v1/Users/me?clientTrackingId=qcuvcvvysiat HTTP/1.1
02:36:08.388800 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [.], ack 1030, win 243, options [nop,nop,TS val 117475693 ecr 242846745], length 0
02:36:08.597069 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [P.], seq 1:1203, ack 1030, win 243, options [nop,nop,TS val 117475901 ecr 242846745], length 1202: HTTP: HTTP/1.1 200 
02:36:08.597436 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [.], ack 1203, win 124, options [nop,nop,TS val 242846808 ecr 117475901], length 0
02:36:43.390602 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [P.], seq 1030:2040, ack 1203, win 124, options [nop,nop,TS val 242855506 ecr 117475901], length 1010: HTTP: GET /identity/v1/Users/me HTTP/1.1
02:36:43.390618 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [.], ack 2040, win 259, options [nop,nop,TS val 117510695 ecr 242855506], length 0
02:36:43.609177 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [P.], seq 1203:1444, ack 2040, win 259, options [nop,nop,TS val 117510914 ecr 242855506], length 241: HTTP: HTTP/1.1 401 
02:36:43.609582 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [.], ack 1444, win 133, options [nop,nop,TS val 242855561 ecr 117510914], length 0
02:37:08.863217 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [P.], seq 2040:3069, ack 1444, win 133, options [nop,nop,TS val 242861874 ecr 117510914], length 1029: HTTP: GET /identity/v1/Users/me?clientTrackingId=ygozigevsvrq HTTP/1.1
02:37:08.863231 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [.], ack 3069, win 275, options [nop,nop,TS val 117536168 ecr 242861874], length 0
02:37:08.959388 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [P.], seq 1444:2646, ack 3069, win 275, options [nop,nop,TS val 117536264 ecr 242861874], length 1202: HTTP: HTTP/1.1 200 
02:37:08.959743 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [.], ack 2646, win 143, options [nop,nop,TS val 242861898 ecr 117536264], length 0
02:37:47.827284 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [P.], seq 3069:3795, ack 2646, win 143, options [nop,nop,TS val 242871615 ecr 117536264], length 726: HTTP: GET /ext-admin/rest/health HTTP/1.1
02:37:47.827307 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [.], ack 3795, win 291, options [nop,nop,TS val 117575132 ecr 242871615], length 0
02:37:47.868991 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [F.], seq 2646, ack 3795, win 291, options [nop,nop,TS val 117575174 ecr 242871615], length 0
02:37:47.869454 IP ip-10-39-142-203.ec2.internal.27000 > 43dccc417533.http-alt: Flags [F.], seq 3795, ack 2647, win 143, options [nop,nop,TS val 242871626 ecr 117575174], length 0
02:37:47.869459 IP 43dccc417533.http-alt > ip-10-39-142-203.ec2.internal.27000: Flags [.], ack 3796, win 291, options [nop,nop,TS val 117575174 ecr 242871626], length 0
2

Please update your haproxy version. Over 200 bugs have been fixed since 2.1.3 including 1 critical and 7 major bugs.

3

I upgraded to HAProxy 2.2.4 and the issue persists. I’ll stick with this version and run another packet capture. I’ll also enable debug logging. I’ve been having trouble getting function tracing to work, but I’ll give that another try as well.

# haproxy -v
HA-Proxy version 2.2.4-de45672 2020/09/30 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.4.html
Running on: Linux 4.14.181-140.257.amzn2.x86_64 #1 SMP Wed May 27 02:17:36 UTC 2020 x86_64