HAPROXY 1.5.4/1.5.18 is sending [RST, ACK] in for closure to httpchk instead of [FIN, ACK]

Help!
1

After some research and reviewing this other case posted on this site, I’m of the believe that version 1.5.4 and above of haproxy was modified to send a [FIN, ACK] for an http closure instead of the [RST, ACK] that earlier versions provided. In my testing I only ever see version 1.5.X send the [RST, ACK]. I’ve done this testing with both the 1.5.4 installed from RPM as well as that 1.5.18 that I’ve built from source and I get the same results.

Here is a trace of haproxy doing an httpchck to my service running on port 8181:

  1 0.000000000    10.41.183.26          10.253.215.32         TCP      76     36435→8181 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=3353589862 TSecr=0 WS=128
  2 0.000018502    10.253.215.32         10.41.183.26          TCP      76     8181→36435 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=4196372022 TSecr=3353589862 WS=256
  3 0.001154189    10.41.183.26          10.253.215.32         TCP      68     36435→8181 [ACK] Seq=1 Ack=1 Win=14720 Len=0 TSval=3353589863 TSecr=4196372022
  4 0.001198735    10.41.183.26          10.253.215.32         HTTP     104    GET /ping 1.1
  5 0.001205593    10.253.215.32         10.41.183.26          TCP      68     8181→36435 [ACK] Seq=1 Ack=37 Win=14592 Len=0 TSval=4196372023 TSecr=3353589863
  6 0.001486994    10.253.215.32         10.41.183.26          HTTP/XML 214    HTTP/1.1 200 OK
  7 0.002579072    10.41.183.26          10.253.215.32         TCP      68     36435→8181 [ACK] Seq=37 Ack=147 Win=15744 Len=0 TSval=3353589864 TSecr=4196372023
  8 0.002626112    10.41.183.26          10.253.215.32         TCP      68     36435→8181 [RST, ACK] Seq=37 Ack=147 Win=15744 Len=0 TSval=3353589864 TSecr=4196372023

This httpchk was driven by this haproxy.cfg

> global
> log 127.0.0.1 local0

> chroot /var/lib/haproxy
> pidfile /var/run/haproxy.pid
> maxconn 4000
> user haproxy
> group haproxy
> daemon

> # turn on stats unix socket
> stats socket /var/lib/haproxy/stats

> defaults
> log global
> retries 3
> timeout connect 10s
> timeout client 10000m
> timeout server 10000m

> listen my-cluster
> bind 127.0.0.1:8181
> mode http
> option httpchk GET /ping 1.1
> http-check expect string AUTH
> option httplog
> option http-server-close
> default-server inter 30s
> server myserv-1 betadb-p:8181 check

I’ve attempted to use the ‘http-server-close’ option with this haproxy.cfg:

> global
> log 127.0.0.1 local0

> chroot /var/lib/haproxy
> pidfile /var/run/haproxy.pid
> maxconn 4000
> user haproxy
> group haproxy
> daemon

> # turn on stats unix socket
> stats socket /var/lib/haproxy/stats

> defaults
> log global
> retries 3
> timeout connect 10s
> timeout client 10000m
> timeout server 10000m

> listen my-cluster
> bind 127.0.0.1:8181
> mode http
> option httpchk GET /ping 1.1
> http-check expect string AUTH
> option httplog
> option http-server-close
> default-server inter 30s
> server myserv-1 betadb-p:8181 check

And that did not change the network trace as far as I can tell:

  1 0.000000000    10.253.215.39         10.253.215.32         TCP      76     36371→8181 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=4193044510 TSecr=0 WS=256
  2 0.000020008    10.253.215.32         10.253.215.39         TCP      76     8181→36371 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1460 SACK_PERM=1 TSval=4196446053 TSecr=4193044510 WS=256
  3 0.000266409    10.253.215.39         10.253.215.32         TCP      68     36371→8181 [ACK] Seq=1 Ack=1 Win=14848 Len=0 TSval=4193044510 TSecr=4196446053
  4 0.000312234    10.253.215.39         10.253.215.32         HTTP     90     GET /ping HTTP/1.0
  5 0.000320249    10.253.215.32         10.253.215.39         TCP      68     8181→36371 [ACK] Seq=1 Ack=23 Win=14592 Len=0 TSval=4196446053 TSecr=4193044510
  6 0.000566198    10.253.215.32         10.253.215.39         HTTP/XML 214    HTTP/1.1 200 OK
  7 0.000709283    10.253.215.39         10.253.215.32         TCP      68     36371→8181 [ACK] Seq=23 Ack=147 Win=15872 Len=0 TSval=4193044510 TSecr=4196446053
  8 0.000776358    10.253.215.39         10.253.215.32         TCP      68     36371→8181 [RST, ACK] Seq=23 Ack=147 Win=15872 Len=0 TSval=4193044510 TSecr=4196446053

So I guess I’m looking for some advice if there is some sort of configuration I need in place to get haproxy to send the [FIN, ACK] instead of the [RST, ACK]. The [RST, ACK] closures are causing my programs socket errors saying the ‘connection was reset by peer’

2

But in this thread you are claiming exactly the opposite (RST vs FIN) for a different feature (health check vs actual traffic) so I assume the other issue has nothing to do with your issue.

There is no behavior change within stable 1.5 releases.

The behavior changed in haproxy 1.5-dev14, commit fd29cc537 (“MEDIUM: checks: avoid accumulating TIME_WAITs during checks”). This was also backported to haproxy-1.4 in commit 2f61455, but the 1.4 backport is different than the 1.5 commit and also does not seem to cause so much RST as the original 1.5 commit.

Can you confirm that haproxy 1.4 (1.4.23 or later) doesn’t have this problem?

Also I suggest you don’t force HTTP/1.1, it doesn’t make sense. Just use 1.0 by declaring:
option httpchk GET /ping

without the 1.1 addition.

3

My understanding is that haproxy release 1.5.4 and above should be sending [FIN,ACK], but my testing and traces show that I’m getting [RST,ACK]. I want to see the [FIN,ACK]. I’m looking for some feedback on how to get haproxy to send the [FIN,ACK] or why I’m seeing [RST,ACK].

4

Your understanding is wrong. The other thread is filled with wrong assumptions, please ignore them.

All haproxy 1.5 releases behave exactly the same, and this is due to the fix “MEDIUM: checks: avoid accumulating TIME_WAITs during checks”:

MEDIUM: checks: avoid accumulating TIME_WAITs during checks

Some checks which do not induce a close from the server accumulate
local TIME_WAIT sockets because they’re cleanly shut down. Typically
TCP probes cause this. This is very problematic when there are many
servers, when the checks are fast or when local source ports are rare.

So now we’ll disable lingering on the socket instead of sending a
shutdown. Before doing this we try to drain any possibly pending data.
That way we avoid sending an RST when the server has closed first.

This change means that some servers will see more RSTs, but this is
needed to avoid local source port starvation.

Seeings RST is expected behavior, although haproxy tries to drain the socket before closing it and therefor minimize RST use.

Please follow the suggestions from the previous post.

5

I know this is a 5yo thread, but I stumbled upon it while trying to explain what looked to me a weird behavior: HAProxy is sending three RST packets after it has sent a FIN. The last two of those three RST are responses to packets arriving from the server:

  • the first one is a FIN from the server itself
  • the second being the ACK for HAP’s FIN

Those packets were clearly sent as part of the FIN sequence.

I guess my question would be, why is HAProxy sending a FIN before the RST, or sending a RST after a FIN?