Backend close changed from RST in 1.4 to FIN in 1.5


#1

Hello everyone

I’ve been trawling the change logs and cannot find any reference to this substantial change:

Haproxy 1.4.22 - closes backend with RST+ACK
Haproxy 1.5.4 - closes backend with FIN+ACK

I’m curious to know if I can control it via configuration as it is causing half-closed connections on a dodgy old Windows backend. Clearly FIN+ACK is perfectly good and presumably preferable but in certain scenarios Windows/IIS is misbehaving.

Anyone noticed this?

Cheers

Kev


HAPROXY 1.5.4/1.5.18 is sending [RST, ACK] in for closure to httpchk instead of [FIN, ACK]
#2

Could you share relevant part of your configuration and give more info on the type of application hosted on IIS?


#3

I’m looking into the windows/IIS/app side separately, the half closed connections occurs on http/1.0 connections and bad site bindings that hit HTTP.SYS instead of IIS.

I wanted to get the an understanding of the change in HAProxy from this forum. Any ideas when/why/how this changed?


#4

Some defaults changed, the most important one is the default http mode. To understand if that is what you are seeing, you would have to provide your configuration.

You can also try setting “option http-tunnel” in the default section in 1.5, that should restore 1.4 behavior (unless you are already using specific http modes).

However, please don’t use 1.5.4 in production, but always the latest bugfix release. 1.5.4 contains 141 bugs that are all fixed in 1.5.17.


#5

Happy to share my haproxy configuration. I don’t believe we are falling on any defaults but that would explain the change from 1.4 to 1.5. Yes we should use 1.5.17 if we can get off 1.4

global
  log     /dev/log local2
  chroot  /var/lib/haproxy
  pidfile /var/run/haproxy.pid
  maxconn 50000
  user    haproxy
  group   haproxy
  daemon
  stats socket /var/lib/haproxy/stats level admin


defaults http
  mode    http
  log     global
  option  httplog
  option  dontlognull
  option  http-server-close
  option  redispatch
  retries 3
  timeout http-request    10s
  timeout queue           1m
  timeout connect         10s
  timeout client          1m
  timeout server          5m
  timeout http-keep-alive 10s
  timeout check           10s
  maxconn 50000

listen webservice
  bind x.x.x.x:80
  balance hdr(X-Forwarded-For)
  option  httpchk GET /healthcheck
  default-server inter 30s
  server  w1 webservice1:80 check
  server  w2 webservice2:80 check

As you can see we set the http defaults to http-server-close and the way this closes connections appears to have changed. Is there anything else you can spot?


#6

Ok, so clearly its not about the http default mode, as that is http-server-close.

Can you tell whether those backend close are from regular production traffic or if they come from the health checks?

With http-server-close we would actually expect the backend server to close the connection after its response.

Do you have tcpdumps available for this traffic?


#7

Yes I have tcpdumps showing this occurring on production traffic. Surely with http-server-close you mean to say that HAProxy closes the connection to the backend server, not the other way around?

It is the way HAProxy closes the connection I am most interested in (RST or FIN)


#8

So you are saying you are sure that this is not about the health checks?

http-server-close means haproxy emits the “Connection: close” header, expecting the backend to actually close the session after the response. However there are backends that don’t respect Connection: close. Thats why we would have to take a look at that tcpdump.


#9

Ah right, turns out the best description is under “http-pretend-keepalive” not “http-server-close”

When running with “option http-server-close” or “option forceclose”, haproxy
adds a “Connection: close” header to the request forwarded to the server.

And I can confirm the health checks don’t include this (you may be implying that this is by design) and neither does the production traffic. However on this application the production traffic is all HTTP/1.0 which does not support keepalive or the Connection header.

GET / HTTP/1.0
Host: 203.171.50.54:80
X-Forwarded-For: 121.90.72.34
Accept: */*


HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 365
Content-Type: text/xml; charset=UTF-8
Accept-Ranges: bytes
Server: Microsoft-IIS/7.5
Ext: 
X-Powered-By: ASP.NET
Date: Thu, 07 Apr 2016 23:19:50 GMT
Connection: close

So, does that mean HA Proxy is forced to decide how to close the connection since http-server-close is not compatible with HTTP 1.0?

Please bare with me, I’m trying not to resort to pasting a full tcp dump.


#10

Right, HTTP/1.0 does not need and does not support the connection header. With HTTP/1.0, the backend server has to close after the response (as keepalive is not supported).

Can you tell how the FIN/FIN+ACK/ACK sequence looks like? Maybe you can just collect the output of “tcpdump -n port 80” and obfuscate the ip addresses? And compare that output to the one in haproxy 1.4 (in fact, the tcpdump output of 1.4 would be more interesting).

A ´curl --http1.0 api.ipify.org´ for example looks like this:

11:17:45.102437 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [S], seq 3118574468, win 29200, options [mss 1460,sackOK,TS val 60317 ecr 0,nop,wscale 6], length 0
11:17:45.206749 IP 54.197.255.190.80 > 172.31.0.5.50806: Flags [S.], seq 9646, ack 3118574469, win 32768, options [mss 1460], length 0
11:17:45.206827 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [.], ack 1, win 29200, length 0
11:17:45.207276 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [P.], seq 1:78, ack 1, win 29200, length 77
11:17:45.313580 IP 54.197.255.190.80 > 172.31.0.5.50806: Flags [P.], seq 1:167, ack 78, win 32691, length 166
11:17:45.313642 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [.], ack 167, win 30016, length 0
11:17:45.313685 IP 54.197.255.190.80 > 172.31.0.5.50806: Flags [F.], seq 167, ack 78, win 32691, length 0
11:17:45.314017 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [F.], seq 78, ack 168, win 30016, length 0
11:17:45.315372 IP 54.197.255.190.80 > 172.31.0.5.50806: Flags [.], ack 79, win 32690, length 0