Hello everyone
I’ve been trawling the change logs and cannot find any reference to this substantial change:
Haproxy 1.4.22 - closes backend with RST+ACK
Haproxy 1.5.4 - closes backend with FIN+ACK
I’m curious to know if I can control it via configuration as it is causing half-closed connections on a dodgy old Windows backend. Clearly FIN+ACK is perfectly good and presumably preferable but in certain scenarios Windows/IIS is misbehaving.
Anyone noticed this?
Cheers
Kev
Could you share relevant part of your configuration and give more info on the type of application hosted on IIS?
I’m looking into the windows/IIS/app side separately, the half closed connections occurs on http/1.0 connections and bad site bindings that hit HTTP.SYS instead of IIS.
I wanted to get the an understanding of the change in HAProxy from this forum. Any ideas when/why/how this changed?
Some defaults changed, the most important one is the default http mode. To understand if that is what you are seeing, you would have to provide your configuration.
You can also try setting “option http-tunnel” in the default section in 1.5, that should restore 1.4 behavior (unless you are already using specific http modes).
However, please don’t use 1.5.4 in production, but always the latest bugfix release. 1.5.4 contains 141 bugs that are all fixed in 1.5.17.
Happy to share my haproxy configuration. I don’t believe we are falling on any defaults but that would explain the change from 1.4 to 1.5. Yes we should use 1.5.17 if we can get off 1.4
global
log /dev/log local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 50000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats level admin
defaults http
mode http
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 5m
timeout http-keep-alive 10s
timeout check 10s
maxconn 50000
listen webservice
bind x.x.x.x:80
balance hdr(X-Forwarded-For)
option httpchk GET /healthcheck
default-server inter 30s
server w1 webservice1:80 check
server w2 webservice2:80 check
As you can see we set the http defaults to http-server-close and the way this closes connections appears to have changed. Is there anything else you can spot?
Ok, so clearly its not about the http default mode, as that is http-server-close.
Can you tell whether those backend close are from regular production traffic or if they come from the health checks?
With http-server-close we would actually expect the backend server to close the connection after its response.
Do you have tcpdumps available for this traffic?
Yes I have tcpdumps showing this occurring on production traffic. Surely with http-server-close you mean to say that HAProxy closes the connection to the backend server, not the other way around?
It is the way HAProxy closes the connection I am most interested in (RST or FIN)
So you are saying you are sure that this is not about the health checks?
http-server-close means haproxy emits the “Connection: close” header, expecting the backend to actually close the session after the response. However there are backends that don’t respect Connection: close. Thats why we would have to take a look at that tcpdump.
Ah right, turns out the best description is under “http-pretend-keepalive” not “http-server-close”
When running with “option http-server-close” or “option forceclose”, haproxy
adds a “Connection: close” header to the request forwarded to the server.
And I can confirm the health checks don’t include this (you may be implying that this is by design) and neither does the production traffic. However on this application the production traffic is all HTTP/1.0 which does not support keepalive or the Connection header.
GET / HTTP/1.0
Host: 203.171.50.54:80
X-Forwarded-For: 121.90.72.34
Accept: */*
HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 365
Content-Type: text/xml; charset=UTF-8
Accept-Ranges: bytes
Server: Microsoft-IIS/7.5
Ext:
X-Powered-By: ASP.NET
Date: Thu, 07 Apr 2016 23:19:50 GMT
Connection: close
So, does that mean HA Proxy is forced to decide how to close the connection since http-server-close is not compatible with HTTP 1.0?
Please bare with me, I’m trying not to resort to pasting a full tcp dump.
Right, HTTP/1.0 does not need and does not support the connection header. With HTTP/1.0, the backend server has to close after the response (as keepalive is not supported).
Can you tell how the FIN/FIN+ACK/ACK sequence looks like? Maybe you can just collect the output of “tcpdump -n port 80” and obfuscate the ip addresses? And compare that output to the one in haproxy 1.4 (in fact, the tcpdump output of 1.4 would be more interesting).
A ´curl --http1.0 api.ipify.org´ for example looks like this:
11:17:45.102437 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [S], seq 3118574468, win 29200, options [mss 1460,sackOK,TS val 60317 ecr 0,nop,wscale 6], length 0
11:17:45.206749 IP 54.197.255.190.80 > 172.31.0.5.50806: Flags [S.], seq 9646, ack 3118574469, win 32768, options [mss 1460], length 0
11:17:45.206827 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [.], ack 1, win 29200, length 0
11:17:45.207276 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [P.], seq 1:78, ack 1, win 29200, length 77
11:17:45.313580 IP 54.197.255.190.80 > 172.31.0.5.50806: Flags [P.], seq 1:167, ack 78, win 32691, length 166
11:17:45.313642 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [.], ack 167, win 30016, length 0
11:17:45.313685 IP 54.197.255.190.80 > 172.31.0.5.50806: Flags [F.], seq 167, ack 78, win 32691, length 0
11:17:45.314017 IP 172.31.0.5.50806 > 54.197.255.190.80: Flags [F.], seq 78, ack 168, win 30016, length 0
11:17:45.315372 IP 54.197.255.190.80 > 172.31.0.5.50806: Flags [.], ack 79, win 32690, length 0