Haproxy replies with TCP FIN on HTTP GET

Help!
1

Hello,

I am happily using haproxy 2.0.17 (also happens with 2.0.19) with quite such configuration:

global
  pidfile haproxy.pid
  master-worker

log log.sck local0
defaults
  log global
  mode http
  option httplog
  timeout queue 60s
  timeout server 600s
  timeout client 600s
  timeout connect 5s
  retries 3

frontend http-backend
  bind <ip>:41080

  acl is_<N> hdr_reg(host) -i ^<N>.example.com($|:.*)
  use_backend <N>-http if is_<N>

backend <N>-http
  timeout server 600s
  timeout connect 5s
  retries 3
  server backend ip:v6:port ssl verify none

Where I have N about 3000 configurations.

It works pretty well ~99,995% of time, but sometimes haproxy replies with TCP FIN to proper HTTP GET from a client on a long lasting connection, on which before it responded a lot with proper reply, see:

shark-ts-ha
shark-ts-ha1185×391 116 KB

This requires the client to reconnect (and for now this is how I workarounded it on client side, by retrying connection when lost).

In the log file for this source port I only find quite correct entries, and none related to the problem I have:

10.0.251.170:42742 [07/Jan/2021:13:02:00.837] haproxy A-http/backend 0/0/222/57/279 304 154 - - ---- 29/3/0/0/0 0/0 "GET /REDACTED/ HTTP/1.1"
10.0.251.170:42742 [07/Jan/2021:13:02:02.565] haproxy B-http/backend 0/0/3/378/381 200 5044 - - ---- 29/3/0/0/0 0/0 "GET /REDACTED/ HTTP/1.1"
10.0.251.170:42742 [07/Jan/2021:13:02:03.984] haproxy C-http/backend 0/0/218/168/386 304 154 - - ---- 29/3/0/0/0 0/0 "GET /REDACTED/ HTTP/1.1"
10.0.251.170:42742 [07/Jan/2021:13:02:05.927] haproxy D-http/backend 0/0/350/56/406 304 154 - - ---- 29/3/0/0/0 0/0 "GET /REDACTED/ HTTP/1.1"
10.0.251.170:42742 [07/Jan/2021:13:02:07.229] haproxy E-http/backend 0/0/217/55/272 304 154 - - ---- 29/3/0/0/0 0/0 "GET /REDACTED/ HTTP/1.1"
10.0.251.170:42742 [07/Jan/2021:13:02:10.614] haproxy F-http/backend 0/0/100/78/178 304 96 - - ---- 29/3/0/0/0 0/0 "GET /REDACTED/ HTTP/1.1"

So, is it possible that source of this TCP FIN is not haproxy itself? Or shall I look in some other place?

If needed I can try to reproduce this problem with any given version or additional debugging configuration.

Regards,
Łukasz

2

Sounds like haproxy is stopped (or crashes). How long those the PID of haproxy that served your request live after this happened?

When Haproxy disconnects it is supposed to log the state at disconnection, always.

Whatever the root cause here may be, retrying idempotent transactions on temporary failures is always a good idea.

3

Hello,

Haproxy PID stays the same, many days while having such connectivity issue.

Regards,
Łukasz

4

Like:

user  2273  0.0  2.2 195564 88932 ?        S    Feb01   0:34 haproxy -sf 10884 -f haproxy.cfg
user 16301  0.2  2.1 201664 87280 ?        Sl   Feb02  24:39 haproxy -sf 10884 -f haproxy.cfg

And since that time I had those issue few times.

5

Can you provide the original screenshot above? The “problem FIN” overlay hides the most important information, which is the timestamp of the FIN’s.

6

Hi,

Sure:

full-screenshot
full-screenshot1785×556 315 KB

Regards,
Łukasz

7

I’m unsure what happens here, I think it would be best to file an bug on github so we can troubleshoot further there:

8

Done: Haproxy replies with TCP FIN on HTTP GET ¡ Issue #1109 ¡ haproxy/haproxy ¡ GitHub

1 Like