Haproxy send FIN/RST to backends unexpectedly

Help!
1

Hi,

We are using haproxy to provide public services, and found unexpected FIN/RST sent by haproxy, which caused the connection disconnected.

client: 47.93.x.x
server public ip(as haproxy): 139.198.y.y
server private ip (as backend) : 172.19.12.82

Please check attachment for more details:

Jietu20190916-135509
Jietu20190916-135509.jpg1894×528 346 KB

Haproxy has enabled transparent proxy, so we’ve seen two TCP streams:

  • stream 597: client -> haproxy
  • stream 598: haproxy -> backend

after normal TLS handshake interfaction between client->haproxy->backend, haproxy sent FIN (#21872) and RST (#21874) to backend directly.

These cause the connection to be interupted, request failed, and client reported:

Exception org.apache.http.NoHttpResponseException ERROR 调用customerHttpClient.execute(request)方法失败，Signals that an I/O exception of some sort has occurred. This class is the general class of exceptions produced by failed or interrupted I/O operations. org.apache.http.NoHttpResponseException: https://xxx:443 failed to respond

And since we don’t see any related packets recieved from client(src client -> dst haproxy), can we say that the haproxy sent the FIN/RST to backend all by itself? Does anybody has any clue why this happened?

haproxy version used:
HA-Proxy version 1.6.13 2017/06/18
Copyright 2000-2017 Willy Tarreau willy@haproxy.org

BTW, this behavior happened rarely, can not be reproduced manually.

2

haproxy.cfg is nearly the same as following:

global
        log 127.0.0.1   local0 info
        maxconn 6250
        daemon
        nbproc 1
        stats socket /haproxy/haproxy.1 process 1




defaults
        log     global
        option  dontlognull
        option  redispatch
        retries 3
        maxconn 6250
        timeout connect 5s
        timeout client 50s
        timeout server 50s
        timeout tunnel  1h
        option  tcpka
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http



    timeout client-fin 30s
listen  lbl-83m1h3iw
        bind *:443
        mode tcp
        option tcplog
        maxconn 6250
        timeout client 50s
        timeout tunnel 3600s

        default_backend lbl-83m1h3iw_default

backend lbl-83m1h3iw_default
        mode tcp
        option tcplog
        balance roundrobin
        timeout server 50s
        timeout tunnel 3600s
        timeout check 5000

        server  lbb-d2o7cylu 192.168.1.30:443 check inter 10000 fall 2 rise 5 weight 1
        server  lbb-emd2efil 192.168.1.32:443 check inter 10000 fall 2 rise 5 weight 1
        source 0.0.0.0 usesrc client


listen  lbl-nblcjr01
        bind *:80
        mode http
        option httplog
        maxconn 6250
        timeout client 50s
        timeout tunnel 3600s

        default_backend lbl-nblcjr01_default

backend lbl-nblcjr01_default
        mode http
        option httplog
        option forceclose
        balance roundrobin
        timeout server 50s
        timeout tunnel 3600s
        timeout check 5000

        server  lbb-zj3lchwb 192.168.1.32:80 check inter 10000 fall 2 rise 5 weight 1
        server  lbb-r3oywb8h 192.168.1.30:80 check inter 10000 fall 2 rise 5 weight 1
        source 0.0.0.0 usesrc client
3

Provide the output of haproxy -vv and the haproxy logs.

4

root@i-mqfiexf7:~# haproxy -vv
HA-Proxy version 1.6.13 2017/06/18
Copyright 2000-2017 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = x86_64
CC = gcc
CFLAGS = -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

root@i-mqfiexf7:~#

And haproxy don’t have any log then, /var/log/haproxy.log was always empty.

5
6

Yes, thanks @lukastribus , we just don’t enable syslog by default, so there wasn’t any (helpful) log for this question, and btw, we can hardly (actually never until now) reproduce this problem.