Layer 7 health check across TCP Proxy emitting CD Session State at Disconnection


#1

Howdy,

First post.

I have a L7 proxy that has backend servers that are actually local (127.0.0.*) TCP proxies. The reason I did this was to encapsulate the two physical ethernet interfaces on the actual backend servers behind the TCP proxy.

When I have a L7 health check that is looking for a “status=ok” response string traverse the TCP proxies, I get a tcp log entry that shows a session state at disconnection of “CD” which is:

C : the TCP session was unexpectedly aborted by the client.
D : the session was in the DATA phase.

This is only happening for the health check requests and not our normal expected traffic. If I remove the health checks the log entries stop.

My first thought was that it might be that the requests are expected to be keepalive and are not. I also thought that perhaps HAProxy might be optimized that once it receives the expected response string from the health check, it would end the connection??

Any insight into this is appreciated. I’d prefer not to have all of these non-actionable tcp log entries in our log files.

Thanks,

Michael


#2

I don’t quite understand your setup.

You have a 2 tier setup with one layer 7 proxy in the front and additionally multiple proxies on 127.0.0.x? Which of those proxy tiers is haproxy and which proxy emits the CD log?

Also provide configuration and output of haproxy -vv please.


#3

Thanks for the reply.

The layer 7 and layer 4 proxies are both the same instance of HAproxy on the same box. Here is a bit of the config and -vv output:

root@foo:/etc/haproxy# haproxy -vv
HA-Proxy version 1.7.9-1ppa1~trusty 2017/08/19
Copyright 2000-2017 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_NS=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.31 2012-07-06
Running on PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with network namespace support

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe


The layer 7 frontend:

frontend rest_FE
bind 1.2.3.4:443 transparent ssl crt company.com.pem
maxconn 64512

    http-request capture req.hdr(Host) len 25
    http-request capture req.hdr(Authorization),djb2 len 10

    acl shaz_dc_is_dead nbsrv(rest-shazprivpub-pool_BE) lt 1
    use_backend shillb-rest-vip_BE if shaz_dc_is_dead
    default_backend rest-shazprivpub-pool_BE

The default backend:

    backend rest-shazprivpub-pool_BE
    option forwardfor
    option log-health-checks
    option httpchk GET /foocheck.php HTTP/1.1\r\nHost:\ bar.company.com
    http-check expect string status=ok
    default-server inter 4s fall 3 rise 2
    balance leastconn
    # PK: added next two options so we can discuss
    #option persist
    #option redispatch
    stick-table type integer size 1M expire 11d peers mypeers
    stick on req.hdr(Authorization),djb2
    stick store-request req.hdr(Authorization),djb2

    server shazweb1-privpub_FE   127.1.0.1:10000   source 127.1.0.1 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt maxconn 400 check
    server shazweb2-privpub_FE   127.1.0.2:10000   source 127.1.0.2 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt maxconn 400 check
    server shazweb3-privpub_FE   127.1.0.3:10000   source 127.1.0.3 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt maxconn 400 check

.
.
.

The TCP proxies that are the backend servers for the above backend:

listen shazweb1-privpub
bind 127.1.0.1:10000
mode tcp
option tcplog
option dontlog-normal

    server shazweb1-int   10.64.9.10:443        check
    server shazweb1       198.143.147.138:443   check backup

listen shazweb2-privpub
bind 127.1.0.2:10000
mode tcp
option tcplog
option dontlog-normal

    server shazweb2-int   10.64.9.11:443        check
    server shazweb2       198.20.69.146:443     check backup

listen shazweb3-privpub
bind 127.1.0.3:10000
mode tcp
option tcplog
option dontlog-normal

    server shazweb3-int   10.64.9.12:443        check
    server shazweb3       198.20.69.58:443      check backup

.
.
.

A log entry from a TCP proxy when the layer 7 proxy on top of it does a health check:

Apr 20 09:30:43 shazlb2 haproxy[29720]: 127.1.0.1:18704 [20/Apr/2018:09:30:43.866] shazweb1-privpub shazweb1-privpub/shazweb1-int 1/0/2 328 CD 2/1/1/1/0 0/0

These are only happening on the health checks and not on all of our production traffic.

If I remove the checks from the server lines on the Layer 7 frontend, the tcp log enties for “CD” stop.