Thanks for the reply.
The layer 7 and layer 4 proxies are both the same instance of HAproxy on the same box. Here is a bit of the config and -vv output:
root@foo:/etc/haproxy# haproxy -vv
HA-Proxy version 1.7.9-1ppa1~trusty 2017/08/19
Copyright 2000-2017 Willy Tarreau willy@haproxy.org
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_NS=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.31 2012-07-06
Running on PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with network namespace support
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe
The layer 7 frontend:
frontend rest_FE
bind 1.2.3.4:443 transparent ssl crt company.com.pem
maxconn 64512
http-request capture req.hdr(Host) len 25
http-request capture req.hdr(Authorization),djb2 len 10
acl shaz_dc_is_dead nbsrv(rest-shazprivpub-pool_BE) lt 1
use_backend shillb-rest-vip_BE if shaz_dc_is_dead
default_backend rest-shazprivpub-pool_BE
The default backend:
backend rest-shazprivpub-pool_BE
option forwardfor
option log-health-checks
option httpchk GET /foocheck.php HTTP/1.1\r\nHost:\ bar.company.com
http-check expect string status=ok
default-server inter 4s fall 3 rise 2
balance leastconn
# PK: added next two options so we can discuss
#option persist
#option redispatch
stick-table type integer size 1M expire 11d peers mypeers
stick on req.hdr(Authorization),djb2
stick store-request req.hdr(Authorization),djb2
server shazweb1-privpub_FE 127.1.0.1:10000 source 127.1.0.1 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt maxconn 400 check
server shazweb2-privpub_FE 127.1.0.2:10000 source 127.1.0.2 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt maxconn 400 check
server shazweb3-privpub_FE 127.1.0.3:10000 source 127.1.0.3 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt maxconn 400 check
.
.
.
The TCP proxies that are the backend servers for the above backend:
listen shazweb1-privpub
bind 127.1.0.1:10000
mode tcp
option tcplog
option dontlog-normal
server shazweb1-int 10.64.9.10:443 check
server shazweb1 198.143.147.138:443 check backup
listen shazweb2-privpub
bind 127.1.0.2:10000
mode tcp
option tcplog
option dontlog-normal
server shazweb2-int 10.64.9.11:443 check
server shazweb2 198.20.69.146:443 check backup
listen shazweb3-privpub
bind 127.1.0.3:10000
mode tcp
option tcplog
option dontlog-normal
server shazweb3-int 10.64.9.12:443 check
server shazweb3 198.20.69.58:443 check backup
.
.
.
A log entry from a TCP proxy when the layer 7 proxy on top of it does a health check:
Apr 20 09:30:43 shazlb2 haproxy[29720]: 127.1.0.1:18704 [20/Apr/2018:09:30:43.866] shazweb1-privpub shazweb1-privpub/shazweb1-int 1/0/2 328 CD 2/1/1/1/0 0/0
These are only happening on the health checks and not on all of our production traffic.
If I remove the checks from the server lines on the Layer 7 frontend, the tcp log enties for “CD” stop.