haproxy -vv (same on both)
HA-Proxy version 1.9.16 2020/07/31 - https://haproxy.org/
No more fixes for branch 1.9 past this version, please upgrade to branch 2.0!
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE2=1 USE_PCRE2_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.1g 21 Apr 2020
Running on OpenSSL version : OpenSSL 1.1.1g 21 Apr 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.35 2020-05-09
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
<default> : mode=HTX side=FE|BE
<default> : mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
config (same on both):
Long story short, we’re proxying a bunch of traffic for RDP and https, with some SSL termination and some SSL passthrough (that’s why all the backend/frontend redirection), but our rdp-proxy backend is the one set up for peer syncing. Please note the actual tables and such are working correctly, it’s just that changes to them are not syncing across peers
global
# * Process management and security
uid 65534
gid 65534
unix-bind uid 65534 gid 65534
nbproc 1
nbthread 3
cpu-map auto:1/1-3 0-2
log 127.0.0.1 local0 debug
maxconn 4096
server-state-file /var/run/haproxy.state
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:!EXPORT:!RC4:!aNULL:!MD5:!DSS
ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets
stats socket /var/run/haproxy.sock mode 666 level admin
stats socket 0.0.0.0:14567
stats timeout 2m
# * Performance tuning
tune.ssl.default-dh-param 2048
tune.ssl.lifetime "${SSL_SESSION_TIMEOUT}"
#
# Proxies
#
defaults
balance leastconn
log global
errorfile 500 /usr/local/etc/haproxy/errorfiles/500.http
errorfile 502 /usr/local/etc/haproxy/errorfiles/502.http
errorfile 503 /usr/local/etc/haproxy/errorfiles/503.http
maxconn 1024
option dontlognull
option http-server-close
option redispatch
retries 3
timeout check 5s
timeout client 301s
timeout connect 10s
timeout http-keep-alive 15s
timeout http-request 10s
timeout queue 1m
timeout server 301s
peers mypeers
peer web-gateway 0.0.0.0:1024
peer web-gateway1 10.1.4.43:8443
peer web-gateway2 10.1.4.44:8443
frontend tcp
bind :443
mode tcp
option tcplog
log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq %hr"
use_backend rdp-proxy if RDP_COOKIE
use_backend tls-redirect
backend tls-redirect
mode tcp
server tls unix@/var/run/tls-frontend.sock send-proxy-v2
frontend tls
bind unix@/var/run/tls-frontend.sock ssl crt /etc/ssl/private/ accept-proxy
mode tcp
default_backend https
backend https
mode http
server https unix@/var/run/https-frontend.sock send-proxy-v2
frontend https
bind unix@/var/run/https-frontend.sock accept-proxy
mode http
option httplog
option logasap
use_backend %[hdr(host),lower,map_dom(/usr/local/etc/haproxy/domain-to-backend.map,waf-backend)]
backend rdp-proxy
mode tcp
load-server-state-from-file global
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }
tcp-request content set-var(req.expiry) req.rdp_cookie(),map_str_int(/usr/local/etc/haproxy/rdp-proxy-uuid-to-expiry.map,0)
tcp-request content reject if RDP_COOKIE { date(),sub(req.expiry) ge 0 }
tcp-request content unset-var(req.expiry)
tcp-request content set-var(req.source) req.rdp_cookie(),map(/usr/local/etc/haproxy/rdp-proxy-uuid-to-src-ip.map,127.0.0.1)
#tcp-request content reject if RDP_COOKIE !{ src -m "str" %[req.rdp_cookie(),map(/usr/local/etc/haproxy/rdp-proxy-uuid-to-src-ip.map,127.0.0.1)] }
tcp-request content unset-var(req.source)
server-template rdp 1-"${RDP_PROXY_MAXIMUM}" foo.bar:3389 disabled init-addr last,0.0.0.0
stick-table type string len 15 size "${RDP_PROXY_MAXIMUM}" nopurge peers mypeers
stick on rdp_cookie(msts),map(/usr/local/etc/haproxy/rdp-proxy-uuid-to-vr-ip.map)
Each instance of haproxy is running in a container, with the host machine set to forward port 8443 to the container’s port 1024. Thus, each haproxy gets started with haproxy -W -db -L web-gateway -f /usr/local/etc/haproxy/haproxy.cfg.
Logging
Even with log level set to debug, there is NOTHING in the logs w/r/t peers of any kind. There is only this at startup:
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy http started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy waf started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy httpd-proxy started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy tcp started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy tls-redirect started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy tls started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy https started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy https started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy waf-backend started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy tcp-ip-abuse started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy http-ip-abuse started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy api-abuse started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: 2020-10-23T15:28:29+00:00 localhost haproxy[1]: Proxy rdp-proxy started.
Oct 23 11:28:29 web-gateway dockerd-current[1479]: [NOTICE] 296/152829 (1) : New worker #1 (13) forked
When I make a change to one of the stick tables via the stats socket, there is no additional logging of any kind
> show table rdp-proxy
# table: rdp-proxy, type: string, size:256, used:1
0x55797785b048: key=10.220.235.5 use=0 exp=0 server_id=15
tcpdumps
I couldn’t get anything reasonable without some kind of filter, so I’m including traffic from the instance where I made the socket call (the source) to the intended peer (the target) as well as traffic on the target coming from the peer.
This shows up immediately after I start the source haproxy:
source to/from target
DEV [root@web-gateway ~]# tcpdump -v -nn dst 10.1.4.44 or src 10.1.4.44
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:52:01.414057 IP (tos 0x0, ttl 63, id 21455, offset 0, flags [DF], proto TCP (6), length 60)
10.1.4.43.59946 > 10.1.4.44.8443: Flags [S], cksum 0x1c87 (incorrect -> 0xf4dc), seq 1615528482, win 29200, options [mss 1460,sackOK,TS val 87708141 ecr 0,nop,wscale 7], length 0
11:52:01.414704 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.1.4.44.8443 > 10.1.4.43.59946: Flags [S.], cksum 0xea0d (correct), seq 2056835192, ack 1615528483, win 28960, options [mss 1460,sackOK,TS val 87931743 ecr 87708141,nop,wscale 7], length 0
11:52:01.415824 IP (tos 0x0, ttl 63, id 21456, offset 0, flags [DF], proto TCP (6), length 95)
10.1.4.43.59946 > 10.1.4.44.8443: Flags [P.], cksum 0x1caa (incorrect -> 0xc00d), seq 1:44, ack 1, win 229, options [nop,nop,TS val 87708142 ecr 87931743], length 43
11:52:01.416466 IP (tos 0x0, ttl 63, id 27416, offset 0, flags [DF], proto TCP (6), length 52)
10.1.4.44.8443 > 10.1.4.43.59946: Flags [.], cksum 0x88e9 (correct), ack 44, win 227, options [nop,nop,TS val 87931745 ecr 87708142], length 0
11:52:01.416916 IP (tos 0x0, ttl 63, id 27417, offset 0, flags [DF], proto TCP (6), length 56)
10.1.4.44.8443 > 10.1.4.43.59946: Flags [P.], cksum 0x20a3 (correct), seq 1:5, ack 44, win 227, options [nop,nop,TS val 87931745 ecr 87708142], length 4
11:52:01.417123 IP (tos 0x0, ttl 63, id 27418, offset 0, flags [DF], proto TCP (6), length 52)
10.1.4.44.8443 > 10.1.4.43.59946: Flags [F.], cksum 0x88e3 (correct), seq 5, ack 44, win 227, options [nop,nop,TS val 87931746 ecr 87708142], length 0
11:52:01.417233 IP (tos 0x0, ttl 63, id 21457, offset 0, flags [DF], proto TCP (6), length 52)
10.1.4.43.59946 > 10.1.4.44.8443: Flags [R.], cksum 0x1c7f (incorrect -> 0x88dc), seq 44, ack 6, win 229, options [nop,nop,TS val 87708144 ecr 87931745], length 0
11:52:06.422893 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.1.4.43 tell 10.1.4.44, length 46
11:52:06.422939 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.43 is-at 00:1a:4a:02:00:1f, length 28
target from source
DEV [root@web-gateway2 ~]# tcpdump -i any -v -nn src 10.1.4.43 or dst 10.1.4.43
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:52:05.371195 IP (tos 0x0, ttl 63, id 21455, offset 0, flags [DF], proto TCP (6), length 60)
10.1.4.43.59946 > 10.1.4.44.8443: Flags [S], cksum 0xf4dc (correct), seq 1615528482, win 29200, options [mss 1460,sackOK,TS val 87708141 ecr 0,nop,wscale 7], length 0
11:52:05.371372 IP (tos 0x0, ttl 62, id 21455, offset 0, flags [DF], proto TCP (6), length 60)
10.1.4.43.59946 > 172.18.0.6.1024: Flags [S], cksum 0x73ec (correct), seq 1615528482, win 29200, options [mss 1460,sackOK,TS val 87708141 ecr 0,nop,wscale 7], length 0
11:52:05.371393 IP (tos 0x0, ttl 62, id 21455, offset 0, flags [DF], proto TCP (6), length 60)
10.1.4.43.59946 > 172.18.0.6.1024: Flags [S], cksum 0x73ec (correct), seq 1615528482, win 29200, options [mss 1460,sackOK,TS val 87708141 ecr 0,nop,wscale 7], length 0
11:52:05.371488 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.18.0.6.1024 > 10.1.4.43.59946: Flags [S.], cksum 0xba72 (incorrect -> 0x691d), seq 2056835192, ack 1615528483, win 28960, options [mss 1460,sackOK,TS val 87931743 ecr 87708141,nop,wscale 7], length 0
11:52:05.371488 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
172.18.0.6.1024 > 10.1.4.43.59946: Flags [S.], cksum 0xba72 (incorrect -> 0x691d), seq 2056835192, ack 1615528483, win 28960, options [mss 1460,sackOK,TS val 87931743 ecr 87708141,nop,wscale 7], length 0
11:52:05.371536 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
10.1.4.44.8443 > 10.1.4.43.59946: Flags [S.], cksum 0x1c87 (incorrect -> 0xea0d), seq 2056835192, ack 1615528483, win 28960, options [mss 1460,sackOK,TS val 87931743 ecr 87708141,nop,wscale 7], length 0
11:52:05.373004 IP (tos 0x0, ttl 63, id 21456, offset 0, flags [DF], proto TCP (6), length 95)
10.1.4.43.59946 > 10.1.4.44.8443: Flags [P.], cksum 0xc00d (correct), seq 1:44, ack 1, win 229, options [nop,nop,TS val 87708142 ecr 87931743], length 43
11:52:05.373107 IP (tos 0x0, ttl 62, id 21456, offset 0, flags [DF], proto TCP (6), length 95)
10.1.4.43.59946 > 172.18.0.6.1024: Flags [P.], cksum 0x3f1d (correct), seq 1:44, ack 1, win 229, options [nop,nop,TS val 87708142 ecr 87931743], length 43
11:52:05.373124 IP (tos 0x0, ttl 62, id 21456, offset 0, flags [DF], proto TCP (6), length 95)
10.1.4.43.59946 > 172.18.0.6.1024: Flags [P.], cksum 0x3f1d (correct), seq 1:44, ack 1, win 229, options [nop,nop,TS val 87708142 ecr 87931743], length 43
11:52:05.373196 IP (tos 0x0, ttl 64, id 27416, offset 0, flags [DF], proto TCP (6), length 52)
172.18.0.6.1024 > 10.1.4.43.59946: Flags [.], cksum 0xba6a (incorrect -> 0x07f9), ack 44, win 227, options [nop,nop,TS val 87931745 ecr 87708142], length 0
11:52:05.373196 IP (tos 0x0, ttl 64, id 27416, offset 0, flags [DF], proto TCP (6), length 52)
172.18.0.6.1024 > 10.1.4.43.59946: Flags [.], cksum 0xba6a (incorrect -> 0x07f9), ack 44, win 227, options [nop,nop,TS val 87931745 ecr 87708142], length 0
11:52:05.373329 IP (tos 0x0, ttl 63, id 27416, offset 0, flags [DF], proto TCP (6), length 52)
10.1.4.44.8443 > 10.1.4.43.59946: Flags [.], cksum 0x1c7f (incorrect -> 0x88e9), ack 44, win 227, options [nop,nop,TS val 87931745 ecr 87708142], length 0
11:52:05.373691 IP (tos 0x0, ttl 64, id 27417, offset 0, flags [DF], proto TCP (6), length 56)
172.18.0.6.1024 > 10.1.4.43.59946: Flags [P.], cksum 0xba6e (incorrect -> 0x9fb2), seq 1:5, ack 44, win 227, options [nop,nop,TS val 87931745 ecr 87708142], length 4
11:52:05.373691 IP (tos 0x0, ttl 64, id 27417, offset 0, flags [DF], proto TCP (6), length 56)
172.18.0.6.1024 > 10.1.4.43.59946: Flags [P.], cksum 0xba6e (incorrect -> 0x9fb2), seq 1:5, ack 44, win 227, options [nop,nop,TS val 87931745 ecr 87708142], length 4
11:52:05.373791 IP (tos 0x0, ttl 63, id 27417, offset 0, flags [DF], proto TCP (6), length 56)
10.1.4.44.8443 > 10.1.4.43.59946: Flags [P.], cksum 0x1c83 (incorrect -> 0x20a3), seq 1:5, ack 44, win 227, options [nop,nop,TS val 87931745 ecr 87708142], length 4
11:52:05.373942 IP (tos 0x0, ttl 64, id 27418, offset 0, flags [DF], proto TCP (6), length 52)
172.18.0.6.1024 > 10.1.4.43.59946: Flags [F.], cksum 0xba6a (incorrect -> 0x07f3), seq 5, ack 44, win 227, options [nop,nop,TS val 87931746 ecr 87708142], length 0
11:52:05.373942 IP (tos 0x0, ttl 64, id 27418, offset 0, flags [DF], proto TCP (6), length 52)
172.18.0.6.1024 > 10.1.4.43.59946: Flags [F.], cksum 0xba6a (incorrect -> 0x07f3), seq 5, ack 44, win 227, options [nop,nop,TS val 87931746 ecr 87708142], length 0
11:52:05.374011 IP (tos 0x0, ttl 63, id 27418, offset 0, flags [DF], proto TCP (6), length 52)
10.1.4.44.8443 > 10.1.4.43.59946: Flags [F.], cksum 0x1c7f (incorrect -> 0x88e3), seq 5, ack 44, win 227, options [nop,nop,TS val 87931746 ecr 87708142], length 0
11:52:05.374393 IP (tos 0x0, ttl 63, id 21457, offset 0, flags [DF], proto TCP (6), length 52)
10.1.4.43.59946 > 10.1.4.44.8443: Flags [R.], cksum 0x88dc (correct), seq 44, ack 6, win 229, options [nop,nop,TS val 87708144 ecr 87931745], length 0
11:52:05.374452 IP (tos 0x0, ttl 62, id 21457, offset 0, flags [DF], proto TCP (6), length 52)
10.1.4.43.59946 > 172.18.0.6.1024: Flags [R.], cksum 0x07ec (correct), seq 44, ack 6, win 229, options [nop,nop,TS val 87708144 ecr 87931745], length 0
11:52:05.374464 IP (tos 0x0, ttl 62, id 21457, offset 0, flags [DF], proto TCP (6), length 52)
10.1.4.43.59946 > 172.18.0.6.1024: Flags [R.], cksum 0x07ec (correct), seq 44, ack 6, win 229, options [nop,nop,TS val 87708144 ecr 87931745], length 0
11:52:10.379713 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.1.4.43 tell 10.1.4.44, length 28
11:52:10.380144 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.1.4.43 is-at 00:1a:4a:02:00:1f, length 46
After that, there is ZERO traffic between the 2 machines, even after I update the stick table referenced in the config section:
DEV [root@web-gateway ~]# socat /var/lib/docker/volumes/haproxy-run/_data/haproxy.sock readline
prompt
> show table rdp-proxy
# table: rdp-proxy, type: string, size:256, used:1
0x5642c440c9c8: key=10.220.235.5 use=0 exp=0 server_id=10
Please note that I have tried modifying the peers section independently in each configuration file to be something like
peers mypeers
peer web-gateway 0.0.0.0:1024
peer web-gateway2 10.1.4.44:8443
on the “source” and
peers mypeers
peer web-gateway 0.0.0.0:1024
peer web-gateway1 10.1.4.43:8443
on the “target”. The behavior was exactly the same