HAProxy on Opnsense Asynchronous routing

Help!
1

Hey,
currently I run into some problems with two seperate opnsenses with installed HAProxy on both. I have a load balancer in front of the Opnsenses and this will balance the traffic over both machines.

HAProxy does also do the SSL-Stuff according to this tutorial Tutorial 2024/06: HAProxy + Let's Encrypt Wildcard Certificates + 100% A+ Rating

My problem is that I could reach out two of my services over this setup perfectly over both opnsenses and haproxy instances. But I created two new containers in my LAN-Network and this two new containers I cant reach over my setup. I dont see any packages that where blocked by the firewall and could not find any differences between the working backendpools and servers and the ones who fails to reach.

The problem ony exist if I establish the connection to my servers over tha backup-opnsense. The HAProxy configuration is created as active-active but in my lan I use IPv4 carp. So the Firewalls are active-passive in general. I also created multiple packet captures and tried to find the problem but without any success. I will add one of my HAProxy configs as example. The other looks nearly the same (expect the peer-settings of course).

The BackendPool for portainer and onlyoffice works as expected via both servers but the connection to the other two backendpools fails. The only irregularity I could find was that the sticktable count for my backendpools where the connection work will increase on both peers but for the backendpools where the connection failed the count only increases on my backup-firewall. It seemed that this sticky table will not be synced to the other peer.

Automatically generated configuration.

Do not edit this file manually.

global
uid 80
gid 80
chroot /var/haproxy
daemon
stats socket /var/run/haproxy.socket group proxy mode 775 level admin
nbthread 8
hard-stop-after 60s
no strict-limits
maxconn 10000
tune.ssl.ocsp-update.mindelay 300
tune.ssl.ocsp-update.maxdelay 3600
httpclient.resolvers.prefer ipv4
tune.ssl.default-dh-param 4096
spread-checks 2
tune.bufsize 16384
tune.lua.maxmem 0
log /var/run/log local0 info
lua-prepend-path /tmp/haproxy/lua/?.lua
localpeer opnsense-1.DOMAIN
cache opnsense-haproxy-cache
total-max-size 4
max-age 60
process-vary off

defaults
log global
option redispatch -1
maxconn 5000
timeout client 30s
timeout connect 30s
timeout server 30s
retries 3
default-server init-addr libc,last
default-server maxconn 5000

autogenerated entries for ACLs

autogenerated entries for config in backends/frontends

autogenerated entries for stats

Resolver: rs_Mgmt_Dns

resolvers 6679d8bd8d0490.33921024
nameserver 127.0.0.1:53 127.0.0.1:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s

Resolver: rs_Web_Dns

resolvers 667a76de2f8575.92867827
nameserver 127.0.0.1:53 127.0.0.1:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s

Resolver: rs_Mail_Dns

resolvers 667a772b465e78.31559580
nameserver 127.0.0.1:53 127.0.0.1:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s

Resolver: rs_Sheeps_Dns

resolvers 667a776fc1b630.05215943
nameserver 127.0.0.1:53 127.0.0.1:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s

Resolver: rs_Labor_Dns

resolvers 667a77a690f6f6.97718965
nameserver 127.0.0.1:53 127.0.0.1:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s

Frontend: ps_SNI (SNIPorts 443/80)

frontend ps_SNI
bind 0.0.0.0:443 name 0.0.0.0:443
bind 0.0.0.0:80 name 0.0.0.0:80
bind :::443 name :::443
bind :::80 name :::80
mode tcp
default_backend bp_SSL
# stickiness
stick-table type binary len 32 size 50k expire 30m peers opnsense-haproxy-peers
tcp-request connection track-sc0 src
# logging options

Frontend: ps_HTTP (HTTPPort 80)

frontend ps_HTTP
bind 127.0.0.1:80 name 127.0.0.1:80 accept-proxy
bind :::80 name :::80 accept-proxy
mode http
option http-keep-alive
option forwardfor
# stickiness
stick-table type binary len 32 size 50k expire 30m peers opnsense-haproxy-peers
tcp-request connection track-sc0 src
# logging options
# ACL: cond_NoSSL
acl acl_665f13c7739b83.54000775 ssl_fc

# ACTION: rule_HttpToHttps
http-request redirect scheme https code 301 if !acl_665f13c7739b83.54000775

Frontend: ps_Https (HTTPSPort 443)

frontend ps_Https
http-response set-header Strict-Transport-Security “max-age=63072000; includeSubDomains; preload”
bind 127.0.0.1:443 name 127.0.0.1:443 accept-proxy ssl curves secp384r1 no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/665f1923f2e993.19432323.certlist
bind :::443 name :::443 accept-proxy ssl curves secp384r1 no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256 alpn h2,http/1.1 crt-list /tmp/haproxy/ssl/665f1923f2e993.19432323.certlist
mode http
option http-keep-alive
option forwardfor
timeout client 15m
# stickiness
stick-table type ip size 50k expire 30m peers opnsense-haproxy-peers
tcp-request connection track-sc0 src
# logging options
# ACL: cond_DOMAINMainDomain
acl acl_669e135c005e40.39178190 hdr_end(host) -i DOMAIN
# ACL: cond_CUST_DOMAIN1Domain
acl acl_66a0033a9ea771.35051049 hdr_end(host) -i CUST_DOMAIN

# ACTION: rule_SubdomainsDOMAIN
use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/665f14a3138776.70554930.txt)] if acl_669e135c005e40.39178190
# ACTION: rule_SubdomainsCUST_DOMAIN
use_backend %[req.hdr(host),lower,map_dom(/tmp/haproxy/mapfiles/66a002da93a445.67801024.txt)] if acl_66a0033a9ea771.35051049

Backend: bp_SSL (SSL Backend pool)

backend bp_SSL
# health checking is DISABLED
mode tcp
balance source
# stickiness
stick-table type ip size 50k expire 30m peers opnsense-haproxy-peers
stick on src
server srv_SSL 127.0.0.1 send-proxy-v2 check-send-proxy

Backend: bp_AcmeChallenge (Acme Challenge Backend Pool)

backend bp_AcmeChallenge
# health checking is DISABLED
mode http
balance source
# stickiness
stick-table type ip size 50k expire 30m peers opnsense-haproxy-peers
stick on src
http-reuse safe
server srv_AcmeChallenge 127.0.0.1:43580

Backend: bp_Portainer (Portainer Backend Pool)

backend bp_Portainer
# health checking is DISABLED
mode http
balance source
# stickiness
stick-table type ip size 50k expire 30m peers opnsense-haproxy-peers
stick on src
http-reuse safe
option forwardfor
server srv_Portainer INTERNALDOMAIN:9443 ssl alpn h2,http/1.1 verify none resolvers 6679d8bd8d0490.33921024

Backend: bp_OnlyOffice (OnlyOffice Backend Pool)

backend bp_OnlyOffice
# health checking is DISABLED
mode http
balance source
# stickiness
stick-table type ip size 50k expire 30m peers opnsense-haproxy-peers
stick on src
http-reuse safe
option forwardfor
server srv_OnlyOffice INTERNALDOMAIN:443 ssl alpn h2,http/1.1 verify none resolvers 6679d8bd8d0490.33921024

Backend: bp_mainwp (MainWP BackendPool)

backend bp_mainwp
# health checking is DISABLED
mode http
balance source
# stickiness
stick-table type ip size 50k expire 30m peers opnsense-haproxy-peers
stick on src
http-reuse safe
option forwardfor
server srv_MainWP INTERNALDOMAIN:443 ssl alpn h2,http/1.1 verify none resolvers 667a76de2f8575.92867827

Backend: bp_CUST_DOMAIN (CUST_DOMAIN BackendPool)

backend bp_CUSTDOMAIN
# health checking is DISABLED
mode http
balance source
# stickiness
stick-table type ip size 50k expire 30m peers opnsense-haproxy-peers
stick on src
http-reuse safe
option forwardfor
server srv_CUSTDOMAIN CUST_INTERNAL_DOMAIN:443 ssl alpn h2,http/1.1 verify none resolvers 667a76de2f8575.92867827

peers opnsense-haproxy-peers
peer PEER1ADDRESS 192.168.98.252:44225
peer PEER2ADDRESS 192.168.98.253:44225

listen local_statistics
bind 127.0.0.1:8822
mode http
stats uri /haproxy?stats
stats realm HAProxy\ statistics
stats admin if TRUE

remote statistics are DISABLED