Random routing issue (tcp, sni, http/2)

Help!
1

Hello,

I have a hub app routed, and sometimes it gets wrongly routed to the ssl_default backend. Leading to a 404. A control + F5 is needed to get it right, which cannot be done for resources files, like javascripts. Here is my configuration :

File: /etc/haproxy/hosts.map

hub.domain.com		hub_server

File: /etc/haproxy/haproxy.cfg

global
    log /dev/log    local0
    log /dev/log    local1 notice

    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
    log    global 
    mode   tcp
    option dontlognull
        
    option tcplog
    option    log-health-checks
        
    timeout connect 5s
    timeout client  50s
    timeout server  50s

    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend http-in
    bind 10.0.16.10:80
    mode http
    option httplog
    http-request redirect scheme https if !{ ssl_fc }
    # redirect scheme https code 301 if !{ ssl_fc }

frontend https-in
    bind 10.0.16.10:443
    mode tcp

    tcp-request inspect-delay 5s

    #tcp-request content capture req.ssl_sni len 25
    #log-format "%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq ssl_fc_has_sni '%[ssl_fc_has_sni]' sni:'%[capture.req.hdr(0)]'"

    tcp-request content accept if { req_ssl_hello_type 1 }

    use_backend %[req.ssl_sni,lower,map(/etc/haproxy/hosts.map,ssl_default)]
    # default_backend ssl_default

backend ssl_default
    mode tcp
    
    option tcplog

    # maximum SSL session ID length is 32 bytes.
    stick-table type binary len 32 size 30k expire 30m

    acl clienthello req_ssl_hello_type 1
    acl serverhello rep_ssl_hello_type 2
    
    # use tcp content accepts to detects ssl client and server hello.
    tcp-request inspect-delay 5s
    tcp-request content accept if clienthello
    
    # no timeout on response inspect delay by default.
    tcp-response content accept if serverhello
    stick on payload_lv(43,1) if clienthello

    # Learn on response if server hello.
    stick store-response payload_lv(43,1) if serverhello
    option ssl-hello-chk
        
    server nginx-local 127.0.0.1:4443 send-proxy-v2 on-marked-up shutdown-backup-sessions weight 11 check inter 1s fall 3 rise 2
    server server-hub-backup 127.0.0.1:4503 send-proxy-v2 backup non-stick

backend hub_server
    mode tcp
    # maximum SSL session ID length is 32 bytes.
    stick-table type binary len 32 size 30k expire 30m
    
    option tcplog
    option log-health-checks
        
    acl clienthello req_ssl_hello_type 1
    acl serverhello rep_ssl_hello_type 2

    # use tcp content accepts to detects ssl client and server hello.
    tcp-request inspect-delay 5s
    tcp-request content accept if clienthello

    # no timeout on response inspect delay by default.
    tcp-response content accept if serverhello
    stick on payload_lv(43,1) if clienthello

    # Learn on response if server hello.
    stick store-response payload_lv(43,1) if serverhello
    option ssl-hello-chk

    server server-hub1 hub.lxd:4443 send-proxy-v2 weight 11 check inter 1s fall 3 rise 2
    server server-hub2 10.97.89.14:4443 send-proxy-v2 on-marked-up shutdown-backup-sessions weight 22 check inter 1s fall 3 rise 2
    server server-hub-backup 127.0.0.1:4503 send-proxy-v2 backup non-stick

The certificates delivered by the backends are wildcard. For the ssl_default backend:

Common name: domain.com
SANs: *.domain.com, *.office.domain.com, domain.com
Valid from January 17, 2021 to April 17, 2021
Issuer: R3

For the hub_server backend

Common name: domain.com
SANs: *.domain.com, domain.com
Valid from January 3, 2021 to April 3, 2021
Issuer: R3
2

You have overlapping SSL certificates, which causes this exact issue.

Once a browser goes to ssl_default backend, it will keep using this connection because the certificate is valid for domain.com, *.domain.com, *.office.domain.com, and domain.com. What you have in /etc/haproxy/hosts.map doesn’t matter at this point, the browser can’t know about this.

You need to avoid overlapping certificates here.

3

Thanks in fact it first uses the right backend and then randomly switches to the backup for some requests.

Do both certificates need to have no wildcard ? Because ssl_default has the same backup server as hub_server, its certificate should at least contains the domains server-hub1 and server-hub2 have.

4

Yes, the symptom is your are connecting to something that belongs on backend X, which works and then the browser needs something from backend Y, but it will reuse the same session that still is going to backend X, because the certificate on backend X covers the hostname that according your rules would actually go to backend Y.

They can’t overlap is the point, this is not necessarily related to wildcard certificates, it’s just easier to hit this issue with them.

The point is there be a hostname a browser would use, for which the certificates in both backends are valid.