My SNI routing does not seem to work - is there some option to debug ACL rule?

[Olivier]

Hello all,
I am trying to configure SNI routing using OPNSense HAProxy plugin (based on HAProxy 2.8).

After configuration in the GUI, it gives me this configuration:

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 debug
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats




# Frontend: public_https (HTTPS Port)
frontend public_https
    bind 0.0.0.0:443 name 0.0.0.0:443 
    mode tcp
    default_backend backend_https

    # logging options
    option tcplog

# Frontend: public_http (HTTP Port)
frontend public_http
    bind 0.0.0.0:80 name 0.0.0.0:80 
    mode http
    option http-keep-alive
    default_backend backend_http

    # logging options
    option httplog

# Backend: backend_https (Backend HTTPS)
backend backend_https
    # health checking is DISABLED
    mode tcp
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m  
    stick on src
    # ACL: iot_myserver_org_condition_sni
    acl acl_65a84ebeebab75.87915974 req.ssl_sni -m sub -i iot.myserver.org
    # ACL: server_myserver_org_condition_sni
    acl acl_65a84ed2d27713.21832231 req.ssl_sni -m sub -i server.myserver.org

    # ACTION: iot_myserver_org_https_sni_rules
    use-server iot_myserver_org_server_https if acl_65a84ebeebab75.87915974
    # ACTION: server_myserver_org_https_sni_rules
    use-server server_myserver_org_server_https if acl_65a84ed2d27713.21832231
    server iot_myserver_org_server_https 192.168.9.5:443 
    server server_myserver_org_server_https 192.168.9.10:443 

# Backend: backend_http (Backend HTTP)
backend backend_http
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m  
    stick on src
    # ACL: iot_myserver_org_condition_host
    acl acl_65a851411fc3d3.35763680 hdr_sub(host) -i iot.myserver.org
    # ACL: server_myserver_org_condition_host
    acl acl_65a85167495161.41598361 hdr_sub(host) -i server.myserver.org

    # ACTION: iot_myserver_org_http_host_rules
    use-server iot_myserver_org_server_http if acl_65a851411fc3d3.35763680
    # ACTION: server_myserver_org_http_host_rules
    use-server server_myserver_org_server_http if acl_65a85167495161.41598361
    http-reuse safe
    server iot_myserver_org_server_http 192.168.9.5:80 
    server server_myserver_org_server_http 192.168.9.10:80 



listen local_statistics
    bind            127.0.0.1:8822
    mode            http
    stats uri       /haproxy?stats
    stats realm     HAProxy\ statistics
    stats admin     if TRUE

# remote statistics are DISABLED

When I am trying to access both servers, it always redirect me to the same server (yesterday it actually always redirect me to server.myserver.org while today is iot.myserver.org even I have not changed anything in the conf). I would suspect my ACL rule are not working and it is just doing load balancing.
Both servers are running nginx. I enabled debug logging on both. I can confirm the SNI is sent through the TLS connections.

Example from this morning, I am trying to access curl https://server.myserver.org and on iot.myserver.org nginx logs:

2024/01/18 08:53:27 [debug] 4317#4317: epoll: fd:6 ev:0001 d:00007FB850AF2100
2024/01/18 08:53:27 [debug] 4317#4317: accept on 0.0.0.0:443, ready: 0
2024/01/18 08:53:27 [debug] 4317#4317: posix_memalign: 000055678966F830:512 @16
2024/01/18 08:53:27 [debug] 4317#4317: *2 accept: 192.168.9.254:26612 fd:15
2024/01/18 08:53:27 [debug] 4317#4317: *2 event timer add: 15: 60000:126347578
2024/01/18 08:53:27 [debug] 4317#4317: *2 reusable connection: 1
2024/01/18 08:53:27 [debug] 4317#4317: *2 epoll add event: fd:15 op:1 ev:80002001
2024/01/18 08:53:27 [debug] 4317#4317: epoll del event: fd:6 op:2 ev:00000000
2024/01/18 08:53:27 [debug] 4317#4317: epoll add event: fd:6 op:1 ev:10000001
2024/01/18 08:53:27 [debug] 4317#4317: timer delta: 314684
2024/01/18 08:53:27 [debug] 4317#4317: worker cycle
2024/01/18 08:53:27 [debug] 4317#4317: epoll timer: 60000
2024/01/18 08:53:27 [debug] 4317#4317: epoll: fd:15 ev:0001 d:00007FB850AF24C0
2024/01/18 08:53:27 [debug] 4317#4317: *2 http check ssl handshake
2024/01/18 08:53:27 [debug] 4317#4317: *2 http recv(): 1
2024/01/18 08:53:27 [debug] 4317#4317: *2 https ssl handshake: 0x16
2024/01/18 08:53:27 [debug] 4317#4317: *2 tcp_nodelay
2024/01/18 08:53:27 [debug] 4317#4317: *2 reusable connection: 0
2024/01/18 08:53:27 [debug] 4317#4317: *2 SSL server name: "server.myserver.org"
2024/01/18 08:53:27 [debug] 4317#4317: *2 SSL ALPN supported by client: h2
2024/01/18 08:53:27 [debug] 4317#4317: *2 SSL ALPN supported by client: http/1.1
2024/01/18 08:53:27 [debug] 4317#4317: *2 SSL ALPN selected: http/1.1
2024/01/18 08:53:27 [debug] 4317#4317: *2 SSL_do_handshake: -1
2024/01/18 08:53:27 [debug] 4317#4317: *2 SSL_get_error: 2
(...)

So the SNI seems to be correct (SSL server name: "server.myserver.org")

The corresponding log entry in HAProxy is:

2024-01-18T08:53:27	Informational	haproxy	82.207.235.198:7748 [18/Jan/2024:08:53:27.851] public_https backend_https/iot_codeur_org_server_https 1/0/115 4302 -- 1/1/0/0/0 0/0

It would be nice to have a setting in HAProxy where the ACL rule can be debugged (for instance for each frontend/backend get the result of the rule). At the moment the debug seems to only for custom Lua code: Observability | Logging | Setting Log Levels | HAProxy Enterprise 2.8r1

… And if someone has an idea what is my issue.

[lukastribus]

The configuration is incomplete, you need to wait for the ssl client_hello to be in the buffer, otherwise you can’t reliable route based on the SNI value if sometimes it’s there and sometimes not.

Put this into your https frontend section:

# Wait for a client hello for at most 5 seconds
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }

[Olivier]

Thanks a lot @lukastribus it works

(the setting to debug ACL rules would still be nice and useful )

[lukastribus]

You can log values, for example if you are troubleshooting SNI routing, log SNI to the logs and then you will see that it’s empty.

[Olivier]

Thanks @lukastribus , what is missing to my configuration file to see such logs? I could not see how to trace such things.

[lukastribus]

See:

Extract SNI value for logging in TCP mode

I have already confirmed that this ACL rule works to extract SNI from raw TCP packets. tcp-request inspect-delay 5s tcp-request content accept if { req_ssl_hello_type 1 } acl is_my_domain req.ssl_sni -i www.domain.com tcp-request content capture req.ssl_sni len 100 Note tcp-request content capture req.ssl_sni len 100, my intent is to log the SNI value in access logs, so somehow transmit this information so I can use it in log-format. In log-format, I tried the following but it…

Log SNI in TCP mode

Is it possible to log a request’s SNI in mode tcp? You can access the SNI for routing decisions, so ideally you could access it for logging as well. Currently using version 1.7.8 Ideally something like the following not working config: defaults log global mode tcp balance roundrobin frontend https-in mode tcp tcp-request inspect-delay 3s tcp-request content accept if { req_ssl_hello_type 1 } # ideally could capture the SNI something like this tcp-request content capture req.s…

[Olivier]

Thanks, I also found this blog: Surfacing ACL Actions in HAProxy Logs | by Carlo Mencarelli | Medium