Intermittent wrong fallback cert served on TCP-mode SNI passthrough (3.4.4), even with recent SNI/session-resumption fixes included

Help!
1

HAProxy version: 3.4.4 (also reproduced earlier on 3.2.23 before upgrading)
OS: Ubuntu 24.04/26.04, installed via vbernat’s PPA
Setup: TCP-mode frontend doing SNI-based passthrough routing to a mail server (SmarterMail), with an HTTP-mode local-termination fallback for everything else.

Relevant config:

frontend fe_tcp_443
    mode tcp
    option tcplog
    bind *:443
    tcp-request inspect-delay 5s
    tcp-request content accept if { req.ssl_hello_type 1 } { req.ssl_sni -m found }

    acl sni_mail req.ssl_sni -i mail.example.com autodiscover.example.com webmail.example.com
    [... ~20 more domain lines ...]

    use_backend be_mail_passthrough if sni_mail
    default_backend be_local_terminate

backend be_mail_passthrough
    mode tcp
    server sm1 <mail-server-ip>:443

backend be_local_terminate
    mode tcp
    server local_ssl 127.0.0.1:8443

Symptom: Clients (mail apps on mobile) very occasionally get a TLS identity warning when connecting to one of the SNI-routed mail domains — the certificate presented is for a different, unrelated domain served via be_local_terminate’s cert store (i.e. traffic appears to fall through to default_backend instead of matching sni_mail). Rare and intermittent; users report it correlates with their phone waking from idle/sleep and the mail app reconnecting.

What we’ve tried:

  • Added { req.ssl_sni -m found } to the tcp-request content accept condition, on the theory HAProxy was sometimes committing to a backend before the SNI extension had fully arrived in a fragmented/delayed ClientHello.

  • Increased tcp-request inspect-delay from 5s to 10s on one instance.

  • Upgraded 2.8 → 3.2 → 3.4.4. Checked the 3.4.4 changelog directly and confirmed it already includes:

    • BUG/MEDIUM: ssl: Fix unprotected 'ssl_sock_choose_sni_ctx' calls
    • BUG/MEDIUM: ssl: isolate TLS session resumption per X509 server certificate
    • BUG/MINOR: ssl: isolate TLS session resumption per crt-list filter
    • BUG/MEDIUM: ssl: isolate TLS session resumption per authentication policy

    These looked like exactly our issue (cert-selection race + resumption reusing the wrong cert context), but the symptom has still recurred on 3.4.4.

Diagnostics: Ran a continuous full-packet tcpdump/tshark capture (systemd service, ring-buffered) watching for TLS ClientHellos on port 443 matching our mail domains, to catch a live occurrence. In every capture window checked so far (multiple hours, hundreds of real client connections across several domains), every single ClientHello with a matching SNI was correctly relayed to the mail backend within milliseconds — no missing SNI, no mismatched SNI, no unrelayed connections. We have not caught a genuine anomaly in the wire traffic corresponding to a reported failure.

Questions:

  1. Given the relevant SNI-ctx/session-resumption fixes are already in 3.4.4, is there a different known issue in this version range that could still cause TCP-mode req.ssl_sni ACL matching to intermittently miss despite a well-formed, correctly-timed ClientHello?
  2. Given our packet captures show clean traffic every time we’ve checked, is it plausible this is actually a client-side (iOS Mail) stale/cached TLS-error notification being resurfaced on app-foreground, rather than a live HAProxy routing failure? Are there known client behaviors that would produce this without any corresponding bad packet on the wire?
  3. Is there a more definitive way to log, per-connection, which backend HAProxy actually selected and why (beyond raw tcpdump), to correlate with client reports after the fact?

Any insight — especially from anyone who’s seen this exact “clean captures, fixes already included, still occasional wrong cert” pattern — would be appreciated.

2

The topic is rather complex, I will have to challenge some of your assumptions.

Did you see this with your own eyes, or at least from a screenshot? How do you come to the conclusion that the wrong certificate is served exactly?

A SSL error can be caused by lots of things, it doesn’t necessarily have to be the wrong cert.

Those changes are unrelated to what you are suggesting the issue is:

Those changes are related to SSL (termination) behaviour in Haproxy. However you have indicated that you have an issue with SSL passthrough, meaning no SSL is involved at all in haproxy.

The config you posted could be run in Haproxy 1.4 that has not SSL support at all.

req.ssl_sni accesses a offset in the client_hello to extract SNI. Afterwards the entire TCP payload is forwarded.

That’s it. There is no certificate selection involved here, it is reading a string and acting upon it.

A couple of hours is probably not enough when the issue is happening “very occasionally”.

You will have to capture everything for days or weeks and have your customers report the exact timestamp, and then check that particolar session matching the timestamp.

Again, the SSL stack has nothing to do with this, but to answer your question: no, there is not. SNI matching has it’s limitis though, as explained in the docs (see below).

I don’t recall specific client behaviors like this, but you have to find out more.

Comparing customer reported timestamp with permanent traces.
Have them screenshot the actual error.
Have them provide additional context information (location, network, etc).

You can log the value of SNI and the selected backend should be in the logs by default.

SNI parsing does have it’s limitis. See the documentation:

for the reasons detailed
below (HelloRetryRequest, Renegotiation, Encrypted Client Hello), the value
returned by this fetch is not reliable enough to be used alone for allowing
or denying access to certain hosts.

This fetch only parses the first ClientHello message found in the request
buffer. If the client sends several ClientHello messages within the same
TCP stream, for instance because the server requested a HelloRetryRequest
(HRR) as part of TLS 1.3, or because the client initiates a TLS
renegotiation (which sends a new ClientHello later in the same TCP stream,
possibly carrying a different SNI), only the SNI carried by that very first
ClientHello will be returned, the content of any subsequent ClientHello
will be ignored.

When Encrypted Client Hello (ECH) is used, the ClientHello seen on the wire
is only the “Outer” ClientHello, which embeds the real, encrypted “Inner”
ClientHello. The SNI extracted by this fetch in that case is the one from
the Outer ClientHello, which is a decoy SNI and not the actual host the
client intends to reach. This fetch is currently not able to decrypt nor
analyze the Inner ClientHello, so it must not be relied upon to make
routing or access control decisions when ECH is in use.

You can try to disable TLSv1.3 where you terminate SSL (127.0.0.1:8443), to make sure it’s none of the TLSv1.3 features mentioned in the docs are causing this.

This was recently documented in:

3

Thanks for the detailed reply — really helpful, and you’re right that I’d conflated the wrong things.

To confirm: no, I never actually verified the wrong cert with my own eyes at the OS/TLS level — only from a couple of end-user screenshots (iOS “Cannot Verify Server Identity” dialogs). Fair point that an SSL error could be many things, and I shouldn’t have assumed cert-selection specifically without checking closer to the wire.

You’re also right that I mixed up two separate code paths — the SSL termination fixes in 3.4.4’s changelog apply to our fe_https bind (which does real cert selection via ssl crt <dir>), not the TCP-mode passthrough frontend where the actual reported issue lives. That passthrough is exactly as you described: read an offset, match a string, forward raw bytes. No cert logic involved at all. Good catch.

On your suggestions, here’s what I’ve done:

  1. Persistent SNI logging — added a custom log-format to the passthrough frontend using tcp-request content set-var(sess.sni) req.ssl_sni (had to use session scope, not txn scope, since txn gets torn down before the final TCP-mode log line is written) plus %[var(sess.sni)] in the format string. Every connection is now permanently logged with the actual parsed SNI and selected backend — no more relying on short capture windows.
  2. Ran it over a real ~6.5 hour production window (several thousand connections) after the fix. Every single mail-domain SNI in that window correctly landed on the mail passthrough backend — zero mismatches. So the passthrough logic itself appears to be working correctly, consistently, under real load.
  3. Given that, and given the correlation with users waking their phones/reconnecting that we’d noticed, I’m treating the client-side stale/cached-notification theory as the more likely explanation for now, rather than continuing to assume it’s a live routing failure.

Next step is just watching this log now that it’s finally accurate, and grabbing the exact timestamp next time a user reports it, to see directly what (if anything) was actually happening on the wire at that moment — rather than reconstructing after the fact.

Appreciate the redirect — this was a much more productive path than where I was headed.