mTLS by path - requiring client verification for some paths

Help!
1

Hi,

I’ve configured HAProxy such that client verification is required for HTTP requests to all paths except /ping. Here’s my (boiled down) configuration, somewhat derived from this post:

global

defaults
  mode              http
  timeout client    2m
  timeout connect   2m
  timeout server    2m

frontend the_frontend
  bind :443 ssl crt server-combined.pem ca-file ca.pem verify optional

  acl ping_page path /ping
  http-request deny if !{ ssl_fc_has_crt } !ping_page
  default_backend the_backend

backend the_backend
  server origin unix@origin.sock

Unfortunately, it seems that because of verify optional, clients that could authenticate sometimes establish an SSL session without authenticating, and this session is subsequently reused, causing the client to receive 403 responses for every path except /ping.

Turning off SSL caching with tune.ssl.cachesize 0 seems to fix the problem, but at the cost of a large amount of SSL negotiation.

I’d be very grateful for any advice on a better solution. I’m actually only permitting unverified clients because I need to accommodate health checks from a piece of upstream infrastructure that’s incapable of supplying a client certificate; I require all ‘genuine’ clients to be verified. I’m aware that choosing whether to require a certificate in the manner described here isn’t possible on the basis of path because the path isn’t available until after the SSL negotiation is complete, but I wonder if there might be an approach whereby the infrastructure health checks can be distinguished from regular traffic in some other way… a SNI kludge maybe?

If it’s relevant, haproxy -v returns:

HA-Proxy version 2.2.8-7bf78d7 2021/01/13 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.8.html
Running on: Linux 5.4.0-65-generic #73~18.04.1-Ubuntu SMP Tue Jan 19 09:02:24 UTC 2021 x86_64
2

That’s sounds like a bug.

This makes me think even more that we are hitting some kind of bug - either in haproxy, openssl or on the client side.

Does the clients where this happens have anything in common (like OS or browser)? Can you provide the output of haproxy -vv (I want to see all the openssl data)?

Yeah I guess a dedicate hostname for healthchecks could be a workaround, but I’d rather clarify the root cause and fix a underlying bug.

3

Thanks for the quick response.

The affected clients certainly include (and may be limited to) Chrome 86.0.4220.111 on Windows 10 Enterprise 1809 - the site is internal to an organisation where the environment is fairly closely controlled.

Of note is the fact that the client certificates are sourced from a smartcard. As I understand it, smartcard transactions on Windows are serial, and from experience, calls to the smartcard sometimes appear to get ‘backed up’. I don’t know if under those circumstances, Chrome might choose not to authenticate.

Regarding the OpenSSL details, I observed the issue on HAProxy 2.2.5 using OpenSSL_1_1_1h.

I’m now running HA-Proxy version 2.2.8-7bf78d7 2021/01/13 with statically linked OpenSSL 1.1.1i 8 Dec 2020, but have had tune.ssl.cachesize 0 set since the upgrade. I’ll flip the config to use the default SSL cache size and let you know if the problem also occurs on this version. Let me know if you need more details from the output of haproxy -vv.

4

Do you also have no-tls-tickets in your configuration, and, can you reproduce the problem when you remove that but leave tune.ssl.cachesize 0 in there?

5

I do have no-tls-tickets - the SSL configuration is verbatim from the Mozilla SSL Configuration Generator. I’ll remove it and leave the tune.ssl.cachesize 0 now.

6

Replace ssl_fc_has_crt with ssl_c_used :

ssl_fc_has_crt : boolean

  Returns true if a client certificate is present in an incoming connection over
  SSL/TLS transport layer. Useful if 'verify' statement is set to 'optional'.
  Note: on SSL session resumption with Session ID or TLS ticket, client
  certificate is not present in the current connection but may be retrieved
  from the cache or the ticket. So prefer "ssl_c_used" if you want to check if
  current SSL session uses a client certificate.
7

Good spot! I’ll change that over now, and report back.

Thanks!

8

On the basis of five days’ running, this has solved the problem.

Many thanks @lukastribus for the rapid and accurate advice!

1 Like