SSL Termination fails when CRL is published

Help!
1

Hi all,

I have HAProxy 1.6.3 on Ubuntu deployed with SSL Termination with client side certificates enabled.

I have realised that everytime time our CA publishes a new CRL users cannot connect. They receive a handshake failure error.

I have to restart HAProxy, without changing anything more, in order for it to work again.

Is that the normal behavior?

Tnaks and best regards,

2

It seems that the CRL publishing is not the problem, at least not only this one. When I try to connect to the service HAProxy asks for the certificate and then the connection fails with google chrome error “ERR_BAD_SSL_CLIENT_AUH_CERT”

I don’t know where the error is, and what exactly produces it. If I restart the HAProxy service it works again.

Part of my config is:


global
        log 127.0.0.1   local0 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3

        # Tune SSL
        tune.ssl.default-dh-param 2048

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        retries 3
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend http_frontend
        bind 10.0.0.233:80
        mode http
        redirect scheme https code 301 if !{ ssl_fc }

frontend https_frontend
        bind 10.0.0.233:443 ssl crt /etc/haproxy/certs/xxx.pem ca-file /etc/ssl/certs/ca.pem verify required crt-ignore-err all crl-file /mnt/CRL/crl.pem
        mode http
        reqadd X-Forwarded-Proto:\ https
        use_backend errors unless { ssl_c_verify 0 }
        default_backend nc_demo_web_servers

backend errors
        mode http
        option http-server-close
        redirect location http://xxxxx/errors/errorcertrevoked.html if { ssl_c_verify 23 }
        redirect location http://xxxxx/errors/errorcertexpired.html if { ssl_c_verify 10 }
        
backend nc_demo_web_servers
        mode http
        balance leastconn
        redirect scheme https if !{ ssl_fc }
        stick-table type ip size 200k expire 30m
        stick on src
        server srv1 10.0.1.211:80 check

Any clue?

Thanks!

3

Anybody?? :frowning:

4

Provide the output of haproxy -vv please.

5

Hi, here you have:

$ haproxy -vv
HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2g-fips  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

This output is displayed when HAProxy is working right.

Thanks.

6

Based on the informations provided it’s hard to tell what’s wrong.

  • when you disable CRL completely, does it work?
  • try with “ca-ignore-err all”
  • try upgrading a a more recent build (1.6.3 is quite old) - use latest 1.7 or 1.8 rebuild - see https://haproxy.debian.net
7

I’m going to upgrade to 1.8 stable first of all and let you know…

thanks,

8

I have updated to 1.8 version, and the problem persists. The CA published a new CRL yesterday and I cannot access now with same error.

This is what I can see on haproxy.log:

Apr	9	09:41:04 localhost haproxy[2988]: 10.239.0.1:16660 [09/Apr/2018:09:41:04.642] https_fronted/1: SSL client CA chain cannot be verified
Apr	9	09:41:04 localhost haproxy[2988]: 10.239.0.1:16660 [09/Apr/2018:09:41:04.642] https_fronted/1: SSL client CA chain cannot be verified
[...]
Apr	9	09:42:02 localhost haproxy[2988]: 10.0.0.133:33228 [09/Apr/2018:09:42:01.847] https_fronted/1: SSL handshake failure
Apr	9	09:42:02 localhost haproxy[2988]: 10.0.0.133:33228 [09/Apr/2018:09:42:01.847] https_fronted/1: SSL handshake failure

It seems a problem with CA chain, but if I restart haproxy service, without changing anything, it starts to work again…

it’s fustrating…

9

Ok, please follow the rest of the steps I proposed.

10

Hi,

the “ca-ignore-err all” is already set.

I’m triyng to remove CRL checking, but that’s a point I don’t want to because I need HAProxy to check the CRL…

11

No, ca-ignore-err is not set, crt-ignore-err is set. There’s a difference.

Completely aware that the entire point here is to check CRL’s, but in troubleshooting we need to track the issue down to a specific feature and code path, to be able to do anything about that.

By the way: a CRL has a very limited lifetime, usually something like 24 hours. Are you sure haproxy is often enough reloaded with a uptodate CRL so that the CRL haproxy uses does not expire? That’s would have exactly the effect you are describing here.

12

Sorry for the mistake…

In order for the error to comes up quickly I have published the CRL every 2 hours. Let me do some changes and investigation and I’ll keep you informed.

Thanks for your help.

13

Hi lukastribus,

I have read in some forums that HAProxy needs to be restarted/reloaded everytime the CRL file is changed because it loads at the time it’s started and does not look for changes.

If this is the normal behavior my problem is that, I think, is it that way?

Regards,

14

Yes, the CRL file is only read when haproxy is started. When you update the CRL file, you need to reload haproxy as well, otherwise the changes will never be applied and the CRL file will expire.

15

Thank you, this was the problem from the beginning…

Thanks so much for your support

16

@lukastribus Is this always the case in hversion 2.4 or 2.5dev ?

17

Yes, this is still the case.