Haproxy SSL old cert is visible

Help!
1

Hi,
I spent all day with this issue, no luck.
I have 10 containers on my server, haproxy is sitting in one of them.
I redirected all the :80 and 443 traffic to this haproxy container, and all worked well.

All pem files are placed in the /haproxy/cert folder by cat:
cat /etc/letsencrypt/live/subdomain.domain.tld/fullchain.pem /etc/letsencrypt/live/subdomain.domain.tld/privkey.pem > /etc/haproxy/certs/subdomain.domain.tld

When my SSL certs were freshly created, everything was fine. Now one of them expired, so I tried to renew it. All went well, the pem file is in the right place. No finding from the old pem file.
If I take the new pem file, it validates as new, valid SSL cert.

But when I open my page, I can see the expired SSL next to my URL and the expired cert warning.

I tried:

  • replace the certificate
  • stop / start / reload
  • refresh apache
  • reboot all servers / containers
  • no old haproxy running in the background
  • deleted cert, added new
  • checked, no invisible .pem file anywhere in my cert folder

The old expired cert is still visible and can’t get rid of it. Even if I delete with certbot.
EVEN if I delete all cert!!!

As I said, all the other 8 containers work with the SSL, just this one, which I renewed doesn’t.

-vv:
HA-Proxy version 1.8.8-1ubuntu0.10 2020/04/03
Copyright 2000-2018 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fdebug-prefix-map=/build/haproxy-RAYurj/haproxy-1.8.8=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

I appreciate any help.

2

Sounds like you checked everything there is to consider on this specific container.

Haproxy doesn’t do any fancy stuff. And it certainly cannot remember an old certificate.

A few questions:

  • are you positive the traffic you are sending to this website is hitting this specific container?
  • If you shutdown the entire container, is the website completely unreachable or does it still connect?
  • what is the output of curl -vv https:// from the same host/container, what is the output from the far end?
  • can you double check there is no TLS intercepting device between your client and the container (FW or other security devices)?
3

Hi lukasstribus!
Thanks for replying.
I tried to submit a question only when I’m out of alternatives.

This is the curl:

ubuntu@automate2c1_ov-4b987c:~$ curl -vv https://domain.tld

  • Rebuilt URL to: https://domain.tld/
  • Trying 45.157.190.89…
  • TCP_NODELAY set
  • Connected to domain.tld (45.157.190.89) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Unknown (8):
  • TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (OUT), TLS alert, Server hello (2):
  • SSL certificate problem: certificate has expired
  • stopped the pause stream!
  • Closing connection 0
    curl: (60) SSL certificate problem: certificate has expired
    More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Now in this output I see the cert path is:
CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs

Is this the path for my SSL? It should not be :slight_smile: Is that the error?

Yes, the container is this one for sure.
maybe the container has an SSL cert somewhere that overwrites the old one? I looked for pem files, in the container but no relevant was found.

4

The location from where you are running the curl output matters very much, so please clarify from WHERE do you actually ran that command.

Also run the curl -vv from the other location as asked above and clarify the question about TLS interception.

None of this has anything to do with haproxy, so the details about your environment do matter very much. I won’t be able to tell you “sure, this flag X in configuration line Y in haproxy”, because this is not a haproxy related problem, but a problem with your environment.

So you are accessing IP 45.157.190.89 on port 443 for this. Please go to the box which is listening to port 443 on that IP and issue a sudo netstat -tulpen request and track down the PID bound to port 443. Also issue the same curl -vv on that box.

No and no. It’s where curl is looking for root CA files. It has nothing to do with your problem.

5

Hi,
Thanks for getting back to me.
You actually solved it to me by saying: “Haproxy doesn’t do any fancy stuff. And it certainly cannot remember an old certificate.”
That was the key to solve the issue.

In short: “user error”.
In long:
I reinstalled the haproxy container, reissued the certificate for that one container that was failing and everything worked. The other containers gave me an error, since I didn’t remake those certs. But I also noticed, that I’m getting the first container’s cert under each container.

Than it hit me:
Altough my certs were saved with a domain name, what If one of those domain names were wrong, and one of those pem files contained the expired certificate of the domain in question under another domain name. Normally I would notice this, but when we move domain containers between servers, we don’t automatically clean out the certs.

So thank you for pointing me to the right direction.