Renew Cert updates but old cert being used?

rayj00 · March 2, 2024, 7:35pm

I am using Haproxy to forward sub-domains to the proper LXD containers. I have multiple containers configured as websites. The method I use to renew LetsEncrypt certs is below and works fine for other sub-domains, but for this one (test.streamingworld.us), the expired cert is used? The cert does renew but the older, expired cert is always used, causing SSL to fail.

certbot certificates yields:
Certificate Name: test.streamingworld.us
Domains: test.streamingworld.us
Expiry Date: 2024-05-27 20:30:36+00:00 (VALID: 86 days)
Certificate Path: /etc/letsencrypt/live/test.streamingworld.us/fullchain.pem
Private Key Path: /etc/letsencrypt/live/test.streamingworld.us/privkey.pem

Here is the haproxy.cfg file:

global
    log 127.0.0.1 syslog debug
    lua-load /etc/haproxy/cors.lua
    maxconn 2000
    tune.ssl.default-dh-param 2048
    user haproxy
    tune.maxrewrite     4096
    group haproxy

    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM

defaults
    log global
    mode http
    option forwardfor
    option http-server-close
    option httplog
    option dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    timeout tunnel  2h  #this is for websocket connections, 2 hours inactivity timeout
    timeout client-fin 5000
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http


frontend http_lb
  bind *:80
  bind *:443 ssl crt /etc/haproxy/certs/
  #redirect scheme https if !{ ssl_fc } # Added code 301
  http-request redirect scheme https unless { ssl_fc }

 acl testbox hdr(host) -i test.streamingworld.us

  acl acl_letsencrypt path_beg /.well-known/acme-challenge/
  use_backend be_letsencrypt if acl_letsencrypt

 use_backend testenv if testbox

backend be_letsencrypt
        server letsencrypt 127.0.0.1:8888

 backend testenv
       balance leastconn
       http-request set-header X-Client-IP %[src]
      redirect scheme https if ! { ssl_fc }
      server testing  10.90.200.247:80 check verify none

I am also posting this to the Letsencrypt forum also.

Your suggestions are much welcome.

Ray

mrit · March 5, 2024, 7:50am

have you done a restart of haproxy to “load” the new cert?
did you link to /etc/letsencrypt/live/… or have you copied the certfiles into the haproxy conf dir
have you checked the certfile with openssl to display the cert?

openssl x509 -in example.crt -text -noout

rayj00 · March 8, 2024, 2:05pm

Yes I restarted Haproxy. I copy the cert into the /etc/haproxy/certs folder.

From sudo find / -name *.pem -print: There are literally thousands of .pem files on my haproxy server in the following folders:

/etc/letsencrypt/archive/
/etc/letsencrypt/csr/
/etc/letsencrypt/keys/
/etc/ssl/certs
/usr/lib/python3/dist-packages/twisted/

/etc/letsencrypt/live/test.streamingworld.us/fullchain.pem
/etc/letsencrypt/live/test.streamingworld.us/chain.pem
/etc/letsencrypt/live/test.streamingworld.us/privkey.pem
/etc/letsencrypt/live/test.streamingworld.us/cert.pem

I do:

certbot certonly --cert-name test.streamingworld.us -d test.streamingworld.us
cd test.streamingworld.us
cat fullchain.pem privkey.pem > /etc/haproxy/certs/test.streamingworld.us.pem

The certs for my websites are in /etc/haproxy/certs/

root@HAProxy:/etc/letsencrypt# sudo find / -name test.streamingworld.us.pem -print
/etc/haproxy/certs/test.streamingworld.us.pem
find: ‘/sys/kernel/debug’: Permission denied
find: ‘/sys/fs/pstore’: Permission denied
find: ‘/sys/fs/fuse/connections/50’: Permission denied
find: ‘/dev/.lxd-mounts’: Permission denied
find: ‘/proc/sys/fs/binfmt_misc’: Permission denied
find: ‘/proc/tty/driver’: Permission denied

Since haproxy is terminating SSL/TLS, how is the security propagated to the browser when haproxy forwards the request to the website?

What other info should I post?

Thanks,

Ray

lukastribus · March 8, 2024, 2:25pm

Check that /etc/haproxy/certs/ only contains only the current certificates you want. If there are older certificates in there, haproxy will load and use them.

Also check that there are no other haproxy instances running. Sometimes it happens that old instances (with old configurations and certificates) are still running in the background.

To do this, stop haproxy completely, kill all remaining haproxy process and then start haproxy again.

systemctl stop haproxy
killall haproxy
systemctl start haproxy

rayj00 · March 8, 2024, 4:39pm

I only have certs in /etc/haproxy/cert that I use on my other websites, 13 other .pem files that are named as such: [subdomain].streamingworld.us.pem. In the case of the failing cert, test.streamingworld.us.pem

I tried the stop/killall/start after which I seen this:
root 8166 1 0 16:33 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

haproxy 8167 8166 0 16:33 ? 00:00:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid

Are these two instances OK?

lukastribus · March 8, 2024, 4:39pm

Yes, those two instances are OK

rayj00 · March 8, 2024, 5:01pm

So I am totally puzzled why this is happening?

lukastribus · March 8, 2024, 5:15pm

mrit already suggested:

openssl x509 -in /etc/haproxy/certs/test.streamingworld.us.pem -text -noout

Which shows the certificate on disk.

Compare it with the output from haproxy, but not from any other place, use the haproxy box pointing to 127.0.0.1 and specifying the servername.

echo | openssl s_client -servername test.streamingworld.us -connect 127.0.0.1:443 2>/dev/null | openssl x509 -text -noout

Post the full outputs here.

rayj00 · March 8, 2024, 10:38pm

Here ya go.

root@HAProxy:~# openssl x509 -in /etc/haproxy/certs/test.streamingworld.us.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:b7:d2:d1:40:d9:ac:b2:f6:15:d1:82:53:fa:fb:e3:2a:22
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Feb 27 20:30:37 2024 GMT
            Not After : May 27 20:30:36 2024 GMT
        Subject: CN = test.streamingworld.us
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:99:16:db:8b:c8:df:c7:13:6b:c9:d1:72:a5:67:
                    57:cf:ae:70:11:e9:e5:12:8f:45:10:8a:64:d6:27:
                    fc:d6:60:36:43:f9:8c:5a:d9:7c:7a:b1:b2:de:b2:
                    30:e7:7d:2d:29:8c:54:24:27:eb:eb:11:56:ee:a6:
                    79:8a:94:8e:2e:fd:4e:aa:f8:40:01:0d:00:02:48:
                    e7:31:10:99:46:3e:f6:da:e7:77:b9:f9:a3:8e:c1:
                    31:2f:ab:18:00:a4:7f:02:38:60:f9:32:c8:b9:7e:
                    da:b9:c4:88:ca:83:d3:d2:98:0d:9d:92:a0:cb:a3:
                    1f:3a:b0:0e:8b:65:74:6a:3a:c1:46:2e:11:9e:c4:
                    d4:01:ff:6f:f9:0c:69:05:83:d5:08:82:79:34:94:
                    9a:da:81:6c:06:bf:41:e3:f7:f5:e1:fc:28:d8:f7:
                    77:7a:b5:60:47:d5:45:ab:af:26:3e:e6:e8:ab:57:
                    84:4f:19:0b:74:8f:1e:3d:e7:be:aa:2e:90:ac:16:
                    f6:db:ae:3a:fe:89:48:c3:28:a8:63:ba:67:a2:7b:
                    25:9d:16:68:8f:14:ab:6f:fc:bc:22:77:37:0f:d9:
                    1e:44:d8:b7:7b:67:9d:cd:55:5f:52:63:a7:e7:cd:
                    07:70:d5:c3:b1:b4:65:d6:4d:4b:e3:69:f8:7f:c5:
                    0c:c9
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                0F:BD:11:4E:14:6D:5F:B2:E6:16:08:99:7E:D2:10:0B:78:F5:1F:BA
            X509v3 Authority Key Identifier:
                keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6

            Authority Information Access:
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/

            X509v3 Subject Alternative Name:
                DNS:test.streamingworld.us
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 3B:53:77:75:3E:2D:B9:80:4E:8B:30:5B:06:FE:40:3B:
                                67:D8:4F:C3:F4:C7:BD:00:0D:2D:72:6F:E1:FA:D4:17
                    Timestamp : Feb 27 21:30:37.223 2024 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:81:1B:35:69:D1:A9:42:A7:04:28:DE:
                                8B:ED:79:F0:07:4A:45:1F:50:AB:79:FC:D7:8D:EA:D1:
                                49:51:33:83:DA:02:20:7D:19:17:1F:D6:79:D0:D4:B6:
                                A5:AB:FB:DF:C3:D3:3B:40:93:44:60:97:2B:FB:AE:B2:
                                60:D4:90:28:FF:0E:60
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : EE:CD:D0:64:D5:DB:1A:CE:C5:5C:B7:9D:B4:CD:13:A2:
                                32:87:46:7C:BC:EC:DE:C3:51:48:59:46:71:1F:B5:9B
                    Timestamp : Feb 27 21:30:37.183 2024 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:A1:04:2C:CA:F8:48:BD:8B:C9:5E:41:
                                95:28:B8:0B:B4:39:78:EE:FF:FA:3C:9C:D4:CE:A0:9E:
                                DE:0A:B2:90:2D:02:20:66:DC:38:FA:62:35:6C:22:8A:
                                A5:3C:CE:7C:18:AD:D7:CC:D4:C6:55:B3:38:EA:0A:15:
                                D1:F3:56:FE:BD:87:83
    Signature Algorithm: sha256WithRSAEncryption
         9b:22:e4:f8:cf:c4:20:60:45:b3:2b:01:6a:b5:08:4e:cc:b8:
         91:33:45:a8:cb:8f:7c:35:2a:99:48:8d:22:f1:a5:f5:d9:88:
         1b:07:7c:d2:24:1c:63:c6:e5:e0:b6:6f:78:2b:d3:43:a9:96:
         49:8d:d0:f1:51:e2:74:b6:33:9b:64:ce:bd:09:97:7c:67:66:
         98:ae:cd:6b:2d:f8:fd:7b:ba:5a:22:7d:e4:6d:cf:70:60:fa:
         5f:44:c1:6a:34:14:7e:35:41:84:e4:40:25:3b:ab:12:6d:98:
         88:c8:ee:d5:15:0c:79:68:71:4a:11:e9:4f:5a:8c:38:ba:4d:
         04:a9:89:81:7e:c1:c5:3e:37:b4:03:a5:8f:23:da:b5:f4:32:
         96:d8:ec:12:7c:2b:4b:46:36:84:1f:8e:58:8a:22:32:d1:2b:
         98:7b:05:03:60:c2:45:cf:30:0b:88:8f:80:cd:e7:d6:eb:a5:
         5a:5f:2c:7c:a8:37:d7:fa:83:eb:20:29:b3:4c:53:44:27:bf:
         6e:28:14:cb:fa:56:0c:5e:55:45:c0:97:7b:72:19:e5:84:47:
         26:67:7a:3b:10:15:02:be:90:46:b3:b7:b4:fd:34:eb:82:7d:
         b9:d7:8d:d9:11:bc:52:14:c9:78:3e:8a:44:82:80:6d:0b:e2:
         ae:61:eb:d3

root@HAProxy:/tmp# echo | openssl s_client -servername test.streamingworld.us -connect 127.0.0.1:443 2>/dev/null | openssl x509 -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            03:0b:e4:14:d4:4d:fe:aa:8e:44:24:c8:71:95:f6:5d:2c:91
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Dec 26 06:18:49 2021 GMT
            Not After : Mar 26 06:18:48 2022 GMT
        Subject: CN = test.streamingworld.us
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:a3:61:d3:2c:b9:1f:a3:98:ea:dd:6e:7d:cc:f3:
                    bb:8a:ea:03:8e:5d:c6:14:37:d8:2f:69:70:88:a4:
                    3f:a9:ed:ca:87:d7:f5:05:9e:37:27:f2:52:74:08:
                    80:f9:80:10:d2:8a:4f:86:13:a9:ef:e9:c5:f4:40:
                    a9:4b:9f:a1:18:e1:3b:62:b8:db:c4:c0:8b:3a:7e:
                    e7:83:d5:fc:e4:7d:c2:35:d4:76:86:f0:ca:44:1d:
                    ad:62:82:a6:29:19:94:df:8c:66:f2:47:59:73:37:
                    d8:5e:b9:af:2f:39:f1:58:d9:1a:17:5f:29:5f:bb:
                    f7:76:43:c9:ae:a8:82:d3:45:1f:76:49:30:2b:f6:
                    ff:40:d1:bf:b4:6e:4e:b9:74:92:97:ef:54:da:c5:
                    1d:9b:a6:f8:16:2b:fd:f8:90:99:9a:16:d3:5a:b6:
                    1e:bc:48:1c:85:6c:3d:aa:5b:47:af:96:48:5f:ab:
                    7b:fb:bf:1a:e9:8b:a0:6a:94:dc:e4:00:a8:ca:7a:
                    d4:a3:80:a6:d4:37:8a:7a:79:9d:27:2b:27:f8:76:
                    fa:61:60:58:b1:47:a0:50:21:25:dd:f8:8c:b9:f8:
                    d5:9b:11:e2:fd:73:5a:c9:a9:a3:a8:01:3d:a4:f2:
                    9d:72:d6:d1:94:c3:11:00:63:83:1e:7c:25:c9:21:
                    29:37
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier:
                38:65:B3:D1:3B:4E:87:A8:D0:41:DA:B9:18:B8:DD:B3:3B:EF:55:7C
            X509v3 Authority Key Identifier:
                keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6

            Authority Information Access:
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/

            X509v3 Subject Alternative Name:
                DNS:test.streamingworld.us
            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            CT Precertificate SCTs:
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : DF:A5:5E:AB:68:82:4F:1F:6C:AD:EE:B8:5F:4E:3E:5A:
                                EA:CD:A2:12:A4:6A:5E:8E:3B:12:C0:20:44:5C:2A:73
                    Timestamp : Dec 26 07:18:49.509 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:45:02:21:00:C3:8E:D7:55:E6:40:F2:10:15:F5:74:
                                31:CE:28:3E:E1:2F:A2:5A:94:11:EE:F3:D6:87:F2:A3:
                                D7:58:69:AC:E9:02:20:67:94:CA:DB:E5:D2:AB:53:A9:
                                C3:AA:E3:A1:78:FA:6A:4B:6A:EB:09:F7:E7:60:F4:81:
                                F6:6C:75:11:9C:50:59
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 29:79:BE:F0:9E:39:39:21:F0:56:73:9F:63:A5:77:E5:
                                BE:57:7D:9C:60:0A:F8:F9:4D:5D:26:5C:25:5D:C7:84
                    Timestamp : Dec 26 07:18:49.621 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:CC:DC:E3:06:7A:0D:C7:9B:FB:EA:48:
                                CB:B9:18:65:84:86:9D:DB:4D:83:11:19:92:2C:06:46:
                                32:4B:3E:36:1D:02:21:00:B9:FD:E8:18:31:A2:66:96:
                                88:B2:15:B8:D8:67:22:A3:4A:77:52:4B:94:9C:70:F5:
                                97:94:88:0F:18:69:72:E7
    Signature Algorithm: sha256WithRSAEncryption
         72:e4:f5:06:e3:0d:2c:3a:21:3f:f9:34:02:15:c5:64:f7:8e:
         5e:d8:54:19:86:2f:37:52:e2:98:41:a8:4d:2e:64:ea:b2:d1:
         76:e7:71:60:11:73:91:32:50:ee:67:20:90:78:84:98:39:07:
         2a:69:f2:f1:39:8d:6a:b6:e0:e6:be:4a:98:94:6b:1c:45:b6:
         e4:41:54:db:42:3f:b2:71:e9:f9:2e:da:7a:f0:85:f9:1d:66:
         25:ae:85:d1:9e:77:f2:cf:c1:98:85:cb:5f:d5:7f:3b:b0:40:
         7e:8f:e3:ad:96:93:aa:af:a1:e7:36:6d:10:42:45:a1:ef:7d:
         d9:d7:e6:c4:7e:f1:13:a9:6d:d2:13:36:87:5c:27:70:dd:68:
         4d:48:16:53:5c:17:91:3a:23:4c:a0:fd:fe:92:36:98:a3:61:
         b3:6a:8b:1a:1c:f1:75:47:7a:71:7b:db:da:20:39:18:f3:72:
         45:f8:69:a2:22:ba:78:5a:4f:1b:d5:55:1f:24:91:9b:7b:aa:
         f5:77:56:a7:6d:07:f5:d3:db:b0:3c:32:a4:42:d7:8c:62:d5:
         b5:ce:dc:a4:08:f9:3c:26:e3:f6:e2:ca:cf:0b:9c:50:96:12:
         46:b8:4a:1d:e5:60:73:38:fb:de:30:d3:df:73:d2:da:9c:e4:
         e9:b0:31:f5

lukastribus · March 9, 2024, 3:27am

You must have an old certificate in /etc/haproxy/certs/ with a different filename then, but with matching SAN.

rayj00 · March 9, 2024, 1:39pm

Do the file names in /etc/haproxy/certs/ have to end in .pem? I renamed all of the files by appending an X (.pemX) to all of the files except the test.streamingworld.us.pem but that did not fix the issue?

lukastribus · March 9, 2024, 2:03pm

https://docs.haproxy.org/2.8/configuration.html#5.1-crt

If a directory name is used instead of a PEM file, then all files found in
that directory will be loaded in alphabetic order unless their name ends
with ‘.key’, ‘.issuer’, ‘.ocsp’ or ‘.sctl’ (reserved extensions). Files
starting with a dot are also ignored.

rayj00 · March 9, 2024, 2:17pm

Success! I removed all of the certs in the certs folder except test.streamingworld.us.pem and now browsing to test.steamingworld.us shows as being secure (https)! Woo hoo!

Now to find the cert that is causing this issue.

Thanks for your help. Appreciate it!

Ray

mrit · March 11, 2024, 9:55am

you refer to /etc/haproxy/certs/ but letsencrypt puts its certs in /etc/letsencrypt/live. so you have to copy the recent (renewed) certs from /etc/letsencrypt/live to /etc/haproxy/cert.

Topic		Replies	Views
Certbot shows valid certs but expired cert being used? Help!	1	486	February 7, 2021
Letsencrypt Renew failing Help!	0	1058	November 6, 2020
Installing Letsencrypt SSL? Help!	1	374	August 29, 2020
SSL renewal with HAProxy Help!	1	1848	February 15, 2023
Unable to renew SSL certificates with servers using SSL passthrough Help!	1	634	October 11, 2021

Renew Cert updates but old cert being used?

Related topics