HAProxy using old Cert File

Help!
1

Hello,

today my website showed that the SSL certificate is outdated.
I’ve searched for hours now and the Cert on the system is renewed, but when i browse the site i get an ssl error…“Das Zertifikat ist am Freitag, 4. Mai 2018, 12:13 abgelaufen. Die aktuelle Zeit ist Freitag, 4. Mai 2018, 19:58.”

I have multiple websites running over https -> http and only the first one won’t use the new certificate.
Yes the certificate is loaded correctly in the certs dir and it shows the right timestamps.

Strange behaviour: If only activate this site with ssl it works but f5 spamming will lead to an error, guess which one… “Das Zertifikat ist am Freitag, 4. Mai 2018, 12:13 abgelaufen. Die aktuelle Zeit ist Freitag, 4. Mai 2018, 19:58.”

It’s like HAProxy is loading the old certificate from heaven or hell…

(Reload, Stop, Restart) done, no solution insight

Please help!

2

But did you check whether there are any obsolete processes still running? A init.d/systemd “stop” doesn’t find those.

3

Yes, with ps aux | grep haproxy and PS -ef | grep haproxy

I also did a Server reboot

Mit freundlichen Grüßen

René Hofmann

(Gesendet von meinem Smartphone)

4

Double and triple check what you did until now, likely an assumption you made is not actually correct.

There are no magic tricks involved with certificate management in haproxy, if after a reboot the old certificate still shows up, it’s because the filesystem delivers it to haproxy.

5

I did more than Triple Check, If i Put The cert /with/filename.pem The correcht one gets loaded. But spamming a refresh in any Browser will make The old one Sometimes appears???

Mit freundlichen Grüßen

René Hofmann

(Gesendet von meinem Smartphone)

6

Ubuntu 16.04.4 (Kernel 4.4.0-122-generic APR 23 2018)
HA-Proxy version 1.6.3 2015/12/25

cat /etc/haproxy/haproxy.cfg

global
maxconn 4096
tune.ssl.default-dh-param 2048
user haproxy
group haproxy
chroot /var/lib/haproxy
daemon
log 127.0.0.1 local0

    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 10s

    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DS  
    ssl-default-bind-options no-sslv3

defaults
maxconn 4096
log global
mode http
option httplog
option dontlognull
retries 3
option redispatch
option http-server-close
option forwardfor
timeout connect 5000
timeout client 50000
timeout server 50000

    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend www-http
bind *:80
mode http
option forwardfor
option http-server-close
option http-pretend-keepalive
reqadd X-Forwarded-Proto:\ http
redirect scheme https code 301 if !{ ssl_fc }

    http-response set-header Strict-Transport-Security max-age=31536000;
    #default_backend nc_cluster

frontend www-https
bind 0.0.0.0:443 ssl crt /etc/haproxy/certs/

    reqadd X-Forwarded-Proto:\ https

    acl letsencrypt-acl path_beg /.well-known/acme-challenge/

    use_backend letsencrypt-backend if letsencrypt-acl

    use_backend nc_cluster if { ssl_fc_sni nc.problemdomain.at }
    use_backend oo_cluster if { ssl_fc_sni oo.problemdomain.at }

    use_backend seconddomain_cluster if { ssl_fc_sni www.seconddomain.co.at }
    use_backend seconddomain_cluster if { ssl_fc_sni seconddomain.co.at }

    use_backend thirddomain_cluster if { ssl_fc_sni www.thirddomain.co.at }
    use_backend thirddomain_cluster if { ssl_fc_sni thirddomain.co.at }

backend nc_cluster
server node1 10.101.114.111:80 check

backend oo_cluster
server node1 10.101.114.91:80 check

backend seconddomain_cluster
server node1 10.10.10.1:80 check

backend thirddomain_cluster
server node1 10.10.10.2:80 check

backend letsencrypt-backend
server letsencrypt 127.0.0.1:54321

/etc/haproxy# ls -l
total 16
drw-r–r-- 2 root root 4096 Apr 16 13:26 certs
drwxr-xr-x 2 root root 4096 Dez 1 14:03 errors
-rw-r–r-- 1 root root 2632 Mai 4 19:53 haproxy.cfg
-rwxr-xr-x 1 root root 33 Dez 4 11:33 haproxy_stats.sh

/etc/haproxy/certs# ls -l
total 48
-rw-r–r-- 1 root root 5490 Mai 4 19:52 nc.problemdomain.at.pem
-rw-r–r-- 1 root root 5490 Mai 4 19:52 oo.problemdomain.at.pem
-rw-r–r-- 1 root root 5514 Mai 4 19:52 seconddomain.co.at.pem
-rw-r–r-- 1 root root 5518 Mai 4 19:52 thirddomain.co.at.pem
-rw-r–r-- 1 root root 5526 Mai 4 19:52 www.seconddomain.co.at.pem
-rw-r–r-- 1 root root 5530 Mai 4 19:52 www.thirddomain.co.at.pem

/etc/haproxy/certs# openssl x509 -in nc.problemdomain.at.pem -text -noout Certificate:
Data:
Version: 3 (0x2)
Serial Number:
03:4b:b5:fc:01:26:f0:1c:99:9f:1f:f7:6d:99:a3:cc:c7:e5
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=US, O=Let’s Encrypt, CN=Let’s Encrypt Authority X3
Validity
Not Before: May 4 10:35:15 2018 GMT
Not After : Aug 2 10:35:15 2018 GMT
Subject: CN=nc.problemdomain.at
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:9a:3b:0d:f2:88:4b:d7:4e:95:a4:b8:eb:15:2b:
03:8b:62:12:b5:69:ee:7b:69:a7:f1:9a:8f:93:36:
61:2a:5c:37:43:1a:65:48:bb:09:8d:ef:60:68:23:
07:1c:49:14:10:04:72:d2:82:1a:dd:21:7b:ed:65:
87:44:03:3d:21:84:91:c0:19:6f:cd:23:ab:a5:95:
12:45:26:6b:6a:e8:a4:3d:9c:5f:1a:5c:c9:44:81:
a3:ce:76:33:fc:f6:d3:12:11:17:c1:7c:44:7c:60:
b9:ed:49:6e:c1:0c:f5:c9:e5:70:d8:88:26:f1:5d:
2f:be:eb:c6:f0:bc:aa:a6:1a:6c:90:5a:26:c8:c8:
b3:ca:4e:b7:15:bf:53:0c:d6:93:82:1b:25:2b:27:
cb:a9:83:73:af:30:a6:a7:95:25:26:8c:85:e5:ab:
b1:58:0d:ed:72:b4:89:c9:e8:34:f6:90:8b:b9:ca:
53:85:21:66:ff:fb:c0:8c:f0:da:57:24:9b:66:18:
b7:a1:87:9b:be:18:73:dd:e4:26:34:e6:77:bc:c9:
d2:9f:4a:0d:e4:c4:90:fd:a3:a7:cf:07:1c:f0:1c:
39:98:2d:2b:de:7d:d0:d1:74:3f:f2:03:81:f9:66:
96:d1:58:6d:b4:db:a9:32:c5:92:6a:2a:22:c9:48:
4f:f7
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
63:8F:3B:20:18:D8:07:2A:CE:AA:D1:DD:0F:BC:3D:2C:C1:83:65:FF
X509v3 Authority Key Identifier:
keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

        Authority Information Access:
            OCSP - URI:http://ocsp.int-x3.letsencrypt.org
            CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

        X509v3 Subject Alternative Name:
            DNS:nc.problemdomain.at
        X509v3 Certificate Policies:
            Policy: 2.23.140.1.2.1
            Policy: 1.3.6.1.4.1.44947.1.1.1
              CPS: http://cps.letsencrypt.org
              User Notice:
                Explicit Text: This Certificate may only be relied upon by Relying Parties and only in accordance with the Certificate Policy found at https://letsencrypt.org/repository/

        CT Precertificate SCTs:
            Signed Certificate Timestamp:
                Version   : v1(0)
                Log ID    : DB:74:AF:EE:CB:29:EC:B1:FE:CA:3E:71:6D:2C:E5:B9:
                            AA:BB:36:F7:84:71:83:C7:5D:9D:4F:37:B6:1F:BF:64
                Timestamp : May  4 11:35:15.142 2018 GMT
                Extensions: none
                Signature : ecdsa-with-SHA256
                            30:45:02:21:00:81:E2:4F:09:71:2E:52:44:78:3C:BD:
                            70:84:F2:E4:3C:0F:65:91:DF:84:7A:FF:3D:56:AE:B8:
                            DE:75:70:95:7C:02:20:2F:EE:DD:86:9D:B1:86:73:5E:
                            43:59:B3:55:A0:E8:25:9D:8D:F3:6B:0B:18:ED:5C:92:
                            E5:65:3D:5B:31:1E:4B
            Signed Certificate Timestamp:
                Version   : v1(0)
                Log ID    : 29:3C:51:96:54:C8:39:65:BA:AA:50:FC:58:07:D4:B7:
                            6F:BF:58:7A:29:72:DC:A4:C3:0C:F4:E5:45:47:F4:78
                Timestamp : May  4 11:35:15.162 2018 GMT
                Extensions: none
                Signature : ecdsa-with-SHA256
                            30:45:02:21:00:D4:8A:D9:BF:02:82:5F:3B:93:77:E8:
                            8E:6C:96:40:38:E6:D2:F2:F0:E8:54:02:37:5B:AC:46:
                            03:B4:89:9F:EF:02:20:4B:1C:55:57:60:1C:9E:89:79:
                            DB:94:30:92:3B:37:11:9E:F2:1B:ED:89:6C:C7:BE:B9:
                            34:0C:47:75:5D:24:92
Signature Algorithm: sha256WithRSAEncryption
     9a:2d:db:03:5d:08:90:c4:4a:b1:d5:0f:bd:8a:b9:ae:35:9f:
     36:f7:d5:49:f4:3e:c7:34:6b:58:53:69:38:d9:55:9e:1a:f0:
     d1:c4:ed:d9:dd:91:72:73:36:2a:72:a2:7c:d2:f0:76:50:72:
     f1:e2:ec:94:74:e2:5e:ed:29:8a:f7:60:cc:84:7b:d1:12:09:
     8f:d9:ec:3d:47:f0:79:32:43:20:9b:cd:ee:51:d0:3a:ca:fb:
     f5:ae:ac:a4:9b:c7:51:e3:40:d6:ec:68:4e:f7:4b:61:af:ee:
     83:0e:fa:27:56:3d:fc:4f:34:94:2e:9c:5b:0b:87:ec:38:15:
     0f:dc:3a:1a:15:bf:2f:85:00:18:e2:3f:d5:c9:3a:7d:45:b6:
     95:54:35:e6:fd:a7:58:31:75:f7:ba:6d:3d:b7:37:8a:70:9d:
     dc:1a:b3:c1:54:85:19:b9:7e:89:e4:e1:b8:c9:38:42:c1:00:
     e5:9f:65:05:2a:f5:03:a9:3a:c0:45:20:a0:85:b5:70:ac:6a:
     36:41:91:92:55:86:6a:cd:89:66:a5:37:29:70:f6:0d:cb:4b:
     1e:ff:32:d9:2f:e7:70:04:14:a2:7a:97:74:20:ac:32:fa:d6:
     e7:4d:c3:a7:3a:c1:5f:02:5b:8f:a6:d7:9f:6e:d5:d4:af:b9:
     ff:23:32:7d

/var/log/haproxy.log == empty

sudo haproxy -f /etc/haproxy/haproxy.cfg -c
Configuration file is valid

this is my actual config orig domains are veiled

7

When accessing this domain from the Internet, the old expired one is always loaded. I don’t see the new certificate ever. I assume you don’t want your real domain name here, so I’m not gonna post it, but if you run it through the SSLtest you will see that the expired one always shows up.

I suggest you stop and kill all haproxy instances and check if there is still something responding on port 443, from your place and from the Internet. If it is, then find it and kill it.

Also check if you are really accessing haproxy from your browser, not something else maybe a development box you redirect to with a hosts entry.

Any security “solutions” like software firewalls, hardware firewalls, anything that may intercept your SSL sessions is something you’d have to look out for as well.

Also, you have to go through all certificates in the directory and make sure the old one isn’t there anymore.

8

i really appreciate your help!
SSLtest shows the old certificate.

When i kill haproxy, it’s not possible to connect to the website -> connection refused

lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
sshd 952 root 3u IPv4 13843 0t0 TCP *:ssh (LISTEN)
sshd 952 root 4u IPv6 13845 0t0 TCP *:ssh (LISTEN)
snmpd 1054 snmp 9u IPv4 14228 0t0 UDP *:snmp
snmpd 1054 snmp 10u IPv4 14226 0t0 UDP *:49863

I’m accessing the website from home so im accessing over external IP not VIP.
The external IP is pointing to the HAProxy external IP, i’ve checked it again.

Yes we have security solutions. But i don’t think thats the problem.
I checked all certificates again, and there is no old one in the crt directory.

Server Key and Certificate #1
Subject xxx.domain.at
Fingerprint SHA256: f696f480de79230ffa3b8d58f273cf7e6d2cee81aa8c8d7985d1b798fddf3d66
Pin SHA256: uWhKBgmXmcZKSwwr0wY8XMKFcpKdYBPA9gLp2XpyO7o=
Common names xxx.domain.at
Alternative names xxx.domain.at
Serial Number 037bb398cc44651f952b4367b45463786c64
Valid from Sat, 03 Feb 2018 10:13:04 UTC
Valid until Fri, 04 May 2018 10:13:04 UTC (expired 1 day, 3 hours ago) EXPIRED
Key RSA 2048 bits (e 65537)
Weak key (Debian) No
Issuer Let’s Encrypt Authority X3
AIA: http://cert.int-x3.letsencrypt.org/
Signature algorithm SHA256withRSA
Extended Validation No
Certificate Transparency No
OCSP Must Staple No
Revocation information OCSP
OCSP: http://ocsp.int-x3.letsencrypt.org
Revocation status Unchecked (only trusted certificates can be checked)
DNS CAA No (more info)
Trusted No NOT TRUSTED (Why?)
Mozilla Apple Android Java Windows

Maybe we can get in contact via Email or something you want, then i would show you the real domain names and config. If you want

BR

9

How much time would you like to spend looking at haproxy before considering that this may be external to haproxy?

What certificate is returned to curl when you do this on the haproxy system:
curl -kv --resolve nc.problemdomain.at:443:127.0.0.1 https://nc.problemdomain.at/

and what about this on the haproxy system:
curl -kv https://nc.problemdomain.at/

And what certificate is returned when you curl from an external box from the Internet (we already know: it’s the expired one):
curl -kv https://nc.problemdomain.at/

If there is a difference, what is in between? Firewall? Router? Another load-balancer?

10

How much time would you like to spend looking at haproxy before considering that this may be external to haproxy?

What certificate is returned to curl when you do this on the haproxy system:
curl -kv --resolve nc.problemdomain.at:443:127.0.0.1 https://nc.problemdomain.at/

  • start date: Feb 3 10:13:04 2018 GMT
  • expire date: May 4 10:13:04 2018 GMT

and what about this on the haproxy system:
curl -kv https://nc.problemdomain.at/
Timeout (Internal to External HAProxy IP Blocked)

And what certificate is returned when you curl from an external box from the Internet (we already know: it’s the expired one):
curl -kv https://nc.problemdomain.at/

  • start date: Feb 3 10:13:04 2018 GMT
  • expire date: May 4 10:13:04 2018 GMT

If there is a difference, what is in between? Firewall? Router? Another load-balancer?
External Access:

{POST} > Firewall > Haproxy > DMZ Internal VIP
{GET} < Firewall < Haproxy < DMZ Internal VIP

11

So haproxy is still serving the old certificate. Clearly what you think you did does not match what you actually did.

There is no way to help you here remotely - you have to go through everything again until you find it.

12

I did everthing, It’s only possible that there is a configuration error or a bug in this haproxy version. You saw my config file except the domain names there are no changes made…

What else can i do then reboot, restart, reload, try crt-list ? I really don’t know what i can do anymore?

As i said we can get in contact via Email, this configuration is not ment to be for public eyes, because they are not my private Webservices / Sites.

If we’ll find the error, i will post the solution here.

UPDATE:

Deleted nc.problemdomain.at.pem the old certificate is still loading
Deleted oo.problemdomain.at.pem -> is trying to use the old nc.problemdomain.at certificate???

All other certs are normally updated… ?!

there is only entry for oo @ dns, so this is ok.

ERROR found… hidden file named (.pem) was in the certs directory… damn never will use ls -l without a

THANK YOU for your time & help!!