Haproxy 2.8 uses self signed certificate - where does it take it from?

[chriskuku]

I messed around with my haproxy.cfg since I would like to differ between two servers, a cms.mydomain.org and a www.mydomain.org. Backend is a Tomcat9 server. While in a former configuration my fullchain.pem worked, haproxy suddenly doesn’t care about that certificate and the browser (FF) says I were using a self signe cert. (Mozilla Error: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT)
My haproxy.cfg:

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        pidfile /var/run/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        maxconn 4000
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/apache2/certs
        crt-base /etc/ssl/apache2/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-dh-param-file /etc/haproxy/dhparams.pem
        ssl-default-bind-options no-sslv3
        ssl-skip-self-issued-ca

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
    compression algo gzip
    compression type text/html text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy application/atom+xml application/javascript application/x-javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest
    balance roundrobin
    option dontlog-normal
    option dontlognull
    option httpclose
    option forwardfor

frontend http-in
    bind *:80
    bind *:443 ssl crt /etc/haproxy/fullchain.pem
    bind quic4@0.0.0.0:443 name quic443 ssl crt /etc/haproxy/fullchain.pem proto quic alpn h3,h3-29,h3-28,h3-27 npn h3,h3-29,h3-28,h3-27 allow-0rtt curves secp521r1:secp384r1
    http-response add-header alt-svc 'h3=":443"; ma=7200,h3-29=":443"; ma=7200,h3-Q050=":443"; ma=7200,h3-Q046=":443"; ma=7200,h3-Q043=":443"; ma=7200,quic=":443"; ma=7200'

    # Redirect if HTTPS is *not* used
    redirect scheme https code 301 if !{ ssl_fc }

    acl letsencrypt-acl path_beg /.well-known/acme-challenge/

    use_backend letsencrypt-backend if letsencrypt-acl
    default_backend website

    log /dev/log local2 debug

    acl is_static       path_beg /export/ /opencms/ /resources/ /javadoc/ /VAADIN/ /workplace /opencms-login/
    acl is_website      hdr_beg(host) -i www.mydomain.org
    acl is_cmssite      hdr_beg(host) -i cms.mydomain.org
    use_backend website-static if is_website is_static
    use_backend website if is_website
    use_backend cmssite if is_cmssite

backend letsencrypt-backend
    server letsencrypt 127.0.0.1:8888

backend website-static
    server www.mydomain.org 127.0.0.1:8080

backend website
    http-request replace-header Destination ^([^\ :]*)\ /(.*) \1\ /opencms/\2
    server www.mydomain.org 127.0.0.1:8080
backend cmssite
    http-request replace-header Destination ^([^\ :]*)\ /(.*) \1\ /opencms/\2
    server cms.mydomain.org 127.0.0.1:8082

[lukastribus]

If you configure haproxy to use the certificate in /etc/haproxy/fullchain.pem it will do exactly that.
Nothing more, nothing less.

Based on your description there is no way for us to know what may be wrong in your environment.

[chriskuku]

Yeah, but /etc/haproxy/fullchain.pem is not a self signed certificate, it’s a Letsencrypt one.

It may not contain the domain that is adressed in the HTTP-request but it looks to me as though haproxy falls back to some “self signed certificate”, whatever that may be and whereever it takes it from.

[lukastribus]

Haproxy certainly does not do that.

Perhaps you have a software or hardware Firewall with SSL inspection enabled between your client and the server, and because the name does not match, the Firewall replaces and intercept the certificate.

Or perhaps you have an old haproxy instance running in the background, that actually did run with a self signed certificate but was not properly closed?

What does the self signed certificate say exactly? Just because it’s a self signed certificate, does not mean it contains absolutely zero informations.

Look at the informations you have at your disposal.

[chriskuku]

Your guess about multiple haproxy can be right:

 ps ax | grep haproxy
 127487 pts/1    S      0:00 /usr/local/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
 127489 pts/1    Sl     0:09 /usr/local/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
 137338 ?        Ss     0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
 137340 ?        Sl     0:01 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
 138837 pts/0    S+     0:00 grep --color=auto haproxy

It looks like this comes from a test run I did before.

Thanks. I see clearer now.

[lukastribus]

Also see post #7 here:

Haproxy returns 403 NOSRV error intermittently

For a clean situation, stop all haproxy processes and kill all the rest, and then start it. Check the process table to see a “normal” situation. Reload and restart haproxy a few teams then check the process table again to see a “normal” situation after reload/restarts. systemctl stop haproxy killall haproxy ps auxwf The reason multiple processes binding to the same port are possible is due to the use of SO_REUSEPORT on the socket. You can disable it with the noreuseport directive, however it’s…

[chriskuku]

I believe, killing is not as easy, since they refork when you say killall haproxy. But I got it managed. Rebooted.

Interesting point, the SOREUSPORT thing. My first idea was that I was wondering about two haproxy instances being able to run on a claimed socket/port.