Binding on both 443 & 8443 used to work, until we implemented Let's Encrypt. Now 443 works but 8443 seems broken

Help!
1

I’m hoping someone can help me figure this out.

I used to have HAProxy set up such that requests on port 80 would redirect to SSL over port 443, and also explicit requests to SSL over port 8443 would also work. Then my CIO decreed that we were moving to using Let’s Encrypt for our SSL certs. I got that working on my HAProxy server, but now there’s no response when I try to go directly to port 8443.

Here’s my complete HAProxy.cfg file:

global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private

Default ciphers to use on SSL-enabled listening sockets.
For more information, see ciphers(1SSL). This list is from:
https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers TLS+ECDHE+RSA+WITH+AES+256+GCM+SHA384:TLS+ECDHE+ECDSA+WITH+AES+256+GCM+SHA384:TLS+ECDH+RSA+WITH+AES+256+GCM+SHA384:TLS+ECDH+ECDSA+WITH+AES+256+GCM+SHA384:TLS+ECDHE+RSA+WITH+AES+128+GCM+SHA256:TLS+ECDHE+ECDSA+WITH+AES+128+GCM+SHA256:TLS+ECDH+RSA+WITH+AES+128+GCM+SHA256:TLS+ECDH+ECDSA+WITH+AES+128+GCM+SHA256:TLS+ECDHE+RSA+WITH+AES+256+CBC+SHA384:TLS+ECDHE+ECDSA+WITH+AES+256+CBC+SHA384:TLS+ECDHE+RSA+WITH+AES+256+CBC+SHA:TLS+ECDHE+ECDSA+WITH+AES+256+CBC+SHA:TLS+ECDH+RSA+WITH+AES+256+CBC+SHA384:TLS+ECDH+ECDSA+WITH+AES+256+CBC+SHA384:TLS+ECDH+RSA+WITH+AES+256+CBC+SHA:TLS+ECDH+ECDSA+WITH+AES+256+CBC+SHA:TLS+ECDHE+RSA+WITH+AES+128+CBC+SHA256:TLS+ECDHE+ECDSA+WITH+AES+128+CBC+SHA256:TLS+ECDHE+RSA+WITH+AES+128+CBC+SHA:TLS+ECDHE+ECDSA+WITH+AES+128+CBC+SHA:TLS+ECDH+RSA+WITH+AES+128+CBC+SHA256:TLS+ECDH+ECDSA+WITH+AES+128+CBC+SHA256:TLS+ECDH+RSA+WITH+AES+128+CBC+SHA:TLS+ECDH+ECDSA+WITH+AES+128+CBC+SHA:HIGH:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
ssl-default-bind-options no-sslv3 no-tlsv10

Maximum allowed simultaneous connections
maxconn 2048

Crypto key size
tune.ssl.default-dh-param 4096

defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

Frontend definitions
frontend www-http
bind *:80
reqadd X-Forwarded-Proto:\ http
default_backend tomcats

frontend www-https-443
    bind *:443 ssl crt /etc/haproxy/certs/server.mycompany.com.pem
    acl secure dst_port eq 443
    http-request replace-header Host ^(.*?)(:[0-9]+)?$ \1:443
    reqadd X-Forwarded-Proto:\ https
    rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains;\ preload
    rsprep ^Set-Cookie:\ (.*) Set-Cookie:\ \1;\ Secure if secure
    acl letsencrypt-acl path_beg /.well-known/acme-challenge/
    use_backend letsencrypt-backend if letsencrypt-acl
    default_backend tomcats

frontend www-https-8443
    bind *:8443 ssl crt /etc/haproxy/certs/server.mycompany.com.pem
    acl secure dst_port eq 8443
    http-request replace-header Host ^(.*?)(:[0-9]+)?$ \1:8443
    reqadd X-Forwarded-Proto:\ https
    rspadd Strict-Transport-Security:\ max-age=31536000;\ includeSubDomains;\ preload
    rsprep ^Set-Cookie:\ (.*) Set-Cookie:\ \1;\ Secure if secure
    acl letsencrypt-acl path_beg /.well-known/acme-challenge/
    use_backend letsencrypt-backend if letsencrypt-acl
    default_backend tomcats

Backend definitions
backend tomcats
http-request redirect scheme https if !{ ssl_fc }
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
balance roundrobin
cookie JSESSIONID prefix nocache
server webapp1 xxx.xxx.xxx.xxx:8080 check cookie webapp1
server webapp2 yyy.yyy.yyy.yyy:8080 check cookie webapp2

backend letsencrypt-backend
    server letsencrypt 127.0.0.1:54321

Interestingly, I see no trace of any attempts at communication on port 8443 in my haproxy.log file. I’ll attach that in a reply message if needed.

2

Double check that only 1 haproxy instance is running. Due to reuseport it can happen that multiple instances are running, with old configurations.

3

Thanks for this Lukas. I did some digging and can confirm that only one instance of HAProxy is currently running.

4

Unsure of your OS, but I’ve gotten caught a couple times with Firewall issues on the server itself (I’m running Centos). Turn off any firewalls you might have? I don’t know why the change to LE would cause this, but I’m trucking along with LE on non-standard ports just fine in HAProxy. Food for thought that LE works in this configuration.

5

Thanks, Aaron. Actually, you’re exactly right. I actually had ended up enabling the firewall and port 8443 and had failed to open port 8443. I feel a little embarrassed for shooting up a flare when the solution had nothing to do with HAProxy, but I’m also relieved that the issue is resolved and my application is working properly again.