Haproxy, letsencrypt and docker compose problem

Help!
1

Hi,

I am very new to all of this, so please bear with me.

I have a docker compose setup with 2 backend services and a haproxy.
The proxy uses the (experimental) ACME integration with letsencrypt.
The DNS is handled by a dynamic DNS setup (noip).
It all seems to work well, a certificate is requested and obtained. See logs here:

haproxy-1 | [NOTICE] (1) : Initializing new worker (8)
haproxy-1 | [NOTICE] (8) : config : No certificate available for 'mpin.ddns.net.pem', generating a temporary key pair before getting the ACME certificate
haproxy-1 | [NOTICE] (8) : Automatically setting global.maxconn to 262126.
haproxy-1 | [NOTICE] (1) : Loading success.
haproxy-1 | acme: mpin.ddns.net.pem: Starting update of the certificate.
haproxy-1 | -:- [26/Jun/2026:15:02:24.335] -/- 4/0/379/181/562 200 1325 - - ---- 0/0/0/0/0 0/0 {172.65.46.172} "GET ``https://acme-staging-v02.api.letsencrypt.org/directory`` HTTP/1.1" 0/0000000000000000/-/-/0 ``acme-staging-v02.api.letsencrypt.org/TLSv1.3/TLS_AES_256_GCM_SHA384
haproxy-1 | -:- [26/Jun/2026:15:02:24.898] -/- 2/0/0/181/183 200 309 - - ---- 0/0/0/0/0 0/0 {172.65.46.172} "HEAD ``https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce`` HTTP/1.1" 0/0000000000000000/-/-/0 ``acme-staging-v02.api.letsencrypt.org/TLSv1.3/TLS_AES_256_GCM_SHA384
haproxy-1 | -:- [26/Jun/2026:15:02:25.080] -/- 3/0/0/185/187 200 720 - - ---- 0/0/0/0/0 0/0 {172.65.46.172} "POST ``https://acme-staging-v02.api.letsencrypt.org/acme/new-acct`` HTTP/1.1" 0/0000000000000000/-/-/0 ``acme-staging-v02.api.letsencrypt.org/TLSv1.3/TLS_AES_256_GCM_SHA384
haproxy-1 | -:- [26/Jun/2026:15:02:25.266] -/- 3/0/0/200/202 201 821 - - ---- 0/0/0/0/0 0/0 {172.65.46.172} "POST ``https://acme-staging-v02.api.letsencrypt.org/acme/new-order`` HTTP/1.1" 0/0000000000000000/-/-/0 ``acme-staging-v02.api.letsencrypt.org/TLSv1.3/TLS_AES_256_GCM_SHA384
haproxy-1 | -:- [26/Jun/2026:15:02:25.467] -/- 3/0/0/183/185 200 1453 - - ---- 0/0/0/0/0 0/0 {172.65.46.172} "POST ``https://acme-staging-v02.api.letsencrypt.org/acme/authz/306667734/2347829964`` HTTP/1.1" 0/0000000000000000/-/-/0 ``acme-staging-v02.api.letsencrypt.org/TLSv1.3/TLS_AES_256_GCM_SHA384
haproxy-1 | -:- [26/Jun/2026:15:02:25.652] -/- 3/0/0/190/191 200 838 - - ---- 0/0/0/0/0 0/0 {172.65.46.172} "POST ``https://acme-staging-v02.api.letsencrypt.org/acme/finalize/306667734/42414363574`` HTTP/1.1" 0/0000000000000000/-/-/0 ``acme-staging-v02.api.letsencrypt.org/TLSv1.3/TLS_AES_256_GCM_SHA384
haproxy-1 | -:- [26/Jun/2026:15:02:28.844] -/- 2/0/0/186/187 200 907 - - ---- 0/0/0/0/0 0/0 {172.65.46.172} "POST ``https://acme-staging-v02.api.letsencrypt.org/acme/order/306667734/42414363574`` HTTP/1.1" 0/0000000000000000/-/-/0 ``acme-staging-v02.api.letsencrypt.org/TLSv1.3/TLS_AES_256_GCM_SHA384
haproxy-1 | acme: mpin.ddns.net.pem: Successful update of the certificate.
haproxy-1 | -:- [26/Jun/2026:15:02:29.031] -/- 3/0/0/184/188 200 5742 - - ---- 0/0/0/0/0 0/0 {172.65.46.172} "POST ``https://acme-staging-v02.api.letsencrypt.org/acme/cert/2c19524b3e98ce2a0dbb5e20e18259e66d31`` HTTP/1.1" 0/0000000000000000/-/-/0 ``acme-staging-v02.api.letsencrypt.org/TLSv1.3/TLS_AES_256_GCM_SHA384

But the SSL handshake is not working correctly. The browser displays a message about an untrusted cert, and haproxy displays this error in the logs:

in/2: SSL handshake failure (error:0A000418:SSL routines::tlsv1 alert unknown ca)

I have no idea at all how to fix it. I thought the certificates obtained through the ACME client should be trusted?

Here is my configuration:


global
  log stdout format raw local0

expose-experimental-directives

defaults
  log     global
  mode    http
  timeout connect 5s
  timeout client  50s
  timeout server  50s

acme LE1
  directory https://acme-staging-v02.api.letsencrypt.org/directory
  contact simon@webtecc.com
  map virt@acme

crt-store
  load crt "mpin.ddns.net.pem" acme LE1 domains "mpin.ddns.net"

frontend in
  bind *:80
  bind *:443 ssl ciphersuites TLS_AES_256_GCM_SHA384:@SECLEVEL=0

  ssl-f-use crt "mpin.ddns.net.pem"

  option forwardfor

  http-request redirect scheme https unless { ssl_fc }
  http-request return status 200 content-type text/plain lf-string   "%[path,field(-1,/)].%[path,field(-1,/),map(virt@acme)]\n" if { path_beg '/.well-known/acme-challenge/' }
  http-request allow if { path_beg /app/ }
  http-request allow if { path_beg /auth/ }

  use_backend app if { path_beg /app/ }
  use_backend keycloak if { path_beg /auth/ }
  default_backend app

backend app
  server s1 mpin-app-1:8080 check

backend keycloak
  server s2 mpin-keycloak-1:8080
2

You are using the LE staging endpoint, which will result in test certificates (which are invalid). You need to switch to the production endpoint.

3

Thank you @lukastribus, that got me one step further. However, I am still getting an SSL error and browser warning. The HA Proxy log shows this error:

in/2: SSL handshake failure (error:0A000412:SSL routines::ssl/tls alert bad certificate)

However, I also see some problem with cert issuance:

acme: mpin.ddns.net.pem: invalid HTTP status code 429 when getting newOrder URL: "too many certificates (5) already issued for this exact set of identifiers in the last 168h0m0s, retry after 2026-06-29 19:17:50 UTC: see Rate Limits - Let's Encrypt " (urn:ietf:params:acme:error:rateLimited), retrying in 119966s (1/5 retries)...

Could it be this will fix itself eventually?

4

You are now hitting Lets Encrypt rate limits. It looks like you may be using a generic DynDns TLD.

In that case your likely subject to TLD level rate limits.

Your best choice is to use your own TLD.

Otherwise you may use a alternative CA, which reduces the likelyhood of you hitting a per TLD rate limit:

5

Hey @lukastribus, thanks for the info once more. You are correct that this is a DynDNS setup, but I’m going to try it as is because my assumption is, that the IP address is relatively static.

NoIP apparently offers a plan whereby one can bind a TLD to a dynamic IP, but all of that would cost more than this is worth.

Thank you kindly for your help!

6

Any TLD on any registrar will do, anyone of them allows you to set a CNAME, which you can point to your existing DDNS service.

The SANs in the certificate is what is relevant, not the fact whether we are talking about A, AAAA or CNAME records.

7

Okay, this is starting to get in the way of things in a serious way.

I added a CNAME to a TLD under a subdomain: gis.mais-parque-industrial-nao.pt, which points to my DDNS mpin.ddns.net. I am still working on the configuration, so I have to restart the whole setup frequently.

After a few restarts, letsencrypt started whining about too many requests again. The result is that I am effectively locked out again until tomorrow. Every single fricken time I am loosing a complete day with this. It is becoming a highly infuriating affair.

We are on a clock fighting a political issue that can not wait. I can not afford being sabotaged by this stuff all the time. This stuff is becoming a serious stumbling stone for our entire group.

What else can I do to finally remove this problem?

8

The built-in ACME client is a experimental feature with some important limitations.

I will refer to the documentation here:

The current HAProxy architecture is a non-blocking model, access to the disk
is not supposed to be done after the configuration is loaded, because it
could block the event loop, blocking the traffic on the same thread. Meaning
that the certificates and keys generated from HAProxy will need to be dumped
from outside HAProxy using “dump ssl cert” on the stats socket. It’s possible
to automate the dump of the certificates by using the dataplaneAPI or the
haproxy-dump-certs script provided in the admin/cli/ directory.

A tutorial is available here:

However you should avoid experimental features entirely if you are just looking for something that works without tinkering.

There are tons of stable ACME clients out there, thats probably a safer bet for your production workload.

9

If the IP is fairly stable, that’s a reasonable thing to try. Just keep in mind that the Let’s Encrypt rate limit won’t be fixed by the IP being stable — it’s about the hostname/certificate identifiers, so you’ll need to wait until the indicated retry time before issuance can succeed again for that name.

For testing changes, stick to the staging endpoint until you’re confident the flow works, then switch to production only once. Also make sure HAProxy is persisting the ACME/certificate storage in a Docker volume, otherwise container recreation can cause repeated issuance attempts and you’ll hit limits much faster.

10

Yes, this is the key part: don’t let the container “forget” its ACME state between restarts. In Docker Compose, the ACME account data/certs need to live on a persistent volume or bind mount, not only inside the container filesystem. Otherwise every rebuild/recreate can look like a fresh client asking Let’s Encrypt again.

Also, while debugging, use Let’s Encrypt staging or even temporarily disable automatic issuance until the routing and challenge path are confirmed. Once HTTP-01 validation works reliably and the storage survives restarts, switch back to production and request the real cert once.