Cloudflare Generated RSA for HAProxy in Digital Ocean: SSL handshake failure

Help!
1

Order of cloudflare.pem

-----BEGIN PRIVATE KEY-----
…
-----END PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
…
-----END CERTIFICATE-----

GNU nano 8.1 /etc/haproxy/haproxy.cfg M
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
    ssl-default-bind-curves X25519:prime256v1:secp384r1
    ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-bind-options prefer-client-ciphers ssl-min-ver TLSv1.3 no-tls-tickets

    ssl-default-server-curves X25519:prime256v1:secp384r1
    ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
    ssl-default-server-options ssl-min-ver TLSv1.3 no-tls-tickets

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

frontend http_front
bind *:80
bind *:443 ssl crt /etc/haproxy/certs/cloudflare.pem alpn h2,http/1.1
http-request redirect scheme https unless { ssl_fc }
default_backend http_back

backend http_back
balance roundrobin
server web1 ip:8080 check

listen stats
bind *:1936
mode http
maxconn 10
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth admin:admin

root@meetxdroplet:~# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: enabled)
Active: active (running) since Wed 2025-03-26 22:26:52 UTC; 22min ago
Invocation: 71532a1a655943e59bed3fd277b241b4
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Main PID: 9002 (haproxy)
Status: “Ready.”
Tasks: 2 (limit: 2317)
Memory: 44.2M (peak: 44.4M)
CPU: 383ms
CGroup: /system.slice/haproxy.service
├─9002 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
└─9005 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

Mar 26 22:26:52 meetxdroplet systemd[1]: Starting haproxy.service - HAProxy Load Balancer…
Mar 26 22:26:52 meetxdroplet haproxy[9002]: [NOTICE] (9002) : New worker (9005) forked
Mar 26 22:26:52 meetxdroplet systemd[1]: Started haproxy.service - HAProxy Load Balancer.
Mar 26 22:26:52 meetxdroplet haproxy[9002]: [NOTICE] (9002) : Loading success.
Mar 26 22:27:32 meetxdroplet haproxy[9005]: 45.9.230.8:64936 [26/Mar/2025:22:27:32.107] http_front/2: SSL handshake failure
Mar 26 22:41:21 meetxdroplet haproxy[9005]: 45.9.230.8:65347 [26/Mar/2025:22:41:21.191] http_front/2: SSL handshake failure

I updated my HAProxy configuration, but requests are not reaching my backend servers.

Context: I’m running HAProxy 2.9 and trying to make https work

2

First comes the certificate, the private key comes after.

3

I changed the order, but I still get the error: “Error: unable to verify the first certificate.” Debugging with sudo haproxy -c -f /etc/haproxy/haproxy.cfg does not return any errors. Earlier, I encountered an issue where my certificate wasn’t readable, but that’s not the case, I believe.

As for HAProxy itself, in my configuration setup, removing the line http-request redirect scheme https unless { ssl_fc } makes HTTP work just fine. This proves that HAProxy is functioning correctly, except that it’s not recognizing HTTPS in my request.

4

You did not report this error previously and your are saying the you don’t get any errors when starting haproxy manually.

Where exactly do you see this error?

5

This one is specifically from postman by sending a request to see ssl works as expected

6

From postman directly to haproxy?

That’s expected, if you are using private cloudflare certificate.

If you want to use haproxy directly in your applications including postman, you need to use a public certificate, for example Lets Encrypt.

You can only use the private Cloudflare certificate if you use Cloudflare only as a frontend to haproxy.

7

and this from curl

curl -v https://api.example.com/liveness

  • Host api.example.com:443 was resolved.
  • IPv6: (none)
  • IPv4: 134.199.236.54
  • Trying 134.199.236.54:443…
  • Connected to api.example.com (134.199.236.54) port 443
  • ALPN: curl offers h2,http/1.1
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
  • ALPN: server accepted h2
  • Server certificate:
  • subject: O=CloudFlare, Inc.; OU=CloudFlare Origin CA; CN=CloudFlare Origin Certificate
  • start date: Mar 27 00:54:00 2025 GMT
  • expire date: Mar 23 00:54:00 2040 GMT
  • subjectAltName: host “api.example.com” matched cert’s “*.example.com”
  • issuer: C=US; O=CloudFlare, Inc.; OU=CloudFlare Origin SSL Certificate Authority; L=San Francisco; ST=California
  • SSL certificate verify ok.
  • Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
  • Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
  • using HTTP/2
  • [HTTP/2] [1] OPENED stream for https://api.example.com/liveness
  • [HTTP/2] [1] [:method: GET]
  • [HTTP/2] [1] [:scheme: https]
  • [HTTP/2] [1] [:authority: api.example.com]
  • [HTTP/2] [1] [:path: /liveness]
  • [HTTP/2] [1] [user-agent: curl/8.5.0]
  • [HTTP/2] [1] [accept: /]

GET /liveness HTTP/2
Host: api.example.com
User-Agent: curl/8.5.0
Accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
    < HTTP/2 503
    < cache-control: no-cache
    < content-type: text/html
    <

503 Service Unavailable

No server is available to handle this request.
8

My frontend is hosted on Cloudflare, and my backend is hosted on a private VPS with an assigned IPv4 address. I have a domain set up as api.example.com.

Going back to the main issue, Cloudflare (with built-in SSL) cannot communicate with my HAProxy (which uses HTTP) due to mixed content—HTTPS to HTTP. That’s why I’m trying to configure HAProxy to work with HTTPS, allowing me to test it again properly.

I think it will be more than enough even if curl could work internally on VPS

9

Mixed content has nothing to do with the SSL connection between Cloudflare and Haproxy.

Haproxy cannot fix mixed content warnings for you. This needs to be either fixed in the application or maybe Cloudflare has some bandaid fix for this, possible on the paid plans, I don’t know.

But your best bed to fix mixed content warning is in your application.

10

yes, the issue is that Cloudflare is running on https(3), my haproxy running on http(1.1)

The only way to fix mixed resources is to be able to test haproxy using https

reverting forced ssl settings back here are the logs:

  1. curl -v http://134.199.236.54:80/liveness

returns

{ “status”:“up”,“host”:“154d99270016” }

  1. curl -v https://134.199.236.54:443/livenes

returns

Trying 134.199.236.54:443…

  • Connected to 134.199.236.54 (134.199.236.54) port 443
  • ALPN: curl offers h2,http/1.1
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / X25519 / RSASSA-PSS
  • ALPN: server accepted h2
  • Server certificate:
  • subject: O=CloudFlare, Inc.; OU=CloudFlare Origin CA; CN=CloudFlare Origin Certificate
  • start date: Mar 27 00:54:00 2025 GMT
  • expire date: Mar 23 00:54:00 2040 GMT
  • subjectAltName does not match 134.199.236.54
  • SSL: no alternative certificate subject name matches target host name ‘134.199.236.54’
  • Closing connection
  • TLSv1.3 (OUT), TLS alert, close notify (256):
    curl: (60) SSL: no alternative certificate subject name matches target host name ‘134.199.236.54’

At least this should return something, correct? like the return I had using http

11

No, that is almost certainly not the issue.

You need to stop, remove all your workarounds for problems you think you had, and then explain your actual problem from the beginning.

2 Likes
12

Okay, I got my backend working miraculously when sending from Cloudflare to HAProxy directly

I see expected results

2025-03-27T18:54:53.766587+00:00 meetxdroplet haproxy[3486]: 45.9.230.8:53439 [27/Mar/2025:18:54:53.764] http_front~ http_back/web1 0/0/0/1/1 204 282 - - ---- 1/1/0/0/0 0/0 "OPTIONS https://example.com/users/whoami/read HTTP/2.0" TLSv1.3 TLS_AES_128_GCM_SHA256

2025-03-27T18:54:54.009929+00:00 meetxdroplet haproxy[3486]: 45.9.230.8:53439 [27/Mar/2025:18:54:54.007] http_front~ http_back/web1 0/0/0/1/1 301 434 - - ---- 1/1/1/1/0 0/0 "GET https://example.com/meetings?status=pending HTTP/2.0" TLSv1.3 TLS_AES_128_GCM_SHA256

2025-03-27T18:54:54.011392+00:00 meetxdroplet haproxy[3486]: 45.9.230.8:53439 [27/Mar/2025:18:54:54.008] http_front~ http_back/web1 0/0/0/2/2 500 462 - - ---- 1/1/0/0/0 0/0 "GET https://example.com/users/whoami/read HTTP/2.0" TLSv1.3 TLS_AES_128_GCM_SHA256

2025-03-27T18:54:54.257106+00:00 meetxdroplet haproxy[3486]: 45.9.230.8:53439 [27/Mar/2025:18:54:54.256] http_front~ http_back/web1 0/0/0/0/0 204 282 - - ---- 1/1/0/0/0 0/0 "OPTIONS https://example.com/meetings/?status=pending HTTP/2.0" TLSv1.3 TLS_AES_128_GCM_SHA256

2025-03-27T18:54:54.506011+00:00 meetxdroplet haproxy[3486]: 45.9.230.8:53439 [27/Mar/2025:18:54:54.502] http_front~ http_back/web1 0/0/0/2/2 200 329 - - ---- 1/1/0/0/0 0/0 "GET https://example.com/meetings/?status=pending HTTP/2.0" TLSv1.3 TLS_AES_128_GCM_SHA256

2025-03-27T18:54:57.812559+00:00 meetxdroplet haproxy[3486]: 45.9.230.8:53439 [27/Mar/2025:18:54:57.810] http_front~ http_back/web1 0/0/0/1/1 204 282 - - ---- 1/1/0/0/0 0/0 "OPTIONS https://example.com/users/profile/undefined HTTP/2.0" TLSv1.3 TLS_AES_128_GCM_SHA256

2025-03-27T18:54:58.056241+00:00 meetxdroplet haproxy[3486]: 45.9.230.8:53439 [27/Mar/2025:18:54:58.053] http_front~ http_back/web1 0/0/0/1/1 404 422 - - ---- 1/1/0/0/0 0/0 "GET https://example.com/users/profile/undefined HTTP/2.0" TLSv1.3 TLS_AES_128_GCM_SHA256

2025-03-27T19:01:14.157830+00:00 meetxdroplet haproxy[3486]: 85.214.12.13:55464 [27/Mar/2025:19:01:14.156] http_front http_front/<NOSRV> 0/-1/-1/-1/0 302 104 - - LR-- 1/1/0/0/0 0/0 "GET / HTTP/1.1" - -

2025-03-27T19:01:14.650864+00:00 meetxdroplet haproxy[3486]: 85.214.12.13:57526 [27/Mar/2025:19:01:14.487] http_front/2: SSL handshake failure (error:0A000418:SSL routines::tlsv1 alert unknown ca)

frontend http_front
bind *:80
bind *:443 ssl crt /etc/haproxy/certs/cloudflare.pem ssl-min-ver TLSv1.3
http-request redirect scheme https unless { ssl_fc }
default_backend http_back
acl exceeds_rate_limit sc_http_req_rate(0) gt 20
http-request set-log-level alert if exceeds_rate_limit
http-request deny if exceeds_rate_limit
option httplog
log-format “%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc”

What’s bothering me is the last two logs, but I understand they aren’t related and are beyond the scope of HAProxy itself. I’m just making sure I ask a relevant question here. The configuration looks fine, and requests are successfully reaching HAProxy. Thank you very much!)

13

I told you:

A private cloudflare certificate only works for cloudflare as a client.

If you want to bypass Cloudflare as well as send traffic through Cloudflare, you need a proper public Certificate, NOT the private Cloudflare certificate.

85.214.12.13 is not a Cloudflare IP address. It is a IONOS IP address, so this is not a request coming from cloudflare.

“unknown ca” means the client connecting from IONOS, which is not Cloudflare, does not recognize Cloudflare as a Certificate Authority, because Cloudflare is not in fact a Certificate Authority.

2 Likes