Single domain with RSA and ECDSA certs

I’m having a difficult time to setup TLS termination on HAProxy,

I have HAProxy 2.6.7, I have 2 certificates for RSA and ECDSA, it is set up ad described in the docs:

bind 10.0.0.3:443 ssl crt /etc/ssl/certs/mycert.pem

ECDSA flow works:
openssl s_client -connect localhost:8883 -cipher ECDSA -CAfile ecdsa.crt -tls1_2

RSA does not
openssl s_client -connect localhost:8883 -cipher RSA -CAfile rsa.crt -tls1_2

haproxy_1   | fd[0xe] OpenSSL error[0x1417a0c1] tls_post_process_client_hello: no shared cipher
haproxy_1   | 172.18.0.1:60336 [20/Dec/2022:16:37:25.542] mqtt/1: SSL handshake failure

what could be a reason

This is the reason.

Provide the complete haproxy configuration, ciphers are likely miss-configured.

this is the global config:

global
  ssl-default-bind-options ssl-min-ver TLSv1.2
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384
frontend mqtt
  bind *:8883 ssl crt /usr/local/etc/haproxy/local/ssl/mqp.pem

available ciphers on the host

openssl ciphers
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:DHE-RSA-AES256-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:DHE-RSA-AES128-SHA:RSA-PSK-AES256-GCM-SHA384:DHE-PSK-AES256-GCM-SHA384:RSA-PSK-CHACHA20-POLY1305:DHE-PSK-CHACHA20-POLY1305:ECDHE-PSK-CHACHA20-POLY1305:AES256-GCM-SHA384:PSK-AES256-GCM-SHA384:PSK-CHACHA20-POLY1305:RSA-PSK-AES128-GCM-SHA256:DHE-PSK-AES128-GCM-SHA256:AES128-GCM-SHA256:PSK-AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:ECDHE-PSK-AES256-CBC-SHA384:ECDHE-PSK-AES256-CBC-SHA:SRP-RSA-AES-256-CBC-SHA:SRP-AES-256-CBC-SHA:RSA-PSK-AES256-CBC-SHA384:DHE-PSK-AES256-CBC-SHA384:RSA-PSK-AES256-CBC-SHA:DHE-PSK-AES256-CBC-SHA:AES256-SHA:PSK-AES256-CBC-SHA384:PSK-AES256-CBC-SHA:ECDHE-PSK-AES128-CBC-SHA256:ECDHE-PSK-AES128-CBC-SHA:SRP-RSA-AES-128-CBC-SHA:SRP-AES-128-CBC-SHA:RSA-PSK-AES128-CBC-SHA256:DHE-PSK-AES128-CBC-SHA256:RSA-PSK-AES128-CBC-SHA:DHE-PSK-AES128-CBC-SHA:AES128-SHA:PSK-AES128-CBC-SHA256:PSK-AES128-CBC-SHA

haproxy -vv

root@prism:/# haproxy -vv
HAProxy version 2.6.7-c55bfdb 2022/12/02 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.7.html
Running on: Linux 5.10.104-linuxkit #1 SMP Thu Mar 17 17:08:06 UTC 2022 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_PROMEX=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -ENGINE +GETADDRINFO +OPENSSL +LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC +PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
Running on OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace

certificate loaded as:
/usr/local/etc/haproxy/local/ssl/mqp.pem
and in
/usr/local/etc/haproxy/local/ssl

  • mqp.pem.ecdsa
  • mqp.pem.rsa
echo 'show ssl cert' | socat unix-connect:/var/run/haproxy.sock stdio
# filename
/usr/local/etc/haproxy/local/ssl/mqp.pem.ecdsa
/usr/local/etc/haproxy/local/ssl/mqp.pem.rsa

anything else will be helpful?

When instructing openssl to use RSA ciphers, you are restricting openssl to only a few RSA ciphers:

lukas@dev:~$ openssl ciphers 'RSA'
TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:AES256-GCM-SHA384:AES256-CCM8:AES256-CCM:ARIA256-GCM-SHA384:AES128-GCM-SHA256:AES128-CCM8:AES128-CCM:ARIA128-GCM-SHA256:AES256-SHA256:CAMELLIA256-SHA256:AES128-SHA256:CAMELLIA128-SHA256:NULL-SHA256:AES256-SHA:CAMELLIA256-SHA:AES128-SHA:SEED-SHA:CAMELLIA128-SHA:NULL-SHA:NULL-MD5

None of those match the RSA ciphers you put into your haproxy config, because RSA does not cover ECDHE (RSA) ciphers you configured in haproxy:

ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384

So your test command needs to be:

openssl s_client -connect localhost:8883 -cipher ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384 -CAfile rsa.crt -tls1_2

You could also specify aRSA instead of RSA, as per /docs/man1.1.1/man1/ciphers.html

To be more precise, as per the openssl manual:

RSA is an alias for kRSA; but with ECDHE ciphers RSA is not used for key exchange, ECDHE is, so thats why kRSA does not imply ECDHE-RSA.

In ECDHE-RSA RSA is only used for authentication, which is why aRSA is the correct keyword.

sorry, I forgot to mention this as well, I did the test with specific ciphers, the same as you did, was the same result, so far I’m really puzzled. Is there a way to instruct haproxy to produce more diagnostics from openssl?

No, I don’t think so, haproxy already provides the error that OpenSSL returns, there is nothing more that haproxy could provide in this case.

Try sigalgs as opposed to ciphers (or try both) in the client, this is more appropriate to simulate a RSA only speaking client:

-sigalgs "RSA-PSS+SHA512:RSA-PSS+SHA384:RSA-PSS+SHA256:RSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA224:RSA+SHA1"

@lukastribus i am facing the same problem as well. For me everything works fine with haproxy 2.0 but as soon as i moved to 2.8 i see it nevers loads my RSA certs into CTX . Is there any change in hybrid configuration ?

Provide the full configuration, full output of haproxy -vv as well as the exact output of the command that makes you think that it does not work.

Here is my config

global
  maxconn 16000
  nbthread 1
  maxsslrate 500
  maxsslconn 18000
  tune.bufsize 32768
  tune.ssl.lifetime 3600
  log /dev/log local5

  stats socket ipv4@127.0.0.1:9999 level admin
  stats socket /var/run/haproxy.sock mode 666 level admin
  stats timeout 2m

  ssl-default-bind-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:AES256-SHA:AES128-SHA:DHE-RSA-AES128-SHA
  ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
  ssl-default-bind-options ssl-min-ver TLSv1.0  ssl-max-ver TLSv1.3 no-tls-tickets

defaults
  log     global
  mode    http
  option  log-health-checks
  option  log-separate-errors
  option  dontlognull
  log-format '%tr %ci:%cp %fp/%fc/%Th/%Tq/%rc %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}HP'
  option  splice-auto
  option  socket-stats
  option  redispatch
  maxconn 5000
  timeout connect 3s
  timeout client 300s
  timeout server 600s
  timeout client-fin 1s
  timeout server-fin 1s



listen ssl


  bind 0.0.0.0:8443   tfo ssl crt /usr/local/HAProxy.pem  npn http/1.1 curves X25519:P-256:P-384:P-521 prefer-client-ciphers


  maxconn 30000
  rate-limit sessions 200
  option tcp-smart-connect
  option forwardfor
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  http-request set-header X-Forwarded-Port %[dst_port]
  default_backend idpserver

And my haproxy -vv

[root@cucm-8 ~]# ./haproxy -vv
HAProxy version 2.8.5-aaba8d0 2023/12/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.5.html
Running on: Linux 4.18.0-372.16.1.el8_6.x86_64 #1 SMP Wed Jul 13 03:56:16 EDT 2022 x86_64
Build options :
  TARGET  = custom
  CPU     = generic
  CC      = cc
  CFLAGS  = -m64 -march=x86-64 -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_EPOLL=1 USE_THREAD=1 USE_LINUX_SPLICE=1 USE_OPENSSL=1 USE_SLZ= USE_CPU_AFFINITY=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES -ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ENGINE +EPOLL -EVPORTS -GETADDRINFO -KQUEUE -LIBATOMIC -LIBCRYPT -LINUX_CAP +LINUX_SPLICE -LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT -RT -SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD -TFO +THREAD -THREAD_DUMP -TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1t.***
Running on OpenSSL version : OpenSSL 1.1.1t.***
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity("identity")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): no
Built with gcc compiler version 8.5.0 20210514 (Red Hat 8.5.0-10)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

here is my certificates
[root@cucm-8 ~]# ls -ltr /usr/local/
-rwxr-xr-x. 1 certbase ccmbase 4119 Dec 13 09:00 HAProxy.pem.rsa
-rwxr-xr-x. 1 certbase ccmbase 1173 Dec 13 09:00 HAProxy.pem.ecdsa

here problem was when i connect to my port using below openssl commands it works fine .
openssl s_client -connect localhost:8443 -tls1_3

but all of my below executions it fails to connect.
openssl s_client -connect localhost:8443 -tls1_2 -cipher ECDHE-RSA-AES256-GCM-SHA384
openssl s_client -connect localhost:8443 -tls1_2 -cipher ECDHE-ECDSA-AES256-GCM-SHA384

fd[0x9] OpenSSL error[0x1417a0c1] tls_post_process_client_hello: no shared cipher
fd[0x9] OpenSSL error[0x1417a0c1] tls_post_process_client_hello: no shared cipher

openssl s_client -connect localhost:8443 -sigalgs RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512

fd[0x9] OpenSSL error[0x14201076] tls_choose_sigalg: no suitable signature algorithm

i have hidden the build number of openssl we use but it has all the vulnerability fixes.

also to add more to the context if i add below bind lines

bind 0.0.0.0:8443 tfo ssl crt /usr/local/HAProxy.pem.rsa npn http/1.1 curves X25519:P-256:P-384:P-521 prefer-client-ciphers
bind 0.0.0.0:8443 tfo ssl crt /usr/localHAProxy.pem.ecdsa npn http/1.1 curves X25519:P-256:P-384:P-521 prefer-client-ciphers

I see it negotiates few times and other times it fail with same above error. So it all looks to me was how haproxy loads the ssl context matters here i believe .

Any anomally you see in my config file ?

So what you are saying is that TLSv1.2 does not work. Not with RSA and not with ECC certificate. So in other words, your problem is unrelated to the certificates, but is related to TLSv1.2.

I suggest you remove some of the more specific configuration and try to find if you can make it work in a standard configuration. This includes npn, curves, prefer-client-ciphers and the global ssl-default-bind-* parameters.

That doesn’t make sense, but can you at least tell us if the number is the same or different (between Built with and Running on)?

It also landed on a github issue: Single domain with RSA and ECDSA certs · Issue #2392 · haproxy/haproxy · GitHub .

The certificate selection mecanism uses the servername, if you don’t specify a servername it will fallback on the first certificate declared.

If you want to correctly test this you should uses -servername in your openssl command or curl:

curl -k --tls-max 1.2 --ciphers ECDHE-RSA-AES256-GCM-SHA384 https://localhost:8443
openssl s_client -servername localhost -connect localhost:8443 -tls1_2 -cipher ECDHE-ECDSA-AES256-GCM-SHA384

Your certificates must also have a CN or a SNI. If that’s not the case you can specify a filter with a crt-list, but clients won’t be able to verify correctly the certificate.