Strange cert chain with all certs in one file

Help!
1

Hi,

so I’ve put all my certs and private keys in a single file for convenience…

cat cert1_fullchain.crt cert1.key cert2_fullchain.crt cert2.key cert_wildcard_fullchain.crt cert_wildcard.key > ssl-certs.pem

…and configured it in my frontend like this:

frontend example_ssl
  bind FQDN:443 ssl alpn h2,http/1.1 crt /path/to/ssl-certs.pem ssl-min-ver TLSv1.2 ssl-max-ver TLSv1.3 ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256 ciphersuites TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256
  mode http
  ...

However, when testing this configuration the cert chain is messed up and clearly SNI is not working:

$ openssl s_client -servername test.example.com HAPROXY-FQDN:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = cert1.example.com
verify return:1
---
Certificate chain
 0 s:CN = cert1.example.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
 3 s:CN = *.example.com
   i:C = US, O = Let's Encrypt, CN = R3
 4 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 5 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
 6 s:CN = cert2.example.com
   i:C = US, O = Let's Encrypt, CN = R3
 7 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 8 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

As you can see, all certs are presented in the chain, instead of only one.
What am I doing wrong? Or is HAProxy doing something wrong?

The same happens if I put all certs into individual files and let HAProxy load all certs from this directory:

frontend example_ssl
  bind FQDN:443 ssl alpn h2,http/1.1 crt /path/to/ssl-dir/ ssl-min-ver TLSv1.2 ...

It only works if I combine each cert with its private key in a single file, and specify all these cert/key files in the config:

frontend example_ssl
  bind FQDN:443 ssl alpn h2,http/1.1 crt /path/to/cert1.pem crt /path/to/cert2.pem crt /path/to/cert_wildcard.pem ssl-min-ver TLSv1.2 ...

Using this config, SNI is working and only one cert is presented in the cert chain:

$ openssl s_client -servername test.example.com HAPROXY-FQDN:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.example.com
verify return:1
---
Certificate chain
 0 s:CN = *.example.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

Does this make any sense to anyone? Why does it get messed-up when everything is in a single file or directory?

I just wanted to keep things simple and load all certs from one file (or one directory), but this seems to confuse HAProxy.

My environment:

$ haproxy -vv
HA-Proxy version 2.0.27-dee3aad 2022/01/26 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_PCRE2=1 USE_REGPARM=1 USE_LINUX_TPROXY=1 USE_CRYPT_H=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.1k  FIPS 25 Mar 2021
Running on OpenSSL version : OpenSSL 1.1.1k  FIPS 25 Mar 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.4
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.23 2017-02-14
PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
	prometheus-exporter

Available filters :
	[SPOE] spoe
	[COMP] compression
	[CACHE] cache
	[TRACE] trace

Thanks

Frank

2

I’ve fiddled more with this and created the all-in-one certs file a bit differently:

# first add all certs + keys
cat cert1_cert.crt cert1.key cert_wildcard_cert.crt cert_wildcard.key cert2_cert.crt cert2.key > ssl-certs.pem
# finally add CA certs
cat cert1_ca.crt cert_wildcard_ca.crt cert2_ca.crt >> ssl-certs.pem

Now the SSL certificate chain shows each certificate exactly in the order in which they appear in the file:

$ openssl s_client -servername test.example.com HAPROXY_FQDN:443
...
---
Certificate chain
 0 s:CN = cert1.example.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:CN = *.example.com
   i:C = US, O = Let's Encrypt, CN = R3
 2 s:CN = cert1.example.com
   i:C = US, O = Let's Encrypt, CN = R3
 3 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 4 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
 5 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 6 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
 7 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 8 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

So I guess I’m doing it wrong, right? The “crt” option only accepts a file with a single certificate entity, it does NOT support multiple different certificates in the same file.

I think this sentence in the HAProxy documentation lead me to believe that multiple certs can be added in one file:

It designates a PEM file containing both the required certificates and any
associated private keys.

Maybe this also explains why it also does not work properly when specifying a directory using “crt /path/to/ssl-dir”, because it also expects each directory to contain only a single certificate entity.

Case solved, I guess. I have to use “crt-list” then, which makes this whole thing a bit more complicated, but that is probably the only way to do it.

Frank

3

No, you absolutely cannot concatenate multiple certificates into a single file, a single file must only contain certificate, intermediates, private key and dh-group (if necessary). In no circumstance must the file contain multiple certificates.

I agree that the documentation is a little misleading, what do you think about this text?

It designates a PEM file containing a certificate and any
associated private keys and intermediate certificates.

No, pointing crt to a directory with thousands of files, each containing a single certificate is a very widely and successfully used configuration.

You tested this and came to the conclusion that this doesn’t work either, but I think something went wrong in your tests. I’d suggest you retry and make sure all the files in the directory contain what you intend them to contain, and not more.

4

Thanks, Lukas! I think the proposed documentation change would be fine and avoid future misunderstanding. :slight_smile:

I think crt was not working because the certificate, private keys and CA chain were stored in individual files, so HAProxy had the impossible task to glue them all together somehow.