SSL handshake failure on Cloudfront backend server

Help!
1

Hello,
I have a HAProxy instance that should serve as a proxy to Here.com maps, adding the API key to all passing requests. However the following backend configuration fails with messages 'SSL handshake failure

backend freehere_maps_redirect
http-send-name-header Host
http-request set-uri http://%[req.hdr(Host)]%[path]?apiKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxx&%[query]
server 1.base.maps.ls.hereapi.com 1.base.maps.ls.hereapi.com:443 check ssl verify none check resolvers mydns
server 2.base.maps.ls.hereapi.com 2.base.maps.ls.hereapi.com:443 check ssl verify none check resolvers mydns
server 3.base.maps.ls.hereapi.com 3.base.maps.ls.hereapi.com:443 check ssl verify none check resolvers mydns
server 4.base.maps.ls.hereapi.com 4.base.maps.ls.hereapi.com:443 check ssl verify none check resolvers mydns

I have tried adding options like sni str(1.base.maps.ls.hereapi.com), but the error persists. Here is what ssldump shows:

New TCP connection #1: localhost.localdomain(34914) ↔ server-205-251-219-9.arn1.r.cloudfront.net(443)
1 1 0.1257 (0.1257) C>S Handshake
ClientHello
Version 3.3
cipher suites
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_DH_DSS_WITH_AES_256_GCM_SHA384
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384
TLS_DH_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_DSS_WITH_AES_256_CBC_SHA256
TLS_DH_RSA_WITH_AES_256_CBC_SHA256
TLS_DH_DSS_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DH_RSA_WITH_AES_256_CBC_SHA
TLS_DH_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA
TLS_DHE_DSS_WITH_CAMELLIA_256_CBC_SHA
TLS_DH_RSA_WITH_CAMELLIA_256_CBC_SHA
TLS_DH_DSS_WITH_CAMELLIA_256_CBC_SHA
TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_CAMELLIA_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_DH_DSS_WITH_AES_128_GCM_SHA256
TLS_DHE_DSS_WITH_AES_128_GCM_SHA256
TLS_DH_RSA_WITH_AES_128_GCM_SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
TLS_DHE_DSS_WITH_AES_128_CBC_SHA256
TLS_DH_RSA_WITH_AES_128_CBC_SHA256
TLS_DH_DSS_WITH_AES_128_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DH_RSA_WITH_AES_128_CBC_SHA
TLS_DH_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_SEED_CBC_SHA
TLS_DHE_DSS_WITH_SEED_CBC_SHA
TLS_DH_RSA_WITH_SEED_CBC_SHA
TLS_DH_DSS_WITH_SEED_CBC_SHA
TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA
TLS_DHE_DSS_WITH_CAMELLIA_128_CBC_SHA
TLS_DH_RSA_WITH_CAMELLIA_128_CBC_SHA
TLS_DH_DSS_WITH_CAMELLIA_128_CBC_SHA
TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_SEED_CBC_SHA
TLS_RSA_WITH_CAMELLIA_128_CBC_SHA
TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DH_DSS_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_IDEA_CBC_SHA
TLS_ECDHE_RSA_WITH_RC4_128_SHA
TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
TLS_ECDH_RSA_WITH_RC4_128_SHA
TLS_ECDH_ECDSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
compression methods
NULL
1 2 0.2422 (0.1164) S>C Alert
level fatal
value handshake_failure
1 3 0.2422 (0.0000) S>C Alert
level warning
value close_notify
1 0.2428 (0.0005) S>C TCP FIN
1 0.2599 (0.0171) C>S TCP RST

However, Curl can establish TLS connection to this server:

New TCP connection #1: localhost.localdomain(60630) ↔ server-13-33-243-73.hel50.r.cloudfront.net(443)
1 1 0.2425 (0.2425) C>S Handshake
ClientHello
Version 3.3
cipher suites
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
Unknown value 0xcca9
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Unknown value 0xcca8
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_GCM_SHA256
Unknown value 0xccaa
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_3DES_EDE_CBC_SHA
TLS_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_RC4_128_MD5
compression methods
NULL
1 2 0.3793 (0.1367) S>C Handshake
ServerHello
Version 3.3
session_id[32]=
f8 f2 8b 5b 48 eb bb 7f d8 5c 70 e0 9c 86 30 0d
f7 3d 6c 52 2f 66 b7 33 84 09 1f bb 25 14 d9 f6
cipherSuite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
compressionMethod NULL
1 3 0.3994 (0.0201) S>C Handshake
Certificate
1 4 0.3994 (0.0000) S>C Handshake
ServerKeyExchange
1 5 0.3994 (0.0000) S>C Handshake
ServerHelloDone
and so on

I tried to specify ciphers ECDHE-RSA-AES128-GCM-SHA256 in the server configuration, but the result is the same.

The following HAProxy versions were tested on CentOS7:

HA-Proxy version 1.8.4-1deb90d 2018/02/08
Copyright 2000-2018 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2

and

HA-Proxy version 2.2.2 2020/07/31 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.2.html
Running on: Linux 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-f
ield-initializers -Wtype-limits
OPTIONS = USE_PCRE2=1 USE_LINUX_TPROXY=1 USE_CRYPT_H=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2
+TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51D
EGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=1).
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2

What else can be done here?

Thanks
Dimitri

2

First of all, don’t health check other peoples cloud servers, this is just gonna get your IP banned from using there services.

Health checks, if enabled, would also need a separate SNI configuration (check-sni), which makes this even more complicated.

However I’d recommend to just disable health-checks, because it doesn’t make sense for the “forwarding” proxy scenario.

Use option redispatch and retries for fail-over from one server to another.

3

Thanks for the idea with check-sni.

The problem is that these backend servers often change IP addresses. For this reason I had to enable DNS resolution, which, according to the manual, “is triggered by health checks. This makes health checks mandatory to allow DNS resolution”.

What is interesting is that on another set of Here.com servers a very similar configuration works well:

        http-send-name-header Host
        http-request set-uri http://%[req.hdr(Host)]%[path]?app_code=xxxxxxxxxxxxxxxxxxxxx&app_id=xxxxxxxxxxxxxxxxxxxx&%[query]
        server 1.base.maps.api.here.com 1.base.maps.api.here.com:443 ssl verify none check resolvers mydns resolve-prefer ipv4
        server 2.base.maps.api.here.com 2.base.maps.api.here.com:443 ssl verify none check resolvers mydns resolve-prefer ipv4
        server 3.base.maps.api.here.com 3.base.maps.api.here.com:443 ssl verify none check resolvers mydns resolve-prefer ipv4
        server 4.base.maps.api.here.com 4.base.maps.api.here.com:443 ssl verify none check resolvers mydns resolve-prefer ipv4
4

Unfortunately, changing sni to check-sni didn’t help. It adds the following to handshake:

          server_name
              host_name: str(1.base.maps.ls.hereapi.com)

But the result is the same, handshake_failure.
It seems to be a problem specific to Cloudfront, because the other set of servers I mentioned in the last message runs on Akamai.

5

Here’s a working configuration:

  http-request set-uri http://%[req.hdr(Host)]%[path]?apiKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&%[query]
  server 1.base.maps.ls.hereapi.com 1.base.maps.ls.hereapi.com:443 ssl sni req.hdr(Host) verify none
  server 2.base.maps.ls.hereapi.com 2.base.maps.ls.hereapi.com:443 ssl sni req.hdr(Host) verify none
  server 3.base.maps.ls.hereapi.com 3.base.maps.ls.hereapi.com:443 ssl sni req.hdr(Host) verify none
  server 4.base.maps.ls.hereapi.com 4.base.maps.ls.hereapi.com:443 ssl sni req.hdr(Host) verify none

However, it breaks as soon as I add resolvers or check. Without resolvers, it breaks (in a different way, though) when the DNS round robin changes the IP addresses.

One more interesting thing I noted while playing with Curl is that the requests pass when the server is identified with a DNS name, but fail during SSL handshake when the name is replaced with IP, even though Host header is present:

1.base.maps.ls.hereapi.com. 3586 IN     CNAME   d2t4vhngii5504.cloudfront.net.
d2t4vhngii5504.cloudfront.net. 46 IN    A       99.84.11.118
d2t4vhngii5504.cloudfront.net. 46 IN    A       99.84.11.65
d2t4vhngii5504.cloudfront.net. 46 IN    A       99.84.11.19
d2t4vhngii5504.cloudfront.net. 46 IN    A       99.84.11.54

$ curl -H 'Host: 1.base.maps.ls.hereapi.com' 'https://1.base.maps.ls.hereapi.com/maptile/2.1/version?apiKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
MRS: 2.1.70.15
MOS: 2.2.37-152.f3a0a2d9
Map: 8.30.115.154

$ curl -H 'Host: 1.base.maps.ls.hereapi.com' 'https://d2t4vhngii5504.cloudfront.net/maptile/2.1/version?apiKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'
MRS: 2.1.70.15
MOS: 2.2.37-152.f3a0a2d9
Map: 8.30.115.154

$ curl -H 'Host: 1.base.maps.ls.hereapi.com' 'https://99.84.11.118/maptile/2.1/version?apiKey=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
6

I never said that, if you need health checking, you need BOTH. After all, there is no point in having a working health check, when your actually traffic doesn’t work.

server 1.base.maps.ls.hereapi.com 1.base.maps.ls.hereapi.com:443 ssl sni req.hdr(Host) check check-sni 1.base.maps.ls.hereapi.com verify none

This WAS true in haproxy 1.7. In haproxy 1.8 and later, you don’t need health checks to trigger DNS resolution.

7

I tried using sni and check-sni together, too. Didn’t help, unfortunately. Handshake failure.
Also, I tried to remove checks. The configuration works only as long as there are no resolvers. With resolvers it returns 502.

8

So, check-sni was the key. Alone, without sni. But I used it in a wrong way. check-sni should be followed by a simple DNS name, as in your example above, not str() or req.hdr() call. The working configuration is:

server 1.base.maps.ls.hereapi.com 1.base.maps.ls.hereapi.com:443 ssl verify none resolvers mydns check-sni 1.base.maps.ls.hereapi.com check

Thanks for your help, @lukastribus!