Layer6 invalid response: SSL handshake failure

Help!
1

Hey guys,

I have a setup with several backends, and where one backend is a third-party API provider which acts as a fallback in case our own servers go down.

Setting it up though, I’m running into issues with what appears to be establishing a TLS connection.
The error message is:
[WARNING] (281465) : Health check for server <fallback/provider> failed, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 24ms, status: 0/2 DOWN.

The following is my backend definition:

backend fallback
        mode http
        option httpchk
        option log-health-checks
        http-check connect ssl port 443
        http-check send meth POST uri /api/fallback hdr Content-Type application/json hdr Host fallback.provider.com body "{json body}"
        http-check expect status 200
        server fallback.provider.com:443 ssl verify none check inter 3s fall 3 rise 2  agent-check  agent-addr localhost  agent-port 10082  agent-inter 5s  agent-send Fallback/provider\n
        http-request set-header Host fallback.provider.com
        http-request set-header Path /api/fallback

I’m not really sure what the issue is. There is no custom certificate on my HAProxy server (would there need to be?) - everything is the same as standard:

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

I have tried using alternate providers. The issue doesn’t appear with every provider, only for certain ones. However, the one in which this issue appears is my preferred provider… So any help would be greatly appreciated!

Thanks

2

An update to this, after reading many a forum entry (with a certain very helpful @lukastribus appearing in most of them):

Initially, I was not able to forward traffic via HAProxy to the relevant backend. However, after specifying an SNI consisting of the relevant hostname, I can successfully forward traffic via HAProxy. However, I’m still getting a L6RSP error for the health checks. I have tried adding check-sni with the same hostname, but with no luck.

My hunch is that this has to do with SNI, mostly because I wasn’t able to forward requests prior to adding the SNI parameter. But I’m not sure what I’m missing at this point. Been scouring forums trying to find a solution.

EDIT: Establishing a handshake by making a simple curl request from the HAProxy host works totally fine. I also tried, as I’ve seen elsewhere, to make a handshake using openssl s_client :

CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = <fallback>
verify return:1
---
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = <fallback>
   i:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
 1 s:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
   i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFdzCCBRygAwIBAgIQCa1Goe1f1OvVsvito85j1zAKBggqhkjOPQQDAjBKMQsw
CQYDVQQGEwJVUzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjEgMB4GA1UEAxMX
Q2xvdWRmbGFyZSBJbmMgRUNDIENBLTMwHhcNMjIwMTA0MDAwMDAwWhcNMjMwMTAz
MjM1OTU5WjBqMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQG
A1UEBxMNU2FuIEZyYW5jaXNjbzEZMBcGA1UEChMQQ2xvdWRmbGFyZSwgSW5jLjET
MBEGA1UEAxMKbW9yYWxpcy5pbzBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABE/I
TgCVS2AThyotMU1C1Wf8rP1L1FTzRqUmrqHarCAuOzLFTCGK7BWpdjVn1jS8naO1
sv6IvU1PVZBEeSD7vN6jggPCMIIDvjAfBgNVHSMEGDAWgBSlzjfq67B1DpRniLRF
+tkkEIeWHzAdBgNVHQ4EFgQUQIjcKS5WDNvA5UJqIKM/q+j8Re4wgYUGA1UdEQR+
MHyCGyouZGVlcC1pbmRleC1kZXYubW9yYWxpcy5pb4ISKi5zdGFnZS5tb3JhbGlz
Lmlvggptb3JhbGlzLmlvgh0qLmRlZXAtaW5kZXgtc3RhZ2UubW9yYWxpcy5pb4IQ
Ki5kZXYubW9yYWxpcy5pb4IMKi5tb3JhbGlzLmlvMA4GA1UdDwEB/wQEAwIHgDAd
BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwewYDVR0fBHQwcjA3oDWgM4Yx
aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0Nsb3VkZmxhcmVJbmNFQ0NDQS0zLmNy
bDA3oDWgM4YxaHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0Nsb3VkZmxhcmVJbmNF
Q0NDQS0zLmNybDA+BgNVHSAENzA1MDMGBmeBDAECAjApMCcGCCsGAQUFBwIBFhto
dHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMwdgYIKwYBBQUHAQEEajBoMCQGCCsG
AQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2VydC5jb20wQAYIKwYBBQUHMAKGNGh0
dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9DbG91ZGZsYXJlSW5jRUNDQ0EtMy5j
cnQwDAYDVR0TAQH/BAIwADCCAYAGCisGAQQB1nkCBAIEggFwBIIBbAFqAHcA6D7Q
2j71BjUy51covIlryQPTy9ERa+zraeF3fW0GvW4AAAF+JU6JYgAABAMASDBGAiEA
r3UVJ5XbQ4tA1+JjCke1fELBaD2zbqcprLHRldOgJUECIQDVK4HEjQrSgTys8E39
aTM/lhks4ohOVPndoSt2vX0uewB3ADXPGRu/sWxXvw+tTG1Cy7u2JyAmUeo/4Srv
qAPDO9ZMAAABfiVOiQEAAAQDAEgwRgIhAOgp0lyBZqOTIBhOOMl9tgSVwT6piPkR
ntDZuw0NqXVlAiEAr/HKqU8afzvf/VrwCrC8E1+nBLB6i3QMLCAjLcpyC+EAdgCz
c3cH4YRQ+GOG1gWp3BEJSnktsWcMC4fc8AMOeTalmgAAAX4lTokgAAAEAwBHMEUC
ICfUHcH+0slqKy3zo3rk4ypOLGFMW/NDiko3Lm3t4rEyAiEA6x2TLg84taCZKei7
3Em5r3DTTur/HfNiJi0V5NlBkxwwCgYIKoZIzj0EAwIDSQAwRgIhAJclQz7/MoRH
Du3MXlx++wv+8bfxr/ptlckeSmtGdgiwAiEAwrxZ8zNd2FPKCSVQupPchPA9LNqd
lghuHXJhgk+RTYo=
-----END CERTIFICATE-----
subject=C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = <fallback>

issuer=C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2693 bytes and written 399 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: B12B1BF9D37DA81B9B97E45A9D09E7B942290FAEEB8DED8F2803CC8D9FEF8C0D
    Session-ID-ctx:
    Resumption PSK: 9BBF589B65140427AE0B941725D6ED6486C68D8FFE340ABD1C07B912B332AFDC67C3441DAD14FD4A12797B960594A0AD
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64799 (seconds)
    TLS session ticket:
    0000 - 77 d2 69 6d 5d 7a 49 9d-93 84 1b c4 83 77 88 99   w.im]zI......w..
    0010 - bf be f4 bb 70 83 cc 06-f3 65 e7 af c2 8d 08 0f   ....p....e......
    0020 - 4f 17 47 b9 a1 f6 49 aa-cb 7d 1f 2d e5 93 72 70   O.G...I..}.-..rp
    0030 - 1c 97 62 14 11 fc 07 85-b2 04 55 e0 00 90 8a 82   ..b.......U.....
    0040 - 47 35 e2 41 25 10 ab fd-17 8d 22 90 b1 2e 36 f6   G5.A%....."...6.
    0050 - f8 54 4b cf 19 f6 32 9e-cf 2b ee 66 98 29 7b 3e   .TK...2..+.f.){>
    0060 - 3e e7 29 a1 3f 50 c0 a1-83 e5 0e 12 3c 6c a4 fd   >.).?P......<l..
    0070 - 13 1b 43 b2 3c 07 40 fa-21 71 e4 7c 71 2c 3b 5b   ..C.<.@.!q.|q,;[
    0080 - a9 af f1 ed ed e2 a3 72-0a 33 a5 e4 30 9a b5 8c   .......r.3..0...
    0090 - 4a ae 0a 58 09 f7 5d 80-6e 1d 93 09 d2 72 1c 4d   J..X..].n....r.M
    00a0 - 4f 13 e5 d8 d8 6d c9 0a-dc 5c f7 17 46 d7 02 dd   O....m...\..F...
    00b0 - f7 8a 2a 41 e3 7d 86 b3-8c e9 bc 6c b5 bd 57 aa   ..*A.}.....l..W.

    Start Time: 1654527241
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 4BD9F321497BDAB8EB1F8FF4A600A347B88E54174221B09D2757AF551455D4F5
    Session-ID-ctx:
    Resumption PSK: 775E372D1F2D5C8CE978EF769DF2B4EF8CF9CE37B523C6CED3F3D9B12B85DAF2E18F61E51CA5A315DD9A323A2A81A5B9
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64799 (seconds)
    TLS session ticket:
    0000 - 77 d2 69 6d 5d 7a 49 9d-93 84 1b c4 83 77 88 99   w.im]zI......w..
    0010 - 3b 90 4e 0a 72 e0 af aa-78 7f 2c c5 87 b9 b5 65   ;.N.r...x.,....e
    0020 - 8e 98 65 98 dc ab 57 1a-ee 8f 89 58 26 9a 00 ac   ..e...W....X&...
    0030 - f8 09 89 39 8e a5 8c 56-59 21 e5 55 6e 05 2c cc   ...9...VY!.Un.,.
    0040 - 69 99 83 b6 ba fa 96 dd-d7 79 86 ea a5 90 04 3a   i........y.....:
    0050 - a5 a7 21 ba f1 ec d7 66-14 cf 00 88 70 70 2a b8   ..!....f....pp*.
    0060 - 8a 15 dc 01 91 9a ac 14-8f d5 40 50 7e 20 79 3b   ..........@P~ y;
    0070 - d5 7b b7 15 df 23 e8 1b-60 67 ce d0 7d 1d 90 32   .{...#..`g..}..2
    0080 - 98 01 41 7e bd 64 50 2f-89 97 fc 64 cf 6a fe 1a   ..A~.dP/...d.j..
    0090 - 0e b5 ae a1 3a ff 43 a1-0d 1a 1b 9c 16 21 a2 4a   ....:.C......!.J
    00a0 - 38 18 6a 2c 01 31 75 f5-f7 ef 23 8d 28 e2 e3 31   8.j,.1u...#.(..1
    00b0 - 37 10 80 b9 6f 4e 3b 7b-4b ae d8 6c 40 48 8f 25   7...oN;{K..l@H.%

    Start Time: 1654527241
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
closed

haproxy -vv:

ubuntu@lb1:~$ haproxy -vv
HAProxy version 2.4.17-1ppa1~focal 2022/05/14 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.17.html
Running on: Linux 5.13.0-1025-aws #27~20.04.1-Ubuntu SMP Thu May 19 15:17:13 UTC 2022 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-ngwd1k/haproxy-2.4.17=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_PROMEX=1
  DEBUG   =

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC +PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
Running on OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[SPOE] spoe
	[CACHE] cache
	[FCGI] fcgi-app
	[COMP] compression
	[TRACE] trace
3

Does it actually make sense to health check this backend server? Since this is an external service, I’m wondering if there are any benefits at all, or if you are just hammering an external service, maybe even get your IP address banned, all for nothing.

Since you are not using standard health check, but a specific configuration with http-check connect, I think the sni value for health checking needs to be configured there.

So remove check-sni and change the http-check configuration to:

http-check connect ssl port 443 sni fallback.provider.com

FYI: if you are trying to hide the destination hostname, don’t post the complete public certificate of it, because it will also contain that hostname.

1 Like
4

Thanks a bunch!! That indeed did the trick. However, I’m now left with an unexpected 401 unauthorised for my health-checks. Good thing that the L6 error is gone, though.

And whether it makes sense - only marginally. I’m using HAProxy largely as a failover ‘router’ between a mix of private-hosted nodes, and then some public providers as emergency fallbacks. In the grand scheme of things I don’t know if it will ever turn out to be worth the trouble, haha.

And yup, I realise that I left in the hostname there. In case it would be beneficial for Tshooting :slight_smile:

Thanks a bunch, this will be the second time you’ve solved my issue

1 Like
5

Sorry to be bumping this up again, but I’m not able to figure out that 401 unauthorised error message on my health checks. Do you have any idea what the issue could be?

Sending the same request via curl works fine, from the same host machine too. It’s just a POST request with a certain body, including Content-Type application/json, towards a specific path (my paid endpoint).
I’ve been scratching my head for a bit and just don’t see why I would be receiving a 401 unauthorised. Since it’s HTTPS it’s hard for me to look at the traffic. But could it be that as part of the communication the client is supposed to receive a cookie or similar before being allowed to submit a request, and that this is something HAProxy may be struggling with? But then on the other hand, forwarding requests is not a problem…

Is there perhaps a way to log HTTP health check requests and responses, so that I can verify/look at the contents of the POST request and response?

EDIT: solved the issue. The JSON-RPC method I was calling was not allowed on the provider side; I was using a different one when forwarding traffic. Please disregard :slight_smile:

1 Like
6

FWIW: in this case I would temporarily use a check towards a local HTTP only server, with otherwise the same check configuration and capture the traffic unencrypted this way. That should be enough to understand what happens at HTTP level.

1 Like
7

Thanks!
This is similar to what I did: set up a new back-end definition, which sent identical health-checks (http) towards a front-end with the ssl backend in question being default. That way I could listen to both parts of the comms locally