Layer6 invalid response: SSL handshake failure

Hey guys,

I have a setup with several backends, and where one backend is a third-party API provider which acts as a fallback in case our own servers go down.

Setting it up though, I’m running into issues with what appears to be establishing a TLS connection.
The error message is:
[WARNING] (281465) : Health check for server <fallback/provider> failed, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 24ms, status: 0/2 DOWN.

The following is my backend definition:

backend fallback
        mode http
        option httpchk
        option log-health-checks
        http-check connect ssl port 443
        http-check send meth POST uri /api/fallback hdr Content-Type application/json hdr Host body "{json body}"
        http-check expect status 200
        server ssl verify none check inter 3s fall 3 rise 2  agent-check  agent-addr localhost  agent-port 10082  agent-inter 5s  agent-send Fallback/provider\n
        http-request set-header Host
        http-request set-header Path /api/fallback

I’m not really sure what the issue is. There is no custom certificate on my HAProxy server (would there need to be?) - everything is the same as standard:

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See:
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

I have tried using alternate providers. The issue doesn’t appear with every provider, only for certain ones. However, the one in which this issue appears is my preferred provider… So any help would be greatly appreciated!


An update to this, after reading many a forum entry (with a certain very helpful @lukastribus appearing in most of them):

Initially, I was not able to forward traffic via HAProxy to the relevant backend. However, after specifying an SNI consisting of the relevant hostname, I can successfully forward traffic via HAProxy. However, I’m still getting a L6RSP error for the health checks. I have tried adding check-sni with the same hostname, but with no luck.

My hunch is that this has to do with SNI, mostly because I wasn’t able to forward requests prior to adding the SNI parameter. But I’m not sure what I’m missing at this point. Been scouring forums trying to find a solution.

EDIT: Establishing a handshake by making a simple curl request from the HAProxy host works totally fine. I also tried, as I’ve seen elsewhere, to make a handshake using openssl s_client :

depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
verify return:1
depth=0 C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = <fallback>
verify return:1
Certificate chain
 0 s:C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = <fallback>
   i:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
 1 s:C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3
   i:C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
Server certificate
subject=C = US, ST = California, L = San Francisco, O = "Cloudflare, Inc.", CN = <fallback>

issuer=C = US, O = "Cloudflare, Inc.", CN = Cloudflare Inc ECC CA-3

No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
SSL handshake has read 2693 bytes and written 399 bytes
Verification: OK
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 256 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
Post-Handshake New Session Ticket arrived:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: B12B1BF9D37DA81B9B97E45A9D09E7B942290FAEEB8DED8F2803CC8D9FEF8C0D
    Resumption PSK: 9BBF589B65140427AE0B941725D6ED6486C68D8FFE340ABD1C07B912B332AFDC67C3441DAD14FD4A12797B960594A0AD
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64799 (seconds)
    TLS session ticket:
    0000 - 77 d2 69 6d 5d 7a 49 9d-93 84 1b c4 83 77 88 99]zI......w..
    0010 - bf be f4 bb 70 83 cc 06-f3 65 e7 af c2 8d 08 0f   ....p....e......
    0020 - 4f 17 47 b9 a1 f6 49 aa-cb 7d 1f 2d e5 93 72 70   O.G...I..}.-..rp
    0030 - 1c 97 62 14 11 fc 07 85-b2 04 55 e0 00 90 8a 82   ..b.......U.....
    0040 - 47 35 e2 41 25 10 ab fd-17 8d 22 90 b1 2e 36 f6   G5.A%....."...6.
    0050 - f8 54 4b cf 19 f6 32 9e-cf 2b ee 66 98 29 7b 3e   .TK...2..+.f.){>
    0060 - 3e e7 29 a1 3f 50 c0 a1-83 e5 0e 12 3c 6c a4 fd   >.).?P......<l..
    0070 - 13 1b 43 b2 3c 07 40 fa-21 71 e4 7c 71 2c 3b 5b   ..C.<.@.!q.|q,;[
    0080 - a9 af f1 ed ed e2 a3 72-0a 33 a5 e4 30 9a b5 8c   .......r.3..0...
    0090 - 4a ae 0a 58 09 f7 5d 80-6e 1d 93 09 d2 72 1c 4d   J..X..].n....r.M
    00a0 - 4f 13 e5 d8 d8 6d c9 0a-dc 5c f7 17 46 d7 02 dd   O....m...\..F...
    00b0 - f7 8a 2a 41 e3 7d 86 b3-8c e9 bc 6c b5 bd 57 aa   ..*A.}.....l..W.

    Start Time: 1654527241
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
read R BLOCK
Post-Handshake New Session Ticket arrived:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 4BD9F321497BDAB8EB1F8FF4A600A347B88E54174221B09D2757AF551455D4F5
    Resumption PSK: 775E372D1F2D5C8CE978EF769DF2B4EF8CF9CE37B523C6CED3F3D9B12B85DAF2E18F61E51CA5A315DD9A323A2A81A5B9
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 64799 (seconds)
    TLS session ticket:
    0000 - 77 d2 69 6d 5d 7a 49 9d-93 84 1b c4 83 77 88 99]zI......w..
    0010 - 3b 90 4e 0a 72 e0 af aa-78 7f 2c c5 87 b9 b5 65   ;.N.r...x.,....e
    0020 - 8e 98 65 98 dc ab 57 1a-ee 8f 89 58 26 9a 00 ac   ..e...W....X&...
    0030 - f8 09 89 39 8e a5 8c 56-59 21 e5 55 6e 05 2c cc   ...9...VY!.Un.,.
    0040 - 69 99 83 b6 ba fa 96 dd-d7 79 86 ea a5 90 04 3a   i........y.....:
    0050 - a5 a7 21 ba f1 ec d7 66-14 cf 00 88 70 70 2a b8   ..!....f....pp*.
    0060 - 8a 15 dc 01 91 9a ac 14-8f d5 40 50 7e 20 79 3b   ..........@P~ y;
    0070 - d5 7b b7 15 df 23 e8 1b-60 67 ce d0 7d 1d 90 32   .{...#..`g..}..2
    0080 - 98 01 41 7e bd 64 50 2f-89 97 fc 64 cf 6a fe 1a   ..A~.dP/...d.j..
    0090 - 0e b5 ae a1 3a ff 43 a1-0d 1a 1b 9c 16 21 a2 4a   ....:.C......!.J
    00a0 - 38 18 6a 2c 01 31 75 f5-f7 ef 23 8d 28 e2 e3 31   8.j,.1u...#.(..1
    00b0 - 37 10 80 b9 6f 4e 3b 7b-4b ae d8 6c 40 48 8f 25   7...oN;{K..l@H.%

    Start Time: 1654527241
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
read R BLOCK

haproxy -vv:

ubuntu@lb1:~$ haproxy -vv
HAProxy version 2.4.17-1ppa1~focal 2022/05/14 -
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs:
Running on: Linux 5.13.0-1025-aws #27~20.04.1-Ubuntu SMP Thu May 19 15:17:13 UTC 2022 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-ngwd1k/haproxy-2.4.17=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  DEBUG   =


Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
Running on OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[SPOE] spoe
	[CACHE] cache
	[FCGI] fcgi-app
	[COMP] compression
	[TRACE] trace

Does it actually make sense to health check this backend server? Since this is an external service, I’m wondering if there are any benefits at all, or if you are just hammering an external service, maybe even get your IP address banned, all for nothing.

Since you are not using standard health check, but a specific configuration with http-check connect, I think the sni value for health checking needs to be configured there.

So remove check-sni and change the http-check configuration to:

http-check connect ssl port 443 sni

FYI: if you are trying to hide the destination hostname, don’t post the complete public certificate of it, because it will also contain that hostname.

1 Like

Thanks a bunch!! That indeed did the trick. However, I’m now left with an unexpected 401 unauthorised for my health-checks. Good thing that the L6 error is gone, though.

And whether it makes sense - only marginally. I’m using HAProxy largely as a failover ‘router’ between a mix of private-hosted nodes, and then some public providers as emergency fallbacks. In the grand scheme of things I don’t know if it will ever turn out to be worth the trouble, haha.

And yup, I realise that I left in the hostname there. In case it would be beneficial for Tshooting :slight_smile:

Thanks a bunch, this will be the second time you’ve solved my issue

1 Like

Sorry to be bumping this up again, but I’m not able to figure out that 401 unauthorised error message on my health checks. Do you have any idea what the issue could be?

Sending the same request via curl works fine, from the same host machine too. It’s just a POST request with a certain body, including Content-Type application/json, towards a specific path (my paid endpoint).
I’ve been scratching my head for a bit and just don’t see why I would be receiving a 401 unauthorised. Since it’s HTTPS it’s hard for me to look at the traffic. But could it be that as part of the communication the client is supposed to receive a cookie or similar before being allowed to submit a request, and that this is something HAProxy may be struggling with? But then on the other hand, forwarding requests is not a problem…

Is there perhaps a way to log HTTP health check requests and responses, so that I can verify/look at the contents of the POST request and response?

EDIT: solved the issue. The JSON-RPC method I was calling was not allowed on the provider side; I was using a different one when forwarding traffic. Please disregard :slight_smile:

1 Like

FWIW: in this case I would temporarily use a check towards a local HTTP only server, with otherwise the same check configuration and capture the traffic unencrypted this way. That should be enough to understand what happens at HTTP level.

1 Like

This is similar to what I did: set up a new back-end definition, which sent identical health-checks (http) towards a front-end with the ssl backend in question being default. That way I could listen to both parts of the comms locally