Layer6 invalid response, info: "SSL handshake failure"

Help!
1

Somehow all the other posts don’t specifically solve my issue so…
Hi all,

I have two backend servers that are running on Port 443 SSL via IIS using the CCS (Centralized Certification Server) module.

Access to those two backend servers works fine:
image

However the health check on HaProxy fails with a Layer 6 issue.
[WARNING] (5477) : Server cso-cs-frontends/otcs01 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 1ms. 1 active and 0 backup servers left.

I already tried to use a verifyhost however it did not solve the issue. Ideas? The certificate is a wildcard one from sectigo since dev machines.

backend config:

backend cso-cs-frontends
    balance roundrobin
    mode http
    cookie SERVERID insert indirect nocache
    server otcs01 msecmcsf01.otxlab.net:443 ssl verifyhost msecmcsf01.otxlab.net verify required ca-file /data/cso-fs-ssl/sectigo-cert-chain.crt cookie msecmcsf01 check
    server otcs02 msecmcsf02.otxlab.net:443 ssl verifyhost msecmcsf02.otxlab.net verify required ca-file /data/cso-fs-ssl/sectigo-cert-chain.crt cookie msecmcsf02 check

Thanks!

2

Did a few additional troubleshooting steps but it doesn’t make any sense:

openssl s_client -connect msecmcsf01.otxlab.net:443 says:

SSL handshake has read 5236 bytes and written 407 bytes
Verification: OK

Which is what I expect since a connect from a clients browser works fine
Anyone else has any idea what could be wrong here?

3

Hi Aebian,

could you solve this issue? I found my self in the same situation. Thanks!

4

There are plenty of reasons a SSL handshake can fail.

There can be a certificate validation error, mismatch in TLS versions/ciphers or other technical issues, or it can be based on missing SNI values etc.

Best pratices are:

  • use supported and uptodate OS and OS provided libraries (do not compile your own ssl lib)
  • do not manually configure SSL ciphers but leave it eather at defaults or at Mozilla suggested configs; also try older ciphers as suggested by mozilla if your backend server is old
  • make sure you send SNI accordingly for both health checks (check-sni) and pass through traffic (sni)

If this doesn’t help, analyzing the issue requires:

  • the full output of haproxy -vv
  • the full configuration
  • the full output of a successfull curl -vv https://... call to the backend server