HAP Load Balancer forward to HAP Reverse Proxy: SSL handshake failure

Sefer · September 19, 2023, 9:42am

Hello community!

I am trying to setup HAP as a Load Balancer to our backends which are running HAP as a reverse proxy (I try to use one tool instead of two, i.e. nginx). I am running HAP 2.8 in docker (default image, haproxy -vv below) on both servers. I know I could use mode tcp for tls forwarding on the load balancer but I need to use cookies for sticky sessions.

So far the setup is running and working, but my backend instance is getting spammed with the following error: loadbalancerIP:port all_frontend/2: SSL handshake failure.

I cannot find the root cause for it neither do I understand it. My guess is this is related to the health check because this is happening “in idle” without anyone trying to request the website.

I also noticed the backend only logs the IP from the load balancer for each site request. But this is another topic I guess.

server-full.pem file consists of (in this order): key, cert, ca2/intermediate, root
chain.pem file consists of: ca2/intermediate, root

Load Balancer

Summary

global
  log stdout format raw local0
  log stdout format raw local1 notice
  stats socket /var/lib/haproxy/admin.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
  stats timeout 30s
  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
  ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
  ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-dh-param-file /usr/local/etc/haproxy/dhparam2048.pem

defaults
  log global
  mode http
  option httpslog
  option dontlognull
  option forwardfor
  timeout connect 120s
  timeout client 120s
  timeout server 120s
  timeout http-request 120s
  errorfile 400 /usr/local/etc/haproxy/errors/400.http
  errorfile 403 /usr/local/etc/haproxy/errors/403.http
  errorfile 408 /usr/local/etc/haproxy/errors/408.http
  errorfile 500 /usr/local/etc/haproxy/errors/500.http
  errorfile 502 /usr/local/etc/haproxy/errors/502.http
  errorfile 503 /usr/local/etc/haproxy/errors/503.http
  errorfile 504 /usr/local/etc/haproxy/errors/504.http

userlist AuthUsers
  user admin password HASH

listen admin
  bind *:8090 ssl crt /etc/ssl/certs/server-full.pem
  mode http
  stats enable
  stats uri /
  stats refresh 5s
  stats realm HAProxy\ Statistics
  stats hide-version
  http-request auth unless { http_auth(AuthUsers) }

frontend all_frontend
  description Front-end for all traffic
  bind *:80
  bind *:443 ssl crt /etc/ssl/certs/server-full.pem
  mode http
  http-request set-header X-Real-IP %ci
  http-request add-header X-Forwarded-Host %[req.hdr(host)]
  http-response add-header X-XSS-Protection "1; mode=block"
  http-response del-header X-Powered-By
  http-response set-header Via "HTTP/2.0 loadbalancer.net"
  redirect scheme https code 301 unless { ssl_fc }
  default_backend jira_nodes

backend jira_nodes
  balance leastconn
  server node1 node1.net:443 check ssl verify required ca-file /etc/ssl/certs/chain.pem cookie node1
  server node2 node2.net:443 check ssl verify required ca-file /etc/ssl/certs/chain.pem cookie node2
  cookie JSESSIONID prefix nocache

Reverse Proxy

Summary

global
  log stdout format raw local0
  log stdout format raw local1 notice
  stats socket /var/lib/haproxy/admin.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
  stats timeout 30s
  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private
  ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
  ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305
  ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-dh-param-file /usr/local/etc/haproxy/dhparam2048.pem

defaults
  log global
  mode http
  option httpslog
  option dontlognull
  option forwardfor
  timeout connect 120s
  timeout client 120s
  timeout server 120s
  timeout http-request 120s
  errorfile 400 /usr/local/etc/haproxy/errors/400.http
  errorfile 403 /usr/local/etc/haproxy/errors/403.http
  errorfile 408 /usr/local/etc/haproxy/errors/408.http
  errorfile 500 /usr/local/etc/haproxy/errors/500.http
  errorfile 502 /usr/local/etc/haproxy/errors/502.http
  errorfile 503 /usr/local/etc/haproxy/errors/503.http
  errorfile 504 /usr/local/etc/haproxy/errors/504.http
  default-server  init-addr last,libc,none

frontend all_frontend
  description Front-end for all traffic
  bind *:80
  bind *:443 ssl crt /etc/ssl/certs/server-full.pem
  mode http
  acl source_loadbalancer hdr(host) -i "loadbalancer.net"
  acl source_directaccess hdr(host) -i "nodeX.net"
  http-request set-header X-Real-IP %ci
  http-request add-header X-Forwarded-Host %[req.hdr(host)]
  redirect scheme https code 301 unless { ssl_fc }
  use_backend app_balanced if source_loadbalancer
  use_backend app_directaccess if source_directaccess

backend app_balanced
  server app_balanced host.docker.internal:8080 check
backend app_directaccess
  server app_directaccess host.docker.internal:8082 check

HAProxy Check

Summary

haproxy@fd1bf6560586:~$ haproxy -vv
HAProxy version 2.8.3-86e043a 2023/09/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.3.html
Running on: Linux 5.4.0-144-generic #161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=2).
Built with OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
Running on OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.36 2020-12-04
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 10.2.1 20210110

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Thanks for your time. I appreciate your help.

Sefer · September 19, 2023, 12:19pm

Reading comprehension. Apparently check-ssl in the server directive is a thing!

EDIT: On the load balancer set send-proxy-v2-ssl and on the reverse proxy set accept-proxy for the bind *:443 line to retain the client ip

Sefer · September 21, 2023, 4:07pm

I accidentally removed the check command and thought it was fixed. It is not. If added again I receive SSL handshake failure once again.

For a test I disabled no-tlsv13 at bind-options and server-options and I do not receive the handshake failure anymore. But again this message only appears on HAProxy health checks and only every so often.

So it is still unclear to me why tls 1.3 seems to be a problem here because every other request, ssl check or tool check works without throwing this error.

Is it worth an issue on github?

To be clear. This is not to be confused with all other SSL handshake failure so far which report this issue on the load balancer side!
→ HAP Load Balancer → HAP Reverse Proxy (Error on Health Check) → Application

Topic		Replies	Views
Https frontend -> http backend : SSL handshake failure Help!	3	366	May 16, 2024
SSL Handshake failures when enabling Proxy-protocol Help!	11	57	September 11, 2024
Troubleshooting SSL Handshake Failure (backend) Help!	4	1022	December 11, 2022
Not working in TCP mode Help!	10	199	July 23, 2024
SSL handhake failure - pulling my hair out! Help!	9	2991	January 3, 2019

HAP Load Balancer forward to HAP Reverse Proxy: SSL handshake failure

Related Topics