I observe SSL Handshake failures in my Haproxy logs. Sometimes this occurs when the web application is trying to save data or is receiving a larger post and it causes the web application to throw an error also.
Environment notes: We are running RHEL 7 - Using Haproxy to interface two active webbackends servers and one backup webbackend servers We are using Cherrypy for the web application.
Here is an example error in the cherrypy logs:
server02 [4.xxx] [INF] (web/services/web/site-packages/cherrypy/_cplogging.py) 10.111.233.123 - - "GET /~api/entity/tMatch%22%3Atrue%7D%5D HTTP/1.1" 200 41 "https://www.example.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
File "/web/services/web/site-packages/OpenSSL/SSL.py", line 1545, in _raise_ssl_error
raise ZeroReturnError()
File "/web/services/web/site-packages/OpenSSL/SSL.py", line 1734, in recv_into
self._raise_ssl_error(self._ssl, result)
File "/web/services/web/site-packages/OpenSSL/SSL.py", line 1545, in _raise_ssl_error
raise ZeroReturnError()
OpenSSL.SSL.ZeroReturnError
And here is the corresponding entry in haproxy:
Apr 26 13:27:38 localhost haproxy[2487]: 192.200.5.43:50600 [26/Apr/2021:13:27:38.136] www.example.com:80/2: SSL handshake failure
Here is the Haproxy build info:
HA-Proxy version 2.0.14 2020/04/02 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_PCRE=1 USE_THREAD=1 USE_LIBCRYPT=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=1).
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.5
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
<default> : mode=HTX side=FE|BE mux=H1
<default> : mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Here is my Haproxy Configuration file:
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2 info
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 2048
ssl-server-verify required
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
# turn on stats unix socket
stats socket /var/run/haproxy/info.sock mode 660 level user user report group zabbix
stats timeout 2m # Wait up to 2 minutes for input
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 5m
timeout server 5m
timeout http-keep-alive 10s
timeout check 10s
maxconn 300
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend example.com:80
bind *:80
bind *:443 ssl crt /etc/ssl/certs/web.pem
log global
option httplog
http-request set-header Forwarded for=%[src]
mode http
option forwardfor
default_backend web_servers
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend web_servers
balance roundrobin
mode http
option httpchk GET /#
cookie SERVERUSED insert indirect nocache
redirect scheme https code 301 if !{ ssl_fc }
redirect prefix https://www.example.com code 301 if { hdr(host) -i old.example.com }
#List all the web servers below
timeout queue 10s
server backendweb1 10.111.233.122:8443 check ssl verify required ca-file /etc/ssl/certs/web.pem cookie web1
server backendweb2 10.111.233.123:8443 check ssl verify required ca-file /etc/ssl/certs/web.pem cookie web2
server backendweb3 10.170.218.123:8443 check ssl verify required ca-file /etc/ssl/certs/web.pem cookie web3 backup
listen stats # Define a listen section called "stats"
bind :::8404 v4v6 ssl crt /etc/ssl/certs/web.pem # Listen on localhost 8404
mode http
stats enable # Enable stats page
stats realm Haproxy\ Statistics # Title text for popup window
stats uri /stats # Stats URI
stats refresh 10s
stats auth xxxxx:xxxxxxxxxxx #Authentication credential
stats admin if LOCALHOST