We are seeing randomly haproxy crashes due to segfault
“haproxy[22735]: segfault at 7f6705601068 ip 00007f670a9b1246 sp 00007f670888a7c0 error 4 in libc-2.23.so[7f670a978000+1c0000]”
Here are details of haproxy
HA-Proxy version 1.9.12 2019/10/24 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_ZLIB=1 USE_POLL=default USE_CPU_AFFINITY=1 USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2u 20 Dec 2019
Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016 (VERSIONS DIFFER!)
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.0
Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.43 2019-02-23
Running on PCRE version : 8.43 2019-02-23
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Available polling systems :
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 2 (2 usable), will use poll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
<default> : mode=HTX side=FE|BE
<default> : mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Here is haproxy config
global
nbproc 1
nbthread 8
cpu-map auto:all 0-
stats socket /var/run/haproxy.sock mode 0777
log /dev/log local1
chroot /usr/local/haproxy
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-bind-ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1
ssl-engine cloudhsm
tune.ssl.default-dh-param 2048
maxconn 50000
Note : we are using cloudHSM Dynamic engine for OPENSSL
From haproxy[22735]: segfault at 7f6705601068 ip 00007f670a9b1246 sp 00007f670888a7c0 error 4 in libc-2.23.so[7f670a978000+1c0000]
I subtracted 00007f670a9b1246 - 7f670a978000 = 39246
Then did the following
addr2line -e /lib/x86_64-linux-gnu/libc-2.23.so -fCi 39246
bsearch
??:?