Haproxy Segfault: segfault at 7f6705601068 ip 00007f670a9b1246 sp 00007f670888a7c0 error 4 in libc-2.23.so[7f670a978000+1c0000]

We are seeing randomly haproxy crashes due to segfault

“haproxy[22735]: segfault at 7f6705601068 ip 00007f670a9b1246 sp 00007f670888a7c0 error 4 in libc-2.23.so[7f670a978000+1c0000]”

Here are details of haproxy

HA-Proxy version 1.9.12 2019/10/24 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_ZLIB=1 USE_POLL=default USE_CPU_AFFINITY=1 USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2u  20 Dec 2019
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016 (VERSIONS DIFFER!)
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.0
Built with transparent proxy support using: IP_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.43 2019-02-23
Running on PCRE version : 8.43 2019-02-23
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 2 (2 usable), will use poll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE
              h2 : mode=HTTP       side=FE
       <default> : mode=HTX        side=FE|BE
       <default> : mode=TCP|HTTP   side=FE|BE

Available filters :
	[SPOE] spoe
	[COMP] compression
	[CACHE] cache
	[TRACE] trace

Here is haproxy config


global
  nbproc 1
  nbthread 8
  cpu-map auto:all 0-

  stats socket /var/run/haproxy.sock mode 0777
  log  /dev/log  local1
  chroot   /usr/local/haproxy
  ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-default-bind-ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1

  ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-default-server-ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1


  ssl-engine cloudhsm


  tune.ssl.default-dh-param 2048

  maxconn 50000

Note : we are using cloudHSM Dynamic engine for OPENSSL

From haproxy[22735]: segfault at 7f6705601068 ip 00007f670a9b1246 sp 00007f670888a7c0 error 4 in libc-2.23.so[7f670a978000+1c0000]

I subtracted 00007f670a9b1246 - 7f670a978000 = 39246

Then did the following

 addr2line -e /lib/x86_64-linux-gnu/libc-2.23.so -fCi 39246
bsearch
??:?

The 1.9 release branch is unsupported and unmaintained. Before 1.9 became unmaintained, 195 bugs were fixed in the 1.9 release tree:

http://www.haproxy.org/bugs/bugs-1.9.12.html

You need to switch to a supported release first of all.

I would recommend using 2.2.8 at this point.

We did upgrade to 2.28 and we got same error

haproxy[22138]: segfault at 7ff76c574478 ip 00007ff7718d4246 sp 00007ff76ffaefa0 error 4 in libc-2.23.so[7ff77189b000+1c0000]

Using same configuration

Ok, please file a bug at github:

Does it crash without the cloudhsm engine? A full backtrace with gdb from a coredump will be necessary to start the investigation, I assume.

It does not crash without cloudhsm engine, only when cloudhsm engine is enabled it crashes.

I will try to get a coredump. Haproxy is running inside a container, so it been bit of challenge to get the coredump

1 Like

Hi Here is the full bracktrace with gdb from a core dump

The haproxy in this case crashed with

" haproxy[17847]: segfault at 7fc7c9da4340 ip 00007fc7cef86129 sp 00007fc7caf9b170 error 4 in libc-2.28.so[7fc7cef6f000+148000]"

gdb /usr/local/haproxy/haproxy -c core

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/haproxy/haproxy -D -f /usr/local/haproxy/config/haproxy.conf -q -p /'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  msort_with_tmp (p=p@entry=0x7fc7caf9b210, b=b@entry=0x7fc7c9d9c010, n=n@entry=65535) at msort.c:85
85	msort.c: No such file or directory.
[Current thread is 1 (Thread 0x7fc7caf9e700 (LWP 989))]
(gdb) bt full
#0  msort_with_tmp (p=p@entry=0x7fc7caf9b210, b=b@entry=0x7fc7c9d9c010, n=n@entry=65535) at msort.c:85
       b1 = 0x7fc7c9da4340 <error: Cannot access memory at address 0x7fc7c9da4340>
       b2 = 0x7fc7c9ddc008 <error: Cannot access memory at address 0x7fc7c9ddc008>
       n1 = 28569
       n2 = 32768
       tmp = 0x7fc7c44577e0 "0}4\270\307\177"
       s = 8
       cmp = 0x7fc7cf3d9d40 <pmeth_cmp>
       arg = 0x0
#1  0x00007fc7cef861f2 in msort_with_tmp (n=65535, b=0x7fc7c9d9c010, p=0x7fc7caf9b210) at msort.c:297
       b2 = <optimized out>
       n2 = <optimized out>
       tmp = <optimized out>
       s = <optimized out>
       b1 = <optimized out>
       n1 = <optimized out>
       cmp = <optimized out>
       arg = <optimized out>
       b1 = <optimized out>
       b2 = <optimized out>
       n1 = <optimized out>
       n2 = <optimized out>
       tmp = <optimized out>
       s = <optimized out>
       cmp = <optimized out>
       arg = <optimized out>
       tmpl = <optimized out>
       bl = <optimized out>
#2  __GI___qsort_r (b=<optimized out>, n=<optimized out>, s=8, cmp=0x7fc7cf3d9d40 <pmeth_cmp>, arg=0x0) at msort.c:297
       size = <optimized out>
       tmp = 0x7fc7c444f4b0 "\200\273\002\270\307\177"
       p = {s = 8, var = 1, cmp = 0x7fc7cf3d9d40 <pmeth_cmp>, arg = 0x0, t = 0x7fc7c444f4b0 "\200\273\002\270\307\177"}
#3  0x00007fc7cf3c3eaa in sk_sort () from /usr/local/ssl/lib/libcrypto.so.1.0.0
No symbol table info available.
#4  0x00007fc7cf3d9fe8 in EVP_PKEY_meth_add0 () from /usr/local/ssl/lib/libcrypto.so.1.0.0
No symbol table info available.
#5  0x00007fc7cece7f19 in cavium_pkey_meths (e=<optimized out>, pmeth=0x7fc7caf9b2d8, nids=<optimized out>, nid=<optimized out>) at hw_cavium.c:2488
       cavium_rsa_pkey_meth = 0x7fc7c442e4f0
       e = <optimized out>
       nids = <optimized out>
       nid = 6
       pmeth = 0x7fc7caf9b2d8
#6  0x00007fc7cf3b7893 in ENGINE_get_pkey_meth () from /usr/local/ssl/lib/libcrypto.so.1.0.0
No symbol table info available.
#7  0x00007fc7cf3da0cc in EVP_PKEY_CTX_new () from /usr/local/ssl/lib/libcrypto.so.1.0.0
No symbol table info available.
--Type <RET> for more, q to quit, c to continue without paging--
#8  0x00007fc7cf3d3c29 in EVP_SignFinal () from /usr/local/ssl/lib/libcrypto.so.1.0.0
No symbol table info available.
#9  0x00007fc7cf717d6c in ssl3_send_server_key_exchange () from /usr/local/ssl/lib/libssl.so.1.0.0
No symbol table info available.
#10 0x00007fc7cf71c0fa in ssl3_accept () from /usr/local/ssl/lib/libssl.so.1.0.0
No symbol table info available.
#11 0x000055ae9880f4a1 in ssl_sock_handshake (conn=conn@entry=0x7fc7c402a3d0, flag=flag@entry=33554432) at src/ssl_sock.c:5446
       ret = <optimized out>
#12 0x000055ae98917234 in conn_fd_handler (fd=<optimized out>) at src/connection.c:88
       conn = 0x7fc7c402a3d0
       flags = 41948934
       io_available = 0
#13 0x000055ae98925d67 in fdlist_process_cached_events (fdlist=<optimized out>) at src/fd.c:443
       fd = 68
       old_fd = 68
       e = <optimized out>
       locked = <optimized out>
       fd = <optimized out>
       old_fd = <optimized out>
       e = <optimized out>
       locked = <optimized out>
       __pl_r = <optimized out>
       __pl_l = <optimized out>
       ret = <optimized out>
#14 fd_process_cached_events () at src/fd.c:461
No locals.
#15 0x000055ae988a3e3e in run_poll_loop () at src/haproxy.c:2710
       next = <optimized out>
       exp = <optimized out>
       next = <optimized out>
       exp = <optimized out>
#16 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2739
       ptif = <optimized out>
       ptdf = <optimized out>
       start_lock = 0
#17 0x00007fc7cf977fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
       ret = <optimized out>
       pd = <optimized out>
       now = <optimized out>
       unwind_buf = {cancel_jmp_buf = {{jmp_buf = {140496080594688, -6395308500951096315, 140731371799390, 140731371799391, 140496080594688, 0, 6390846732468888581,
               6390840845923525637}, mask_was_saved = 0}}, priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
       not_first_call = <optimized out>
#18 0x00007fc7cf0464cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
No locals.

Haproxy crashed with
[7044844.539664] haproxy[8782]: segfault at 7f9fc8b408e8 ip 00007f9fce20fd40 sp 00007f9fca5d1ba8 error 4 in libcrypto.so.1.0.0[7f9fce0d8000+231000]

Here is full bt from core dump

(gdb) bt
#0  0x00007f9fce20fd40 in pmeth_cmp () from /usr/local/ssl/lib/libcrypto.so.1.0.0
#1  0x00007f9fcddbc125 in msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b40808, n=n@entry=64) at msort.c:83
#2  0x00007f9fcddbbe81 in msort_with_tmp (n=64, b=0x7f9fc8b40808, p=0x7f9fca5d2120) at msort.c:53
#3  msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b40808, n=n@entry=128) at msort.c:53
#4  0x00007f9fcddbbe81 in msort_with_tmp (n=128, b=0x7f9fc8b40808, p=0x7f9fca5d2120) at msort.c:53
#5  msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b40808, n=n@entry=256) at msort.c:53
#6  0x00007f9fcddbbe97 in msort_with_tmp (n=256, b=0x7f9fc8b40808, p=0x7f9fca5d2120) at msort.c:54
#7  msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b40008, n=n@entry=512) at msort.c:54
#8  0x00007f9fcddbbe81 in msort_with_tmp (n=512, b=0x7f9fc8b40008, p=0x7f9fca5d2120) at msort.c:53
#9  msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b40008, n=n@entry=1024) at msort.c:53
#10 0x00007f9fcddbbe81 in msort_with_tmp (n=1024, b=0x7f9fc8b40008, p=0x7f9fca5d2120) at msort.c:53
#11 msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b40008, n=n@entry=2048) at msort.c:53
#12 0x00007f9fcddbbe81 in msort_with_tmp (n=2048, b=0x7f9fc8b40008, p=0x7f9fca5d2120) at msort.c:53
#13 msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b40008, n=n@entry=4096) at msort.c:53
#14 0x00007f9fcddbbe81 in msort_with_tmp (n=4096, b=0x7f9fc8b40008, p=0x7f9fca5d2120) at msort.c:53
#15 msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b40008, n=n@entry=8192) at msort.c:53
#16 0x00007f9fcddbbe97 in msort_with_tmp (n=8192, b=0x7f9fc8b40008, p=0x7f9fca5d2120) at msort.c:54
#17 msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b30008, n=n@entry=16384) at msort.c:54
#18 0x00007f9fcddbbe81 in msort_with_tmp (n=16384, b=0x7f9fc8b30008, p=0x7f9fca5d2120) at msort.c:53
#19 msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b30008, n=n@entry=32768) at msort.c:53
#20 0x00007f9fcddbbe81 in msort_with_tmp (n=32768, b=0x7f9fc8b30008, p=0x7f9fca5d2120) at msort.c:53
#21 msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8b30008, n=n@entry=65536) at msort.c:53
#22 0x00007f9fcddbbe97 in msort_with_tmp (n=65536, b=0x7f9fc8b30008, p=0x7f9fca5d2120) at msort.c:54
#23 msort_with_tmp (p=p@entry=0x7f9fca5d2120, b=b@entry=0x7f9fc8ab0010, n=n@entry=131071) at msort.c:54
#24 0x00007f9fcddbc1f2 in msort_with_tmp (n=131071, b=0x7f9fc8ab0010, p=0x7f9fca5d2120) at msort.c:297
#25 __GI___qsort_r (b=<optimized out>, n=<optimized out>, s=8, cmp=0x7f9fce20fd40 <pmeth_cmp>, arg=0x0) at msort.c:297
#26 0x00007f9fce1f9eaa in sk_sort () from /usr/local/ssl/lib/libcrypto.so.1.0.0
#27 0x00007f9fce1f9f82 in sk_find () from /usr/local/ssl/lib/libcrypto.so.1.0.0
#28 0x00007f9fce20fd8a in EVP_PKEY_meth_find () from /usr/local/ssl/lib/libcrypto.so.1.0.0
#29 0x00007f9fcdb1ded1 in cavium_pkey_meths (e=<optimized out>, pmeth=0x7f9fca5d22d8, nids=<optimized out>, nid=<optimized out>)
    at hw_cavium.c:2481
#30 0x00007f9fce1ed893 in ENGINE_get_pkey_meth () from /usr/local/ssl/lib/libcrypto.so.1.0.0
#31 0x00007f9fce2100cc in EVP_PKEY_CTX_new () from /usr/local/ssl/lib/libcrypto.so.1.0.0
#32 0x00007f9fce209c29 in EVP_SignFinal () from /usr/local/ssl/lib/libcrypto.so.1.0.0
#33 0x00007f9fce54dd6c in ssl3_send_server_key_exchange () from /usr/local/ssl/lib/libssl.so.1.0.0
#34 0x00007f9fce5520fa in ssl3_accept () from /usr/local/ssl/lib/libssl.so.1.0.0
#35 0x000055b2fa6894a1 in ssl_sock_handshake (conn=conn@entry=0x7f9fbc076c60, flag=flag@entry=33554432) at src/ssl_sock.c:5446
#36 0x000055b2fa791234 in conn_fd_handler (fd=<optimized out>) at src/connection.c:88
#37 0x000055b2fa79fd67 in fdlist_process_cached_events (fdlist=<optimized out>) at src/fd.c:443
#38 fd_process_cached_events () at src/fd.c:461
#39 0x000055b2fa71de3e in run_poll_loop () at src/haproxy.c:2710
#40 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2739
#41 0x00007f9fce7adfa3 in start_thread (arg=<optimized out>) at pthread_create.c:486
#42 0x00007f9fcde7c4cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Please file a bug over at Github.

1 Like