Hi,
We’ve configured HAProxy to operate in multi-threaded mode and we’ve seen HAProxy crashing occasionally. Please find the details below:
haproxy -vv
HA-Proxy version 1.8.18 2019/02/06
Copyright 2000-2019 Willy Tarreau willy@haproxy.org
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O0 -g -fno-strict-aliasing
OPTIONS = USE_SLZ=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with PCRE version : 8.41 2017-07-05
Running on PCRE version : 8.41 2017-07-05
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
$ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
Snippet of HAProxy config file:
global
log /var/lib/tmp/log.sock local0 info alert
stats socket /var/lib/tmp/run/haproxy.sock mode 600 level admin expose-fd listeners
stats timeout 2m
stats maxconn 2
nbproc 1
nbthread 8
maxconn 20000
daemon
pidfile /var/lib/tmp/run/haproxy.pid
ssl-dh-param-file /var/lib/keys/dhparam.pem
tune.ssl.lifetime 86400
tune.ssl.cachesize 600000
ssl-default-bind-options no-tls-tickets ssl-min-ver TLSv1.0
Core dump analysis using GDB:
gdb /opt//external/bin/haproxy core.11974
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7_4.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-redhat-linux-gnu”.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/…
Reading symbols from /opt//external/bin/haproxy…done.
[New LWP 11982]
[New LWP 11977]
[New LWP 11976]
[New LWP 11974]
[New LWP 11978]
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib64/libthread_db.so.1”.
Core was generated by `/opt//external/bin/haproxy -f /var/lib//runtime/config/ha’.
Program terminated with signal 6, Aborted.
#0 0x00007f559e1f3277 in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install [redacted]-0.0.0-1141.x86_64
(gdb) bt
#0 0x00007f559e1f3277 in raise () from /lib64/libc.so.6
#1 0x00007f559e1f4968 in abort () from /lib64/libc.so.6
#2 0x00007f559e235d37 in __libc_message () from /lib64/libc.so.6
#3 0x00007f559e23e499 in _int_free () from /lib64/libc.so.6
#4 0x00000000004d047d in deinit_log_buffers () at src/log.c:1387
#5 0x00000000004d0336 in deinit_log_buffers_per_thread () at src/log.c:1365
#6 0x00000000004cbd08 in run_thread_poll_loop (data=0x1f4bf2c) at src/haproxy.c:2485
#7 0x00007f559ec64e25 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f559e2bbbad in clone () from /lib64/libc.so.6
Related code in log.c:
1380 /* Deinitialize log buffers used for syslog messages */
1381 void deinit_log_buffers()
1382 {
1383 free(logheader);
1384 free(logheader_rfc5424);
1385 free(logline);
1386 free(logline_rfc5424);
1387 free(startup_logs);
1388 logheader = NULL;
1389 logheader_rfc5424 = NULL;
1390 logline = NULL;
1391 logline_rfc5424 = NULL;
1392 startup_logs = NULL;
1393 }
Any idea on what could possibly be wrong?
Apologies for the bad formatting. I’m new to this forum and couldn’t figure out how to format the lines properly.
Thank you.
Regards,
Kiran