Haproxy 2.0.0 got coredump with SSL traffic

Help!
1

We got a coredump with our haproxy latest version 2.0.0, this process only handles just SSL traffic.

“bt full” gdb output as follow:

Program terminated with signal 6, Aborted.
#0 0x00007f44f3cef207 in __GI_raise (sig=sig@entry=6) at …/nptl/sysdeps/unix/sysv/linux/raise.c:55
55 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt full
#0 0x00007f44f3cef207 in __GI_raise (sig=sig@entry=6) at …/nptl/sysdeps/unix/sysv/linux/raise.c:55
resultvar = 0
pid = 6653
selftid = 6658
#1 0x00007f44f3cf08f8 in __GI_abort () at abort.c:90
save_stage = 2
act = {__sigaction_handler = {sa_handler = 0x55555554, sa_sigaction = 0x55555554}, sa_mask = {__val = {1179670597, 22142000, 8, 0 <repeats 13 times>}}, sa_flags = 5418867,
sa_restorer = 0xfffffffffffdfed8}
sigs = {__val = {32, 0 <repeats 15 times>}}
#2 0x000000000053a87c in ha_panic () at src/debug.c:164
No locals.
#3 0x000000000053aa78 in wdt_handler (sig=14, si=, arg=) at src/wdt.c:123
p = 3
thr = 5
#4
No locals.
#5 si_state_bit (state=) at include/proto/stream_interface.h:147
No locals.
#6 si_state_in (mask=, state=) at include/proto/stream_interface.h:154
No locals.
#7 process_stream (t=t@entry=0x7f44ca722b20, context=0x7f449e73fd70, state=) at src/stream.c:2418
s = 0x7f449e73fd70
sess =
rqf_last = 1300234240
rpf_last = 2147483648
rq_prod_last =
rq_cons_last =
rp_cons_last = 8
rp_prod_last = 3
req_ana_back =
req = 0x7f449e73fd80
res = 0x7f449e73fde0
si_f = 0x7f449e740018
si_b = 0x7f449e740070
#8 0x00000000005192b5 in process_runnable_tasks () at src/task.c:412
t = 0x7f44ca722b20
state =
ctx =
process =
lrq =
grq =
t =
max_processed = 6
#9 0x000000000048b407 in run_poll_loop () at src/haproxy.c:2513
next =
wake =
#10 run_thread_poll_loop (data=) at src/haproxy.c:2634
ptaf =
ptif =
ptdf =
ptff =
init_left = 0
init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = ‘\000’ <repeats 39 times>, __align = 0}
init_cond = {__data = {__lock = 0, __futex = 18, __total_seq = 9, __wakeup_seq = 9, __woken_seq = 9, __mutex = 0xae5ea0 <init_mutex.42992>, __nwaiters = 0, __broadcast_seq = 5},
__size = “\000\000\000\000\022\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\240^\256\000\000\000\000\000\000\000\000\000\005\000\000”, __align = 77309411328}
#11 0x00007f44f4295dd5 in start_thread (arg=0x7f44d27b7700) at pthread_create.c:307
__res =
pd = 0x7f44d27b7700
now =
unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139933565810432, -2451856451746657714, 0, 8392704, 0, 139933565810432, 2553793320893811278, 2553727761988323918}, mask_was_saved = 0}},
priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
not_first_call =
pagesize_m1 =
sp =
freesize =
#12 0x00007f44f3db6ead in clone () at …/sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

haproxy -vv output

HA-Proxy version 2.0.0 2019/06/16 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_PCRE2_JIT=1 USE_PTHREAD_PSHARED=1 USE_STATIC_PCRE2=1 USE_LINUX_SPLICE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_TFO=1
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT -PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD +PTHREAD_PSHARED -REGPARM -STATIC_PCRE +STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.1c 28 May 2019
Running on OpenSSL version : OpenSSL 1.1.1c 28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with PCRE2 version : 10.33 2019-04-16
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
: mode=HTX side=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace

Have no idea for what happened.

2

Thanks for the report, this is the thread lockup detection that fires.

@willy

3

Thanks!

Could you please run “p thread_dump_buffer->area” so that we get the debugging output ?

Thanks,
Willy

4

I run the command “p thread_dump_buffer->area” (inside gdb) and get nothing but the output "Cannot access memory at address 0x8.
What can I do to get the debugging output? (sorry for my 0-knowledge of gdb).

BTW, I also dump the full bt of all threads as below

Thread 6 (Thread 0x7f44d37b9700 (LWP 6656)):
#0  0x00000000004447cc in check_req_may_abort (req=req@entry=0x7f44b757ba90, s=<optimized out>) at src/stream.c:999
No locals.
#1  0x000000000044bb87 in sess_update_stream_int (s=0x7f44b757ba80) at src/stream.c:1154
    si = 0x7f44b757bd80
    req = 0x7f44b757ba90
#2  process_stream (t=t@entry=0x7f44c16870c0, context=0x7f44b757ba80, state=<optimized out>) at src/stream.c:2432
    s = 0x7f44b757ba80
    sess = <optimized out>
    rqf_last = <optimized out>
    rpf_last = 2147483648
    rq_prod_last = <optimized out>
    rq_cons_last = <optimized out>
    rp_cons_last = <optimized out>
    rp_prod_last = 3
    req_ana_back = <optimized out>
    req = 0x7f44b757ba90
    res = 0x7f44b757baf0
    si_f = 0x7f44b757bd28
    si_b = 0x7f44b757bd80
#3  0x00000000005192b5 in process_runnable_tasks () at src/task.c:412
    t = 0x7f44c16870c0
    state = <optimized out>
    ctx = <optimized out>
    process = <optimized out>
    lrq = <optimized out>
    grq = <optimized out>
    t = <optimized out>
    max_processed = 7
#4  0x000000000048b407 in run_poll_loop () at src/haproxy.c:2513
    next = <optimized out>
    wake = <optimized out>
#5  run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2634
    ptaf = <optimized out>
    ptif = <optimized out>
    ptdf = <optimized out>
    ptff = <optimized out>
    init_left = 0
    init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
      __size = '\000' <repeats 39 times>, __align = 0}
    init_cond = {__data = {__lock = 0, __futex = 18, __total_seq = 9, __wakeup_seq = 9, __woken_seq = 9, __mutex = 0xae5ea0 <init_mutex.42992>, __nwaiters = 0, __broadcast_seq = 5}, 
      __size = "\000\000\000\000\022\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\240^\256\000\000\000\000\000\000\000\000\000\005\000\000", __align = 77309411328}
#6  0x00007f44f4295dd5 in start_thread (arg=0x7f44d37b9700) at pthread_create.c:307
    __res = <optimized out>
    pd = 0x7f44d37b9700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139933582595840, -2451856451746657714, 0, 8392704, 0, 139933582595840, 2553795520990808654, 2553727761988323918}, mask_was_saved = 0}}, 
      priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
#7  0x00007f44f3db6ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 5 (Thread 0x7f44d47bb700 (LWP 6654)):
#0  si_sync_send (si=si@entry=0x7f44a25734e0) at src/stream_interface.c:896
No locals.
#1  0x000000000044c0fa in process_stream (t=t@entry=0x7f448d73f480, context=0x7f44a25731e0, state=<optimized out>) at src/stream.c:2459
    s = 0x7f44a25731e0
    sess = <optimized out>
    rqf_last = <optimized out>
    rpf_last = 2147483648
    rq_prod_last = <optimized out>
    rq_cons_last = <optimized out>
    rp_cons_last = <optimized out>
    rp_prod_last = 3
    req_ana_back = <optimized out>
    req = 0x7f44a25731f0
    res = 0x7f44a2573250
    si_f = 0x7f44a2573488
    si_b = 0x7f44a25734e0
#2  0x00000000005192b5 in process_runnable_tasks () at src/task.c:412
    t = 0x7f448d73f480
    state = <optimized out>
    ctx = <optimized out>
    process = <optimized out>
    lrq = <optimized out>
    grq = <optimized out>
    t = <optimized out>
    max_processed = 42
#3  0x000000000048b407 in run_poll_loop () at src/haproxy.c:2513
    next = <optimized out>
    wake = <optimized out>
#4  run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2634
    ptaf = <optimized out>
    ptif = <optimized out>
    ptdf = <optimized out>
    ptff = <optimized out>
    init_left = 0
    init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
      __size = '\000' <repeats 39 times>, __align = 0}
    init_cond = {__data = {__lock = 0, __futex = 18, __total_seq = 9, __wakeup_seq = 9, __woken_seq = 9, __mutex = 0xae5ea0 <init_mutex.42992>, __nwaiters = 0, __broadcast_seq = 5}, 
      __size = "\000\000\000\000\022\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\240^\256\000\000\000\000\000\000\000\000\000\005\000\000", __align = 77309411328}
#5  0x00007f44f4295dd5 in start_thread (arg=0x7f44d47bb700) at pthread_create.c:307
    __res = <optimized out>
    pd = 0x7f44d47bb700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139933599381248, -2451856451746657714, 0, 8392704, 0, 139933599381248, 2553797721087806030, 2553727761988323918}, mask_was_saved = 0}}, 
      priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
#6  0x00007f44f3db6ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 4 (Thread 0x7f44d3fba700 (LWP 6655)):
#0  process_stream (t=t@entry=0x7f44c54b7800, context=0x7f44a947de90, state=<optimized out>) at src/stream.c:2489
    s = 0x7f44a947de90
    sess = <optimized out>
    rqf_last = <optimized out>
    rpf_last = 2147483648
    rq_prod_last = <optimized out>
    rq_cons_last = <optimized out>
    rp_cons_last = <optimized out>
    rp_prod_last = 3
    req_ana_back = <optimized out>
    req = 0x7f44a947dea0
    res = 0x7f44a947df00
    si_f = 0x7f44a947e138
    si_b = 0x7f44a947e190
#1  0x00000000005192b5 in process_runnable_tasks () at src/task.c:412
    t = 0x7f44c54b7800
    state = <optimized out>
    ctx = <optimized out>
    process = <optimized out>
    lrq = <optimized out>
    grq = <optimized out>
    t = <optimized out>
    max_processed = 18
#2  0x000000000048b407 in run_poll_loop () at src/haproxy.c:2513
    next = <optimized out>
    wake = <optimized out>
#3  run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2634
    ptaf = <optimized out>
    ptif = <optimized out>
    ptdf = <optimized out>
    ptff = <optimized out>
    init_left = 0
    init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
      __size = '\000' <repeats 39 times>, __align = 0}
    init_cond = {__data = {__lock = 0, __futex = 18, __total_seq = 9, __wakeup_seq = 9, __woken_seq = 9, __mutex = 0xae5ea0 <init_mutex.42992>, __nwaiters = 0, __broadcast_seq = 5}, 
      __size = "\000\000\000\000\022\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\240^\256\000\000\000\000\000\000\000\000\000\005\000\000", __align = 77309411328}
#4  0x00007f44f4295dd5 in start_thread (arg=0x7f44d3fba700) at pthread_create.c:307
    __res = <optimized out>
    pd = 0x7f44d3fba700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139933590988544, -2451856451746657714, 0, 8392704, 0, 139933590988544, 2553794420942309966, 2553727761988323918}, mask_was_saved = 0}}, 
      priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
#5  0x00007f44f3db6ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 3 (Thread 0x7f44f4d0da00 (LWP 6653)):
#0  0x000000000044b96c in process_stream (t=t@entry=0x2d46600, context=0xe7a0400, state=<optimized out>) at src/stream.c:2331
    s = 0xe7a0400
    sess = <optimized out>
    rqf_last = 1300234240
    rpf_last = 2147483648
    rq_prod_last = <optimized out>
    rq_cons_last = <optimized out>
    rp_cons_last = 8
    rp_prod_last = 3
    req_ana_back = <optimized out>
    req = 0xe7a0410
    res = 0xe7a0470
    si_f = 0xe7a06a8
    si_b = 0xe7a0700
#1  0x00000000005192b5 in process_runnable_tasks () at src/task.c:412
    t = 0x2d46600
    state = <optimized out>
    ctx = <optimized out>
    process = <optimized out>
    lrq = <optimized out>
    grq = <optimized out>
    t = <optimized out>
    max_processed = 17
#2  0x000000000048b407 in run_poll_loop () at src/haproxy.c:2513
    next = <optimized out>
    wake = <optimized out>
#3  run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:2634
    ptaf = <optimized out>
    ptif = <optimized out>
    ptdf = <optimized out>
    ptff = <optimized out>
    init_left = 0
    init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
      __size = '\000' <repeats 39 times>, __align = 0}
    init_cond = {__data = {__lock = 0, __futex = 18, __total_seq = 9, __wakeup_seq = 9, __woken_seq = 9, __mutex = 0xae5ea0 <init_mutex.42992>, __nwaiters = 0, __broadcast_seq = 5}, 
      __size = "\000\000\000\000\022\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\240^\256\000\000\000\000\000\000\000\000\000\005\000\000", __align = 77309411328}
#4  0x0000000000407b4b in main (argc=<optimized out>, argv=0x7ffe056ce1f8) at src/haproxy.c:3308
    blocked_sig = {__val = {18446744067199990583, 18446744073709551615 <repeats 15 times>}}
    old_sig = {__val = {0, 139934125255732, 1, 20243296, 11139984, 140728989442608, 140728989442552, 4773466, 6, 6, 8, 4228587, 11, 20243296, 255, 0}}
    i = 6
    err = <optimized out>
    retry = <optimized out>
    limit = {rlim_cur = 400103, rlim_max = 400103}
    errmsg = "\000\003{\000\000\000\000\000H\340l\005\376\177\000\000\030\000\000\000\000\000\000\000 \000\000\000\060\000\000\000q \223\372\001\200\377\377\003\000\000\000\376\177", '\000' <repeats 18 times>, "\060\000\000\000[\000\000\000\220\337l\005\376\177", '\000' <repeats 18 times>, "n\000\000"
    pidfd = <optimized out>

Thread 2 (Thread 0x7f44d2fb8700 (LWP 6657)):
#0  si_sync_send (si=si@entry=0x7f44bd486a20) at src/stream_interface.c:898
No locals.
#1  0x000000000044c0fa in process_stream (t=t@entry=0x7f44bdf00c40, context=0x7f44bd486720, state=<optimized out>) at src/stream.c:2459
    s = 0x7f44bd486720
    sess = <optimized out>
    rqf_last = <optimized out>
    rpf_last = 2147483648
    rq_prod_last = <optimized out>
    rq_cons_last = <optimized out>
    rp_cons_last = <optimized out>
    rp_prod_last = 3
    req_ana_back = <optimized out>
    req = 0x7f44bd486730
    res = 0x7f44bd486790
    si_f = 0x7f44bd4869c8
    si_b = 0x7f44bd486a20
#2  0x00000000005192b5 in process_runnable_tasks () at src/task.c:412
    t = 0x7f44bdf00c40
    state = <optimized out>
    ctx = <optimized out>
    process = <optimized out>
    lrq = <optimized out>
    grq = <optimized out>
    t = <optimized out>
    max_processed = 16
#3  0x000000000048b407 in run_poll_loop () at src/haproxy.c:2513
    next = <optimized out>
    wake = <optimized out>
#4  run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2634
    ptaf = <optimized out>
    ptif = <optimized out>
    ptdf = <optimized out>
    ptff = <optimized out>
    init_left = 0
    init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
      __size = '\000' <repeats 39 times>, __align = 0}
    init_cond = {__data = {__lock = 0, __futex = 18, __total_seq = 9, __wakeup_seq = 9, __woken_seq = 9, __mutex = 0xae5ea0 <init_mutex.42992>, __nwaiters = 0, __broadcast_seq = 5}, 
      __size = "\000\000\000\000\022\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\240^\256\000\000\000\000\000\000\000\000\000\005\000\000", __align = 77309411328}
#5  0x00007f44f4295dd5 in start_thread (arg=0x7f44d2fb8700) at pthread_create.c:307
    __res = <optimized out>
    pd = 0x7f44d2fb8700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139933574203136, -2451856451746657714, 0, 8392704, 0, 139933574203136, 2553792220845312590, 2553727761988323918}, mask_was_saved = 0}}, 
      priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
#6  0x00007f44f3db6ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.

Thread 1 (Thread 0x7f44d27b7700 (LWP 6658)):
#0  0x00007f44f3cef207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
    resultvar = 0
    pid = 6653
    selftid = 6658
#1  0x00007f44f3cf08f8 in __GI_abort () at abort.c:90
    save_stage = 2
    act = {__sigaction_handler = {sa_handler = 0x55555554, sa_sigaction = 0x55555554}, sa_mask = {__val = {1179670597, 22142000, 8, 0 <repeats 13 times>}}, sa_flags = 5418867, 
      sa_restorer = 0xfffffffffffdfed8}
    sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x000000000053a87c in ha_panic () at src/debug.c:164
No locals.
#3  0x000000000053aa78 in wdt_handler (sig=14, si=<optimized out>, arg=<optimized out>) at src/wdt.c:123
    p = 3
    thr = 5
#4  <signal handler called>
No locals.
#5  si_state_bit (state=<optimized out>) at include/proto/stream_interface.h:147
No locals.
#6  si_state_in (mask=<optimized out>, state=<optimized out>) at include/proto/stream_interface.h:154
No locals.
#7  process_stream (t=t@entry=0x7f44ca722b20, context=0x7f449e73fd70, state=<optimized out>) at src/stream.c:2418
    s = 0x7f449e73fd70
    sess = <optimized out>
    rqf_last = 1300234240
    rpf_last = 2147483648
    rq_prod_last = <optimized out>
    rq_cons_last = <optimized out>
    rp_cons_last = 8
    rp_prod_last = 3
    req_ana_back = <optimized out>
    req = 0x7f449e73fd80
    res = 0x7f449e73fde0
    si_f = 0x7f449e740018
    si_b = 0x7f449e740070
#8  0x00000000005192b5 in process_runnable_tasks () at src/task.c:412
    t = 0x7f44ca722b20
    state = <optimized out>
    ctx = <optimized out>
    process = <optimized out>
    lrq = <optimized out>
    grq = <optimized out>
    t = <optimized out>
    max_processed = 6
#9  0x000000000048b407 in run_poll_loop () at src/haproxy.c:2513
    next = <optimized out>
    wake = <optimized out>
#10 run_thread_poll_loop (data=<optimized out>) at src/haproxy.c:2634
    ptaf = <optimized out>
    ptif = <optimized out>
    ptdf = <optimized out>
    ptff = <optimized out>
    init_left = 0
    init_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}}, 
      __size = '\000' <repeats 39 times>, __align = 0}
    init_cond = {__data = {__lock = 0, __futex = 18, __total_seq = 9, __wakeup_seq = 9, __woken_seq = 9, __mutex = 0xae5ea0 <init_mutex.42992>, __nwaiters = 0, __broadcast_seq = 5}, 
      __size = "\000\000\000\000\022\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\t\000\000\000\000\000\000\000\240^\256\000\000\000\000\000\000\000\000\000\005\000\000", __align = 77309411328}
#11 0x00007f44f4295dd5 in start_thread (arg=0x7f44d27b7700) at pthread_create.c:307
    __res = <optimized out>
    pd = 0x7f44d27b7700
    now = <optimized out>
    unwind_buf = {cancel_jmp_buf = {{jmp_buf = {139933565810432, -2451856451746657714, 0, 8392704, 0, 139933565810432, 2553793320893811278, 2553727761988323918}, mask_was_saved = 0}}, 
      priv = {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype = 0}}}
    not_first_call = <optimized out>
    pagesize_m1 = <optimized out>
    sp = <optimized out>
    freesize = <optimized out>
#12 0x00007f44f3db6ead in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
No locals.
5

I run the command “p thread_dump_buffer->area” (inside gdb) and get nothing
but the output "Cannot access memory at address 0x8.
What can I do to get the debugging output? (sorry for my 0-knowledge of gdb).

Thank you, it is just as if no dump was emitted except that the abort()
was called after emitting the dump. Ah no, I understand now, the pointer
is reset by the last who dumps it. Maybe that’s something we could change
in the future.

From your dump I think you can get it by looking at thread 1 by typing these
two commands :

thread 1
p trash.area

Thank you for your help, it’s quite valuable!
Willy

6

@willy:

(gdb) thread 1
[Switching to thread 1 (Thread 0x7f44d27b7700 (LWP 6658))]
#0  0x00007f44f3cef207 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:55
55	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) p trash.area
$9 = 0x7f44c8000940 "Thread 6 is about to kill the process.\n  Thread 1 : act=1 glob=1 wq=1 rq=1 tl=1 tlsz=20 rqsz=8\n", ' ' <repeats 13 times>, "stuck=0 fdcache=1 prof=0 harmless=0 wantrdv=0\n", ' ' <repeats 13 times>, "cpu_ns: poll=194390270986760 now="...
7

Thanks. gdb is truncated the string to the first 200 chars, so I’m afraid
you’ll have to do it again after removing the limit with this commant :

set print elements 0

It will be large (your terminal will scroll a bit), so please be careful to
copy it all.

Thanks again
Willy

8

Hope this help.

$2 = 0x7f44c8000940 "Thread 6 is about to kill the process.\n  Thread 1 : act=1 glob=1 wq=1 rq=1 tl=1 tlsz=20 rqsz=8\n", ' ' <repeats 13 times>, "stuck=0 fdcache=1 prof=0 harmless=0 wantrdv=0\n", ' ' <repeats 13 times>, "cpu_ns: poll=194390270986760 now=194391346425077 diff=1075438317\n", ' ' <repeats 13 times>, "curr_task=0x2d46600 (task) calls=4 last=0\n", ' ' <repeats 15 times>, "fct=0x44b3d0 (process_stream) ctx=0xe7a0400\n", ' ' <repeats 13 times>, "strm=0xe7a0400 src=115.73.9.102 fe=HTTPS be=JAVA dst=app20\n", ' ' <repeats 13 times>, "rqf=4d800000 rqa=8000 rpf=80000000 rpa=0 sif=EST,250028 sib=TAR,20013a\n", ' ' <repeats 13 times>, "af=(nil),0 csf=0x36154b0,4000\n", ' ' <repeats 13 times>, "ab=(nil),0 csb=0xca0b670,100\n", ' ' <repeats 13 times>, "cof=0x2d080b0,201306:H1(0x4595420)/SSL(0x4bb3cf0)/tcpv4(45099)\n", ' ' <repeats 13 times>, "cob=0x567d700,202000:H1(0xa35d400)/NONE((nil))/NONE(-35738115)\n  Thread 2 : act=1 glob=1 wq=1 rq=0 tl=1 tlsz=21 rqsz=0\n", ' ' <repeats 13 times>, "stuck=0 fdcache=1 prof=0 harmless=0 wantrdv=0\n", ' ' <repeats 13 times>, "cpu_ns: poll=228545505262979 now=228546831248292 diff=1325985313\n", ' ' <repeats 13 times>, "curr_task=0x7f448d73f480 (task) calls=4 last=0\n", ' ' <repeats 15 times>, "fct=0x44b3d0 (process_stream) ctx=0x7f44a25731e0\n", ' ' <repeats 13 times>, "strm=0x7f44a25731e0 src=14.184.96.122 fe=HTTPS be=JAVA dst=app20\n", ' ' <repeats 13 times>, "rqf=4d800000 rqa=8000 rpf=80000000 rpa=0 sif=EST,250028 sib=TAR,20013a\n", ' ' <repeats 13 times>, "af=(nil),0 csf=0x7f44cc602130,4000\n", ' ' <repeats 13 times>, "ab=(nil),0 csb=0x7f44cceec710,100\n", ' ' <repeats 13 times>, "cof=0x7f44b0188970,201306:H1(0x7f44a20ca5d0)/SSL(0x7f448c6829d0)/tcpv4(10917)\n", ' ' <repeats 13 times>, "cob=0x7f448db61a00,202000:H1(0x7f44cf263db0)/NONE((nil))/NONE(-35738115)\n >Thread 3 : act=1 glob=1 wq=1 rq=1 tl=1 tlsz=25 rqsz=3\n", ' ' <repeats 13 times>, "stuck=1 fdcache=1 prof=0 harmless=0 wantrdv=0\n", ' ' <repeats 13 times>, "cpu_ns: poll=197149898884324 now=197151364668941 diff=1465784617\n", ' ' <repeats 13 times>, "curr_task=0x7f44c54b7800 (task) calls=4 last=0\n", ' ' <repeats 15 times>, "fct=0x44b3d0 (process_stream) ctx=0x7f44a947de90\n", ' ' <repeats 13 times>, "strm=0x7f44a947de90 src=42.114.39.243 fe=HTTPS be=JAVA dst=app20\n", ' ' <repeats 13 times>, "rqf=4d800000 rqa=8000 rpf=80000000 rpa=0 sif=EST,250028 sib=TAR,20013a\n", ' ' <repeats 13 times>, "af=(nil),0 csf=0x7f44c6521770,4000\n", ' ' <repeats 13 times>, "ab=(nil),0 csb=0x7f44c5138f90,100\n", ' ' <repeats 13 times>, "cof=0x7f44c504b750,201306:H1(0x7f44bba2e980)/SSL(0x7f44c4684060)/tcpv4(43532)\n", ' ' <repeats 13 times>, "cob=0x7f44c58f58e0,202000:H1(0x7f44a86647a0)/NONE((nil))/NONE(-35738115)\n  Thread 4 : act=1 glob=1 wq=1 rq=1 tl=1 tlsz=14 rqsz=12\n", ' ' <repeats 13 times>, "stuck=0 fdcache=1 prof=1 harmless=0 wantrdv=0\n", ' ' <repeats 13 times>, "cpu_ns: poll=192280062388377 now=192280759436960 diff=697048583\n", ' ' <repeats 13 times>, "curr_task=0x7f44c16870c0 (task) calls=4 last=696647863 ns ago\n", ' ' <repeats 15 times>, "fct=0x44b3d0 (process_stream) ctx=0x7f44b757ba80\n", ' ' <repeats 13 times>, "strm=0x7f44b757ba80 src=112.215.239.9 fe=HTTPS be=JAVA dst=app20\n", ' ' <repeats 13 times>, "rqf=4d800000 rqa=8000 rpf=80000000 rpa=0 sif=EST,250028 sib=TAR,20013a\n", ' ' <repeats 13 times>, "af=(nil),0 csf=0x7f44b6a10780,4000\n", ' ' <repeats 13 times>, "ab=(nil),0 csb=0x7f44b5e287b0,100\n", ' ' <repeats 13 times>, "cof=0x7f44c0c15430,201306:H1(0x7f44b56f7f80)/SSL(0x7f44c0020420)/tcpv4(4446)\n", ' ' <repeats 13 times>, "cob=0x7f4499ce1890,202000:H1(0x7f44c2d07e40)/NONE((nil))/NONE(-35738115)\n  Thread 5 : act=1 glob=0 wq=1 rq=1 tl=1 tlsz=30 rqsz=9\n", ' ' <repeats 13 times>, "stuck=0 fdcache=1 prof=1 harmless=0 wantrdv=0\n", ' ' <repeats 13 times>, "cpu_ns: poll=194343321249255 now=194344013430807 diff=692181552\n", ' ' <repeats 13 times>, "curr_task=0x7f44bdf00c40 (task) calls=4 last=691740774 ns ago\n", ' ' <repeats 15 times>, "fct=0x44b3d0 (process_stream) ctx=0x7f44bd486720\n", ' ' <repeats 13 times>, "strm=0x7f44bd486720 src=27.67.7.32 fe=HTTPS be=JAVA dst=app20\n", ' ' <repeats 13 times>, "rqf=4d800000 rqa=8000 rpf=80000000 rpa=0 sif=EST,250028 sib=TAR,20013a\n", ' ' <repeats 13 times>, "af=(nil),0 csf=0x7f4497171e10,4000\n", ' ' <repeats 13 times>, "ab=(nil),0 csb=0x7f44bd64fa60,100\n", ' ' <repeats 13 times>, "cof=0x7f44bef9f860,201306:H1(0x7f44bcd3ff50)/SSL(0x7f4494d902a0)/tcpv4(529)\n", ' ' <repeats 13 times>, "cob=0x7f4494e248f0,202000:H1(0x7f44bd453450)/NONE((nil))/NONE(-35738115)\n*>Thread 6 : act=1 glob=1 wq=1 rq=0 tl=1 tlsz=1 rqsz=0\n", ' ' <repeats 13 times>, "stuck=1 fdcache=1 prof=0 harmless=0 wantrdv=0\n", ' ' <repeats 13 times>, "cpu_ns: poll=228601091574070 now=228603844579726 diff=2753005656\n", ' ' <repeats 13 times>, "curr_task=0x7f44ca722b20 (task) calls=4 last=0\n", ' ' <repeats 15 times>, "fct=0x44b3d0 (process_stream) ctx=0x7f449e73fd70\n", ' ' <repeats 13 times>, "strm=0x7f449e73fd70 src=113.169.155.111 fe=HTTPS be=JAVA dst=app20\n", ' ' <repeats 13 times>, "rqf=4d800000 rqa=8000 rpf=80000000 rpa=0 sif=EST,250028 sib=TAR,20013a\n", ' ' <repeats 13 times>, "af=(nil),0 csf=0x7f44a6380920,4000\n", ' ' <repeats 13 times>, "ab=(nil),0 csb=0x7f4490dd1300,100\n", ' ' <repeats 13 times>, "cof=0x7f4491c18a60,201306:H1(0x7f4490fd5770)/SSL(0x7f44c82699e0)/tcpv4(43367)\n", ' ' <repeats 13 times>, "cob=0x7f44c9860380,202000:H1(0x7f44ca1e9130)/NONE((nil))/NONE(-35738115)\n"
9

I hope so, thank you. I needed to see the various connection flags and so, now they are present so that should help is figure what happened.