HAProxy crashing with thread stuck.
I’ve seen this on several occasions on our test servers running AWS Linux 2023. Older systems running older HAProxy with thousands of hours in production never exhibited this behavior. Very very worried about putting this into our production server mix. Load in test way lower than production obviously.
Version:
HAProxy version 2.8.3-86e043a 2023/09/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.3.html
Running on: Linux 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64
What is logged:
... haproxy[1285490]: Thread 8 is about to kill the process.
... haproxy[1285490]: >Thread 1 : id=0x7f56f4f4a240 act=1 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
... haproxy[1285490]: 1/1 stuck=1 prof=0 harmless=0 isolated=0
... haproxy[1285490]: cpu_ns: poll=39484613817 now=40638584750 diff=1153970933
... haproxy[1285490]: curr_task=0x5613af91ebc0 (task) calls=1098966 last=0
... haproxy[1285490]: fct=0x5613ad8ebef0(process_resolvers) ctx=0x5613af8b5f80
... haproxy[1285490]: call trace(16):
... haproxy[1285490]: | 0x5613ad9443f2 [48 85 dd 75 e2 64 48 8b]: debug_handler+0x52/0x75
... haproxy[1285490]: | 0x7f56f4c54dd0 [48 c7 c0 0f 00 00 00 0f]: libc:+0x54dd0
... haproxy[1285490]: | 0x5613ad9b9099 [66 0f 6f 05 af 85 04 00]: wdt_handler+0x129/0x1c8
... haproxy[1285490]: | 0x7f56f4c54dd0 [48 c7 c0 0f 00 00 00 0f]: libc:+0x54dd0
... haproxy[1285490]: | 0x5613ad7c5b2d [3d c7 00 00 00 77 f4 0f]: main+0x596d
... haproxy[1285490]: | 0x5613ad8ec42d [48 89 e8 f0 49 0f c1 04]: process_resolvers+0x53d/0x74d
... haproxy[1285490]: Thread 2 : id=0x7f56f4bff640 act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
... haproxy[1285490]: 1/2 stuck=0 prof=0 harmless=1 isolated=0
... haproxy[1285490]: cpu_ns: poll=78193631873 now=78193661174 diff=29301
... haproxy[1285490]: curr_task=0
... haproxy[1285490]: Thread 3 : id=0x7f56effff640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
... haproxy[1285490]: 1/3 stuck=0 prof=0 harmless=1 isolated=0
... haproxy[1285490]: cpu_ns: poll=231502370886 now=231502451916 diff=81030
... haproxy[1285490]: curr_task=0
... haproxy[1285490]: Thread 4 : id=0x7f56ef5fe640 act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
... haproxy[1285490]: 1/4 stuck=0 prof=0 harmless=1 isolated=0
... haproxy[1285490]: cpu_ns: poll=106352851925 now=106352868865 diff=16940
... haproxy[1285490]: curr_task=0
... haproxy[1285490]: Thread 5 : id=0x7f56eebfd640 act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
... haproxy[1285490]: 1/5 stuck=0 prof=0 harmless=1 isolated=0
... haproxy[1285490]: cpu_ns: poll=110246089634 now=110246101105 diff=11471
... haproxy[1285490]: curr_task=0
... haproxy[1285490]: Thread 6 : id=0x7f56ee1fc640 act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0
... haproxy[1285490]: 1/6 stuck=0 prof=0 harmless=1 isolated=0
... haproxy[1285490]: cpu_ns: poll=50076340244 now=50076347174 diff=6930
... haproxy[1285490]: curr_task=0
... haproxy[1285490]: Thread 7 : id=0x7f56ed7fb640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
... haproxy[1285490]: 1/7 stuck=0 prof=0 harmless=1 isolated=0
... haproxy[1285490]: cpu_ns: poll=198023777549 now=198023796889 diff=19340
... haproxy[1285490]: curr_task=0
... haproxy[1285490]: *>Thread 8 : id=0x7f56ecdfa640 act=0 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=0
... haproxy[1285490]: 1/8 stuck=1 prof=0 harmless=0 isolated=0
... haproxy[1285490]: cpu_ns: poll=161501432730 now=184937485174 diff=23436052444
... haproxy[1285490]: curr_task=0
... haproxy[1285490]: call trace(14):
... haproxy[1285490]: | 0x5613ad943f89 [eb bf 0f 1f 44 00 00 f3]: ha_thread_dump+0x99/0x9b
... haproxy[1285490]: | 0x5613ad9440e2 [64 48 8b 73 08 64 48 8b]: ha_panic+0x152/0x3a9
... haproxy[1285490]: | 0x7f56f4c54dd0 [48 c7 c0 0f 00 00 00 0f]: libc:+0x54dd0
... haproxy[1285490]: | 0x5613ad7c5b2d [3d c7 00 00 00 77 f4 0f]: main+0x596d
... haproxy[1285490]: | 0x5613ad8ea8fd [48 89 e8 f0 49 0f c1 04]: main+0x12a73d
... haproxy[1285490]: | 0x5613ad9b8b84 [48 8b 05 c5 41 0e 00 8b]: dgram_fd_handler+0x34/0x67
... haproxy[1285490]: | 0x5613ad987f3c [48 8b 35 0d 4e 11 00 48]: fd_update_events+0x34c/0x43d
... haproxy[1285490]: | 0x5613ad7c716c [49 39 dc 74 2f 64 48 8b]: main+0x6fac
... haproxy[1285488]: [NOTICE] (1285488) : haproxy version is 2.8.3-86e043a
... haproxy[1285488]: [NOTICE] (1285488) : path to executable is /usr/sbin/haproxy
... haproxy[1285488]: [ALERT] (1285488) : Current worker (1285490) exited with code 134 (Aborted)
... haproxy[1285488]: [ALERT] (1285488) : exit-on-failure: killing every processes with SIGTERM
... haproxy[1285488]: [WARNING] (1285488) : All workers exited. Exiting... (134)
... systemd[1]: haproxy.service: Main process exited, code=exited, status=134/n/a
... systemd[1]: haproxy.service: Failed with result 'exit-code'.
... systemd[1]: haproxy.service: Consumed 16min 40.572s CPU time.