Wdt_handler crash

Help!
1

In our RHEL/AlmaLinux 9 haproxy instances I experienced ocasional crashes. Distro ships haporxy 2.8.14, I know it’s very outdated… It seems related to crash in 3.2.4: wdt_handler · Issue #3101 · haproxy/haproxy · GitHub , but I don’t know if 2.8 branch was affected and backported. We indeed use DNS resolution. Our instances have low traffic.

Jan 20 09:12:46 haproxy-02 haproxy[1638]: Thread 2 is about to kill the process.
Jan 20 09:18:22 haproxy-02 haproxy[1638]: >Thread 1 : id=0x7fb89d3d21c0 act=1 glob=0 wq=1 rq=0 tl=0 tlsz=0 rqsz=1
Jan 20 09:18:22 haproxy-02 haproxy[1638]:      1/1    stuck=1 prof=0 harmless=0 isolated=0
Jan 20 09:18:22 haproxy-02 haproxy[1638]:             cpu_ns: poll=79581090602 now=81871962226 diff=2290871624
Jan 20 09:18:22 haproxy-02 haproxy[1638]:             curr_task=0x563bb0784e20 (tasklet) calls=1
Jan 20 09:18:22 haproxy-02 haproxy[1638]:               fct=0x563bafa1d510(ssl_sock_io_cb) ctx=0x563bb07a0890
Jan 20 09:18:22 haproxy-02 haproxy[1638]:             call trace(11):
Jan 20 11:03:14 haproxy-02 haproxy[1638]: *>Thread 2 : id=0x7fb89c705640 act=1 glob=0 wq=1 rq=0 tl=1 tlsz=6 rqsz=6
Jan 20 11:03:14 haproxy-02 haproxy[1638]:      1/2    stuck=1 prof=0 harmless=0 isolated=0
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             cpu_ns: poll=78230449313 now=99587029635 diff=21356580322
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             curr_task=0x563bb0712d40 (task) calls=17318 last=0
Jan 20 11:03:14 haproxy-02 haproxy[1638]:               fct=0x563bafb6d270(process_chk) ctx=0x563bb05a7188
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             call trace(31):
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x563bafb94449 [eb bf 0f 1f 44 00 00 f3]: ha_thread_dump_fill+0xa9/0xab
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x563bafb944ee [48 89 c3 48 85 c0 74 d3]: main+0x18828e
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x563bafc0a340 [e8 cb 08 e0 ff 66 66 2e]: wdt_handler+0x1c0/0x1c5
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89c83fc30 [48 c7 c0 0f 00 00 00 0f]: libc:+0x3fc30
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cedccc0 [f3 0f 1e fa f2 ff 25 ed]: libcrypto:+0xdccc0
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cffc161 [48 85 c0 74 c8 5d 41 5c]: libcrypto:CRYPTO_malloc+0x61/0x99
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cffc1fe [49 89 c0 48 85 c0 74 10]: libcrypto:CRYPTO_zalloc+0xe/0x2c
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cf9ed10 [49 89 c4 48 85 c0 0f 84]: libcrypto:OSSL_DECODER_CTX_new_for_pkey+0x190/0x99a
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89d0a2756 [48 89 c5 58 5a 48 85 ed]: libcrypto:+0x2a2756
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cef5d0b [48 83 c4 20 e9 0e fc ff]: libcrypto:+0xf5d0b
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cef7978 [48 83 c4 30 41 89 c5 85]: libcrypto:+0xf7978
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cef5e89 [48 83 c4 20 85 c0 0f 84]: libcrypto:+0xf5e89
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cef7978 [48 83 c4 30 41 89 c5 85]: libcrypto:+0xf7978
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cef5e89 [48 83 c4 20 85 c0 0f 84]: libcrypto:+0xf5e89
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cef7978 [48 83 c4 30 41 89 c5 85]: libcrypto:+0xf7978
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cef7c22 [48 83 c4 20 48 8b 54 24]: libcrypto:+0xf7c22
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cef5e89 [48 83 c4 20 85 c0 0f 84]: libcrypto:+0xf5e89
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89cef65d3 [48 83 c4 30 85 c0 7e 5d]: libcrypto:ASN1_item_d2i_ex+0x63/0xdc
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x7fb89d47de4c [49 89 c6 48 85 c0 0f 84]: libssl:d2i_SSL_SESSION_ex+0x4c/0x90b
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x563bafa1d07c [49 89 c5 48 85 c0 74 1c]: main+0x10e1c
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x563bafaa9cd7 [4c 8b 5c 24 10 85 c0 0f]: tcpcheck_eval_connect+0x717/0x14c9
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x563bafaaf5d0 [49 8b 54 24 28 f6 42 18]: tcpcheck_main+0x3a0/0xc6c
Jan 20 11:03:14 haproxy-02 haproxy[1638]:             | 0x563bafb6cbef [e9 cd fb ff ff 0f 1f 40]: process_chk_conn+0xc4f/0x12cf
Jan 20 11:03:14 haproxy-02 haproxy[1407]: [NOTICE]   (1407) : haproxy version is 2.8.14-c23fe91

Just wanted to know if this bug is listed under http://www.haproxy.org/bugs/bugs-2.8.14.html , I do not see a directly related candidate there. I wanted to know if this is solved in 2.8 branch.

In order to have the latest version of 2.8.x, is it feasible to execute haproxy via podman and systemd quadlet?

Thanks in advance,

Gabriel

2

All watchdog related crashes will be in wdt_handler because it is the watchdog handler; it is unlikely that you are actually facing the DNS related issue you found in 3.2.

2.8 and 3.2 are very different in this regards.

In fact the stack trace points to SSL issues, not DNS issues.

Yes, I do suggest you run it in a docker container if you can, I would suggest haproxytech/haproxy-alpine:3.0 or haproxytech/haproxy-alpine:3.2

1 Like
3

Ok, didn’t know about the internals. I had no problems on these hosts with RH/Alma 9 and haproxy 2.4 before 9.7. I will try to port my install to podman/docker instances.