Segfault with tcp-check connect with threads

Help!
1

Hello,

I faced a segfault in HAProxy when using tcp-check connect configuration setup. Google showed a lot of people having the same issue and saying the bug has been fixed in version 1.9.6 or later versions (https://www.mail-archive.com/haproxy@formilux.org/msg33141.html).

However I check with the last available version at https://haproxy.debian.net/ and still have the issue.

I have made several tests and it turns out that the more frontends / backends you have the more frequently the coredumps occurs. With 4 frontends I had this about 4 times a day, with over 100 frontend it happens every 2 or 3 minutes event if only one of them is using the tcp-check connect.

In dmsg I got logs like that:

[Mon Jul  1 10:55:38 2019] haproxy[29988]: segfault at 10 ip 000055d2a1731e85 sp 00007f01f37e2390 error 4 in haproxy[55d2a1660000+1e6000]
[Mon Jul  1 10:55:58 2019] haproxy[30027]: segfault at 10 ip 0000560f4a9e7e85 sp 00007f94fd508390 error 4 in haproxy[560f4a916000+1e6000]
[Mon Jul  1 10:56:11 2019] haproxy[30114]: segfault at 10 ip 000056534d544e85 sp 00007fa56ac1a390 error 4 in haproxy[56534d473000+1e6000]
[Mon Jul  1 10:56:16 2019] haproxy[30150]: segfault at 6 ip 00005565aa6b4203 sp 00007f5b8ee442f0 error 4
[Mon Jul  1 10:56:16 2019] haproxy[30151]: segfault at 6 ip 00005565aa6b4203 sp 00007f5b8e6432f0 error 4
[Mon Jul  1 10:56:16 2019]  in haproxy[5565aa5e4000+1e6000]
[Mon Jul  1 10:56:16 2019]  in haproxy[5565aa5e4000+1e6000]
[Mon Jul  1 10:57:12 2019] traps: haproxy[30168] general protection ip:55f8194a1ec8 sp:7f9fb7606390 error:0
[Mon Jul  1 10:57:12 2019]  in haproxy[55f8193d0000+1e6000]
[Mon Jul  1 10:57:17 2019] haproxy[30353]: segfault at 6 ip 000055a43df1b203 sp 00007f4fa68832f0 error 4 in haproxy[55a43de4b000+1e6000]
[Mon Jul  1 10:57:29 2019] haproxy[30378]: segfault at 6 ip 000056028e238203 sp 00007f88d36882f0 error 4 in haproxy[56028e168000+1e6000]

How to reproduce on Debian Stretch:

  • Install HAproxy 1.9.x
  • Add EXTRAOPTS="-f /etc/haproxy/sites-enabled/" in /etc/default/haproxy
  • Use the default haproxy.cfg provided by the package and add nbthread 4 in the global section (I didn’t manage to trigger the bug without adding threads).
  • generate extra configuration using the following script:
#!/bin/bash

cd /etc/haproxy
mkdir -p sites-enabled

rm sites-enabled/*

cat <<EOF > sites-enabled/00-stats.cfg
listen stats
    bind *:80
    stats uri /
    stats admin if TRUE
EOF

for i in $(seq 50); do
cat <<EOF > sites-enabled/front-$i.cfg
listen frontend$i
        mode tcp
        option tcplog

        bind 127.0.0.$i:12345

        option tcp-check
        tcp-check connect port 25
        tcp-check expect rstring "^220 "
        tcp-check send "quit\r\n"

        server 127.0.0.1 127.0.0.1:25 check inter 500ms
EOF

done
  • Make sure you have a smtp server installed on localhost (exim, postfix, opensmtpd would do the job).
  • restart HAProxy and wait.

I can provide further details if required.

My HAProxy installed version on a Debian Stretch.

HA-Proxy version 1.9.8-1~bpo9+1 2019/05/18 - https://haproxy.org/
Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-1.9.8=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0j  20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.0f  25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.22 2016-07-29
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE
              h2 : mode=HTTP       side=FE
       <default> : mode=HTX        side=FE|BE
       <default> : mode=TCP|HTTP   side=FE|BE

Available filters :
	[SPOE] spoe
	[COMP] compression
	[CACHE] cache
	[TRACE] trace
2

Since this is not the place to report bug, see https://github.com/haproxy/haproxy/issues/148.

This topic can be closed or archived by an admin if required.

Sorry for the inconvenience.