Hi,
I’m using HAProxy 1.8.4 on FreeBSD 11.1. Unfortunately it’s unusable, because it consumes 100% CPU a few seconds after startup. I was not able to track this down to a specific option in the configuration file.
Technically speaking is HAProxy still up and running, but in this state the connections behave very strange. For example MySQL connections may or may not work, or they abort or get stuck. Besides that I’m using this HAProxy box for mail and web traffic too. FWIW, this configuration works flawlessly with 1.7.10 (which I was forced to downgrade to).
I’ve tried nbproc 1, nbproc 8, nbthread 1, nbthread 10, no matter what I choose, it always ends up with 100% CPU usage.
The gdb output does not look very useful, or can you spot something useful?
root@host:~ # gdb --pid 58391
GNU gdb (GDB) 8.0.1 [GDB v8.0.1 for FreeBSD]
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd11.1".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 58391
Reading symbols from /usr/local/sbin/haproxy...done.
Reading symbols from /lib/libcrypt.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/libz.so.6...(no debugging symbols found)...done.
Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
Reading symbols from /usr/lib/libssl.so.8...(no debugging symbols found)...done.
Reading symbols from /lib/libcrypto.so.8...(no debugging symbols found)...done.
Reading symbols from /usr/local/lib/liblua-5.3.so...(no debugging symbols found)...done.
Reading symbols from /lib/libm.so.5...(no debugging symbols found)...done.
Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols found)...done.
[Switching to LWP 100644 of process 58391]
0x000000867b1b8fea in _kevent () from /lib/libc.so.7
(gdb) bt full
#0 0x000000867b1b8fea in _kevent () from /lib/libc.so.7
No symbol table info available.
#1 0x000000867a25fcd2 in ?? () from /lib/libthr.so.3
No symbol table info available.
#2 0x000000000041d85c in _do_poll (p=0x8ab7d0 <cur_poller>, exp=45596037) at src/ev_kqueue.c:115
status = 2074008608
count = 45596037
fd = 200
delta_ms = 5
timeout = {tv_sec = 0, tv_nsec = 5000000}
updt_idx = 0
en = 0
eo = 0
changes = 0
#3 0x000000000050f7be in run_poll_loop () at src/haproxy.c:2424
next = 45596037
exp = 45596037
#4 0x000000000050cf10 in run_thread_poll_loop (data=0x867bbfe6c0) at src/haproxy.c:2460
start_lock = 0
ptif = 0x89e2f8 <per_thread_init_list>
ptdf = 0x206
#5 0x0000000000509a90 in main (argc=6, argv=0x793eabda1b38) at src/haproxy.c:3049
tids = 0x867bbfe6c0
threads = 0x867bbfe6e0
i = 1
err = 0
retry = 200
limit = {rlim_cur = 500096, rlim_max = 500096}
errmsg = "\000\033\332\253>y\000\000p\033\332\253>y\000\000\070\033\332\253>y\000\000\006\000\000\000\000\000\000\000\217c}\310;\"sA\240O\212\000\000\000\000\000\060\033\332\253>y\000\000p\033\332\253>y\000\000\070\033\332\253>y\000\000\006\000\000\000\000\000\000\000\320\032\332\253>y\000\000\322\005 {\206\000\000\000\001\000\000"
pidfd = 40
(gdb) cont
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x000000867b1b8fea in _kevent () from /lib/libc.so.7
(gdb) bt full
#0 0x000000867b1b8fea in _kevent () from /lib/libc.so.7
No symbol table info available.
#1 0x000000867a25fcd2 in ?? () from /lib/libthr.so.3
No symbol table info available.
#2 0x000000000041d85c in _do_poll (p=0x8ab7d0 <cur_poller>, exp=45597649) at src/ev_kqueue.c:115
status = 2074006208
count = 45597649
fd = 200
delta_ms = 134
timeout = {tv_sec = 0, tv_nsec = 134000000}
updt_idx = 0
en = 0
eo = 0
changes = 0
#3 0x000000000050f7be in run_poll_loop () at src/haproxy.c:2424
next = 45597649
exp = 45597649
#4 0x000000000050cf10 in run_thread_poll_loop (data=0x867bbfe6c0) at src/haproxy.c:2460
start_lock = 0
ptif = 0x89e2f8 <per_thread_init_list>
ptdf = 0x206
#5 0x0000000000509a90 in main (argc=6, argv=0x793eabda1b38) at src/haproxy.c:3049
tids = 0x867bbfe6c0
threads = 0x867bbfe6e0
i = 1
err = 0
retry = 200
limit = {rlim_cur = 500096, rlim_max = 500096}
errmsg = "\000\033\332\253>y\000\000p\033\332\253>y\000\000\070\033\332\253>y\000\000\006\000\000\000\000\000\000\000\217c}\310;\"sA\240O\212\000\000\000\000\000\060\033\332\253>y\000\000p\033\332\253>y\000\000\070\033\332\253>y\000\000\006\000\000\000\000\000\000\000\320\032\332\253>y\000\000\322\005 {\206\000\000\000\001\000\000"
pidfd = 40
(gdb) cont
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x000000867b1b8fea in _kevent () from /lib/libc.so.7
(gdb) bt full
#0 0x000000867b1b8fea in _kevent () from /lib/libc.so.7
No symbol table info available.
#1 0x000000867a25fcd2 in ?? () from /lib/libthr.so.3
No symbol table info available.
#2 0x000000000041d85c in _do_poll (p=0x8ab7d0 <cur_poller>, exp=45605353) at src/ev_kqueue.c:115
status = 2074007168
count = 45605353
fd = 200
delta_ms = 32
timeout = {tv_sec = 0, tv_nsec = 32000000}
updt_idx = 0
en = 0
eo = 0
changes = 0
#3 0x000000000050f7be in run_poll_loop () at src/haproxy.c:2424
next = 45605353
exp = 45605353
#4 0x000000000050cf10 in run_thread_poll_loop (data=0x867bbfe6c0) at src/haproxy.c:2460
start_lock = 0
ptif = 0x89e2f8 <per_thread_init_list>
ptdf = 0x206
#5 0x0000000000509a90 in main (argc=6, argv=0x793eabda1b38) at src/haproxy.c:3049
tids = 0x867bbfe6c0
threads = 0x867bbfe6e0
i = 1
err = 0
retry = 200
limit = {rlim_cur = 500096, rlim_max = 500096}
errmsg = "\000\033\332\253>y\000\000p\033\332\253>y\000\000\070\033\332\253>y\000\000\006\000\000\000\000\000\000\000\217c}\310;\"sA\240O\212\000\000\000\000\000\060\033\332\253>y\000\000p\033\332\253>y\000\000\070\033\332\253>y\000\000\006\000\000\000\000\000\000\000\320\032\332\253>y\000\000\322\005 {\206\000\000\000\001\000\000"
pidfd = 40
root@host:~ # haproxy -vv
HA-Proxy version 1.8.4-1deb90d 2018/02/08
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = freebsd
CPU = generic
CC = cc
CFLAGS = -pipe -g -fstack-protector -fno-strict-aliasing -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-null-dereference -Wno-unused-label -DFREEBSD_PORTS
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_ACCEPT4=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 200000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.40 2017-01-11
Running on PCRE version : 8.40 2017-01-11
PCRE library supports JIT : yes
Built with multi-threading support.
Encrypted password support via crypt(3): yes
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with Lua version : Lua 5.3.4
Built with OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-freebsd 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Available polling systems :
kqueue : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use kqueue.
Available filters :
[TRACE] trace
[COMP] compression
[SPOE] spoe
This sounds similar to this report:
But the patch did not solve it for me. Maybe it’s totally unrelated to SSL in general.
Thanks!
- Frank
