Haproxy consuming 100% cpu

Help!
1

My Haproxy 1.7.9 instance is using 100% of cpu core - for 10-50min, then all goes to normal.

chart2
chart2.png1258×446 16 KB

Couple of mbit/s of traffic.

haproxy -vvv
HA-Proxy version 1.7.9 2017/08/18
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = native
  CC      = gcc
  CFLAGS  = -O2 -march=native -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_TFO=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
        [COMP] compression
        [TRACE] trace
        [SPOE] spoe

Runnig on Ubuntu 16.04

Trace from haproxy process consuming whole core:
https://pilv.laevakompanii.ee/s/R8sYfgYNpVjbwdQ

No errors in debug log.

2

This may be related to SSL handshakes.
Are you reloading or restarting haproxy often? Does the high CPU load always comes after a reload/restart?

Please share you configuration.

3

My haproxy gets reloaded max 10 times a day, but those cpu pikes are not reload related.
I do have conf rendered from consul-template and haproxy is runnging through Multibinder (https://github.com/github/multibinder/tree/master/haproxy).

But I guess those pikes started when I changed ssl cert path to folder (ca 15 certs in it).

I do have 2 other haproxy (same version) instances running on the same server, which do not cause troubles (don’t have ssl folder as cert path)

Conf is here.
https://pilv.laevakompanii.ee/s/ZwxkKESOTQ0LPIW

4

Can you try the following snapshot:
http://www.haproxy.org/download/1.7/src/snapshot/haproxy-ss-20171017.tar.gz

There are fixes post-1.7.9 that may be relevant here (epoll and stream-int fixes).

5

Sure, running snapshot now…
I’ll report back…

6

Everything’s running ok with this snapshot. Not a single cpu spike for last 22 hours.