HAProxy memory usage reaches 75.7%

Help!
1

On a production server the memory usage of HAProxy has reached 75.7% out of 8 GB and the usage is growing as time passes by, so I have the feeling that we might have memory leak somewhere. We use HAProxy 2.5, any tips on how to fix/diagnose the problem is much appreciated.

2

The number 1 reason is maxconn values incompatible with your RAM situation, often combined with too large timeouts.

So maxconn and timeout values are the first thing to look at.

The other reason can be tune. values that are set too large.

I guess you could cause large memory consumption with bugs in LUA scripts too. Compression, caching and other filters may also contribute to excessive memory consumption.

The output of haproxy -vv and the configuration would help here, however 2.5 is unsupported at this point. But I’m not aware of any memory leaks in the last decade to be honest, so I doubt this is an actual bug in haproxy.

3

Hello @lukastribus thanks for your time.
these are our timeout and maxconn values

  maxconn 60000
  timeout connect 5000ms
  timeout client 50000ms
  timeout server 3000ms
  timeout tunnel 1h

and here is haproxy -vv output

HAProxy version 2.5.10 2022/12/05 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2023.
Known bugs: http://www.haproxy.org/bugs/bugs-2.5.10.html
Running on: Linux 5.10.0-23-cloud-amd64 #1 SMP Debian 5.10.179-2 (2023-07-14) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_PROMEX=1
  DEBUG   = 

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC +PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
Running on OpenSSL version : OpenSSL 1.1.1n  15 Mar 2022
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Support for malloc_trim() is enabled.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.32 2018-09-10
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.3.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG

Available services : prometheus-exporter
Available filters :
	[SPOE] spoe
	[CACHE] cache
	[FCGI] fcgi-app
	[COMP] compression
	[TRACE] trace

I am sharing this info here for you to have a closer look on the problem we are facing,
We saw that our fresh copy of haproxy image started from almost 130MB, but in few days when almost 80 backends were created dynamically, with each has 100 servers, haproxy memory usage was increased to almost 6GB, although it increased gradually in 10 - 13 days to this level.

I will take little time to think about your suggestion regarding lua script. And will update

4

Not sure anymore, there are quite some LUA related leaks in there:

http://www.haproxy.org/bugs/bugs-2.5.10.html

You should probably work on upgrading to a supported branch with the latest bugfix release as per haproxy.org