On a production server the memory usage of HAProxy has reached 75.7% out of 8 GB and the usage is growing as time passes by, so I have the feeling that we might have memory leak somewhere. We use HAProxy 2.5, any tips on how to fix/diagnose the problem is much appreciated.
The number 1 reason is maxconn values incompatible with your RAM situation, often combined with too large timeouts.
So maxconn and timeout values are the first thing to look at.
The other reason can be
tune. values that are set too large.
I guess you could cause large memory consumption with bugs in LUA scripts too. Compression, caching and other filters may also contribute to excessive memory consumption.
The output of
haproxy -vv and the configuration would help here, however 2.5 is unsupported at this point. But I’m not aware of any memory leaks in the last decade to be honest, so I doubt this is an actual bug in haproxy.
Hello @lukastribus thanks for your time.
these are our timeout and maxconn values
maxconn 60000 timeout connect 5000ms timeout client 50000ms timeout server 3000ms timeout tunnel 1h
and here is haproxy -vv output
HAProxy version 2.5.10 2022/12/05 - https://haproxy.org/ Status: stable branch - will stop receiving fixes around Q1 2023. Known bugs: http://www.haproxy.org/bugs/bugs-2.5.10.html Running on: Linux 5.10.0-23-cloud-amd64 #1 SMP Debian 5.10.179-2 (2023-07-14) x86_64 Build options : TARGET = linux-glibc CPU = generic CC = cc CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_PROMEX=1 DEBUG = Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL +THREAD +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +ACCEPT4 -CLOSEFROM -ZLIB +SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL -PROCCTL +THREAD_DUMP -EVPORTS -OT -QUIC +PROMEX -MEMORY_PROFILING Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_THREADS=64, default=2). Built with OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022 Running on OpenSSL version : OpenSSL 1.1.1n 15 Mar 2022 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 Built with Lua version : Lua 5.3.3 Built with the Prometheus exporter as a service Built with network namespace support. Built with libslz for stateless compression. Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Support for malloc_trim() is enabled. Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with PCRE2 version : 10.32 2018-09-10 PCRE2 library supports JIT : yes Encrypted password support via crypt(3): yes Built with gcc compiler version 8.3.0 Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG <default> : mode=HTTP side=FE|BE mux=H1 flags=HTX h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG <default> : mode=TCP side=FE|BE mux=PASS flags= none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG Available services : prometheus-exporter Available filters : [SPOE] spoe [CACHE] cache [FCGI] fcgi-app [COMP] compression [TRACE] trace
I am sharing this info here for you to have a closer look on the problem we are facing,
We saw that our fresh copy of haproxy image started from almost 130MB, but in few days when almost 80 backends were created dynamically, with each has 100 servers, haproxy memory usage was increased to almost 6GB, although it increased gradually in 10 - 13 days to this level.
I will take little time to think about your suggestion regarding lua script. And will update