Haproxy Memory Issue : its old process have occupied hefty memory

A day before we have taken reload for haproxy and next day we noticed that old process still occupying 4GB memory

~~Trace logs: Top 4 memory occupied processes are :
root@ip-10-0-x-xx:~# ps ax -o rss,user,command | sort -nr | head -n 4
8799424 haproxy /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 630
4918408 haproxy /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 27615
86292 ganglia /usr/sbin/gmond --pid-file=/var/run/ganglia-monitor.pid
29628 syslog rsyslogd

root@ip-10-0-x-xx:~#ps -ef | grep ha
haproxy 630 1 19 Jul28 ? 5-11:40:10 /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 27615
haproxy 13810 1 19 Aug24 ? 03:51:22 /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 630

processid :630 is old one which have occupied approx 4GB of ram
processid :13810 is latest one which have occupied approx 8GB of ram

Then we have checked that old id still have some few connection established :
root@ip-10-0-x-xx:~# lsof -p 630| grep TCP |wc -l
Note: 104 here means one for incoming and one for outgoing (i.e backend node)
root@ip-10-0-x-xx:~# lsof -p 630 | grep -v “ap-xxxx-y.compute.internal:xxxx” |grep TCP |wc -l

Please help me understand even with very less number of these connection on old process why it has occupied hefty memory

Please help me in order to find out the rca

Please use a single thread, not 3 threads for one problem. Crossposting is not getting you help any sooner, it will just led to wrong advice for the wrong people.

Post the output of “haproxy -vv” and the configuration (especially important are timeout values).

ubuntu@ip-10-0-x-xx:~$ haproxy -vv
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = native
CC = gcc
CFLAGS = -O2 -march=native -g -fno-strict-aliasing

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built without zlib support (USE_ZLIB not set)
Compression algorithms supported : identity
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.


ubuntu@ip-10-0-x-xx:~$ cat /etc/haproxy/haproxy.cfg
maxconn 500000
user haproxy
group haproxy
tune.ssl.default-dh-param 2048
stats socket /var/run/haproxy.socket group ganglia mode 775 level operator

retries 3
option redispatch
maxconn 500000
timeout connect 300000
timeout client 660000
timeout server 660000

frontend inbound
rate-limit sessions 5000
mode tcp
bind *:port1
bind *:port2
bind *:port3
bind *:443 ssl crt /etc/ssl-cert-path/new-x.y.z.crt.pem
default_backend bk_servers

backend bk_servers
balance listen
server namexyz 10.0.x.yz:port check weight 100 inter 5000

Don’t use DLMALLOC unless you know exactly what you are doing. I have not seen a DLMALLOC build in a long time, it won’t certainly resolve your problems magically.

Seems to me the connection don’t timeout, I guess they continue to generate traffic so they don’t timeout.

I suggest you tcpdump those connections to see what kind of traffic still runs through those old processes (pick some of the ports from the lsof output and make a tcpdump filter for it).