A day before we have taken reload for haproxy and next day we noticed that old process still occupying 4GB memory
~~Trace logs: Top 4 memory occupied processes are :
root@ip-10-0-x-xx:~# ps ax -o rss,user,command | sort -nr | head -n 4
8799424 haproxy /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 630
4918408 haproxy /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 27615
86292 ganglia /usr/sbin/gmond --pid-file=/var/run/ganglia-monitor.pid
29628 syslog rsyslogd
root@ip-10-0-x-xx:~#ps -ef | grep ha
haproxy 630 1 19 Jul28 ? 5-11:40:10 /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 27615
haproxy 13810 1 19 Aug24 ? 03:51:22 /usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 630
Note:
processid :630 is old one which have occupied approx 4GB of ram
processid :13810 is latest one which have occupied approx 8GB of ram
Then we have checked that old id still have some few connection established :
root@ip-10-0-x-xx:~# lsof -p 630| grep TCP |wc -l
104
Note: 104 here means one for incoming and one for outgoing (i.e backend node)
i.e
root@ip-10-0-x-xx:~# lsof -p 630 | grep -v “ap-xxxx-y.compute.internal:xxxx” |grep TCP |wc -l
52
Please help me understand even with very less number of these connection on old process why it has occupied hefty memory
Please help me in order to find out the rca
Please use a single thread, not 3 threads for one problem. Crossposting is not getting you help any sooner, it will just led to wrong advice for the wrong people.
Post the output of “haproxy -vv” and the configuration (especially important are timeout values).
ubuntu@ip-10-0-x-xx:~$ haproxy -vv
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau willy@haproxy.org
Build options :
TARGET = linux2628
CPU = native
CC = gcc
CFLAGS = -O2 -march=native -g -fno-strict-aliasing
OPTIONS = DLMALLOC_SRC=…/malloc.c USE_OPENSSL=1 USE_STATIC_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built without zlib support (USE_ZLIB not set)
Compression algorithms supported : identity
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
ubuntu@ip-10-0-x-xx:~$
ubuntu@ip-10-0-x-xx:~$ cat /etc/haproxy/haproxy.cfg
global
maxconn 500000
user haproxy
group haproxy
daemon
tune.ssl.default-dh-param 2048
stats socket /var/run/haproxy.socket group ganglia mode 775 level operator
defaults
retries 3
option redispatch
maxconn 500000
timeout connect 300000
timeout client 660000
timeout server 660000
frontend inbound
rate-limit sessions 5000
mode tcp
bind *:port1
bind *:port2
bind *:port3
bind *:443 ssl crt /etc/ssl-cert-path/new-x.y.z.crt.pem
default_backend bk_servers
backend bk_servers
balance listen
server namexyz 10.0.x.yz:port check weight 100 inter 5000
ubuntu@ip-10-0-x-xx:~$
Don’t use DLMALLOC unless you know exactly what you are doing. I have not seen a DLMALLOC build in a long time, it won’t certainly resolve your problems magically.
Seems to me the connection don’t timeout, I guess they continue to generate traffic so they don’t timeout.
I suggest you tcpdump those connections to see what kind of traffic still runs through those old processes (pick some of the ports from the lsof output and make a tcpdump filter for it).