The old process will not terminate about a few days after haproxy reload

Help!
1

  • Haproxy version
    haproxy -v
    HA-Proxy version 1.5.18 2016/05/10
    Copyright 2000-2016 Willy Tarreau willy@haproxy.org

  • I want to refresh the haproxy configuration without any interrupt,so when I edited haproxy.cfg file ,and do the service haproxy reload action. but some old processes will not terminated after a few days

cps 891 1 0 Oct30 ? 00:13:39 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 892 1 0 Oct30 ? 00:13:42 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 893 1 0 Oct30 ? 00:13:44 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 894 1 0 Oct30 ? 00:13:45 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 1059 1 0 Oct31 ? 00:09:07 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 1061 1 0 Oct31 ? 00:09:06 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 1062 1 0 Oct31 ? 00:09:09 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 1063 1 0 Oct31 ? 00:09:08 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 6385 1 0 Oct25 ? 00:44:45 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 6387 1 0 Oct25 ? 00:46:08 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 6388 1 0 Oct25 ? 00:46:15 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 6390 1 0 Oct25 ? 00:47:05 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 15206 1 0 Oct26 ? 00:41:27 /usr/sbin/haproxy -p /var/run/haproxy.pid -D -f /etc/haproxy/haproxy.cfg -sf 21877 21878 21879 21880
cps 15208 1 0 Oct26 ? 00:41:53 /usr/sbin/haproxy -p /var/run/haproxy.pid -D -f /etc/haproxy/haproxy.cfg -sf 21877 21878 21879 21880
cps 15209 1 0 Oct26 ? 00:41:48 /usr/sbin/haproxy -p /var/run/haproxy.pid -D -f /etc/haproxy/haproxy.cfg -sf 21877 21878 21879 21880
cps 15210 1 0 Oct26 ? 00:41:31 /usr/sbin/haproxy -p /var/run/haproxy.pid -D -f /etc/haproxy/haproxy.cfg -sf 21877 21878 21879 21880
cps 16608 1 0 Oct30 ? 00:13:28 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 16610 1 0 Oct30 ? 00:13:31 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 16613 1 0 Oct30 ? 00:13:32 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 16616 1 0 Oct30 ? 00:13:29 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 25144 1 0 Oct30 ? 00:13:41 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 25145 1 0 Oct30 ? 00:13:42 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 25146 1 0 Oct30 ? 00:13:43 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 25147 1 0 Oct30 ? 00:13:45 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 26476 1 0 Oct30 ? 00:13:39 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 26478 1 0 Oct30 ? 00:13:36 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 26479 1 0 Oct30 ? 00:13:38 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid
cps 26480 1 0 Oct30 ? 00:13:40 /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /etc/haproxy/haproxy.pid

  • Is there anybody have any ideas to resolve this issues?
2

Depending on whether you still see established connections to the old process in netstat or ss, those are either active connections or it also could be a haproxy bug.

I suggest you check what connections still point to the old PID and review the configured timeouts in the configuration - if they are too high or not set, old connections may take a very long time or forever to go away.

3

Yes，you are correct, I have seen some connections binded to the process with established state.

  • My timeout configuration as follows
    global
    log 127.0.0.1 local2 debug
    maxconn 250000
    user root
    group root
    daemon
    nbproc 4
    defaults
    log global
    mode tcp
    option tcplog
    option dontlognull
    retries 3
    option redispatch
    maxconn 4000
    timeout connect 5000
    timeout client 120000
    timeout server 120000

  • Now I have two questions

  1. Is the old process will accept the connection?
  2. Is there any ideas to close the connections after we do the reload action, at least a timeout , because we are worried about the old processes will process the request with some unexpected issues.
4

No, the old process does not accept new connections. Those are old connections that keep making traffic (refreshing timeouts), which is why they are not closed.

Either you kill the old process manually, or you upgrade the a newer haproxy release, supporting the hard-stop-after directive, which can auto-kill such connections based on a timer. You need at least 1.7 for that (hard-stop-after is in 1.7.4 and later).

5

Thanks,I got your point. and I have found the parameter it could resolve my issues,But I am so sad because our env version is lower than this version