# Old haproxy processes accepting new connections after reload

**URL:** <https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614>\
**Category:** Help!\
**Created:** [August 26, 2016, 8:31am UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614 "2016-08-26T08:31:16Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![utkarsh.k](https://avatars.discourse-cdn.com/v4/letter/u/ecb155/32.png) [@utkarsh.k](https://discourse.haproxy.org/u/utkarsh.k)\
**Post date:** [August 26, 2016, 8:31am UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/1 "2016-08-26T08:31:16Z")

</div>

Hey,

So I’ve read several questions regarding haproxy having old processes still running.  
such as:  
[https://www.mail-archive.com/search?l=haproxy@formilux.org&q=subject:“Re%3A+HAProxy+graceful+restart+old+process+not+going+away”&o=newest&f=1](https://www.mail-archive.com/search?l=haproxy@formilux.org&q=subject:%22Re%3A+HAProxy+graceful+restart+old+process+not+going+away%22&o=newest&f=1)

We are using haproxy version 1.5. According to the documentation a reload for the configration file using -sF options issues a -SIGUSR1 signal to the old processes . However only when I issue this signal explicitly the old processes gracefully exit.  
Is sending this signal explicitly a bad practice ? Why isn’t it happening automatically ?

Further the main issue at hand here is ,that the old processes continue to accept new connections .  
Apart from sending the -SIGUSR1 signal , explicitly when the reload script executed is/are there any alternative solution(s) ?

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 26, 2016, 1:19pm UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/2 "2016-08-26T13:19:49Z")

</div>

Which 1.5 release exactly? Does the PID file contain the correct PID and is the file read and writable?

---

<div class="post-metadata">

**Author:** ![utkarsh.k](https://avatars.discourse-cdn.com/v4/letter/u/ecb155/32.png) [@utkarsh.k](https://discourse.haproxy.org/u/utkarsh.k)\
**Post date:** [August 29, 2016, 9:21am UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/3 "2016-08-29T09:21:38Z")

</div>

Hey @lukastribus we are using HA-Proxy version 1.5-dev25-a339395 2014/05/10, and yes the PID file contains the correct PID and file is readable and writable.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 29, 2016, 12:36pm UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/4 "2016-08-29T12:36:36Z")

</div>

Is this a duplicate thread?

> [@Haproxy multiple processes after reload](http://discourse.haproxy.org/t/haproxy-multiple-processes-after-reload/616):
>
> Hi, We experienced this recent issue with haproxy where multiple haproxy processes were spawned after a reload. We use the following haproxy build: HA-Proxy version 1.5-dev25-a339395 2014/05/10 Copyright 2000-2014 Willy Tarreau [w@1wt.eu](mailto:w@1wt.eu) Build options : TARGET = linux26 CPU = generic CC = gcc CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing OPTIONS = USE\_OPENSSL=1 USE\_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 En…

Anyway, the release you are using is not supported; its an ancient development build that you should not use in production. Please upgrade to latest 1.5 or 1.6 stable release.

---

<div class="post-metadata">

**Author:** ![amitgera](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.haproxy.org/amitgera/32/111_2.png) [@amitgera](https://discourse.haproxy.org/u/amitgera)\
**Post date:** [September 9, 2016, 6:51am UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/5 "2016-09-09T06:51:59Z")

</div>

1.5.18, haven’t check about pid , will revert you soon for the following info

---

<div class="post-metadata">

**Author:** ![rlib](https://avatars.discourse-cdn.com/v4/letter/r/74df32/32.png) [@rlib](https://discourse.haproxy.org/u/rlib)\
**Post date:** [October 5, 2016, 8:42pm UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/6 "2016-10-05T20:42:46Z")

</div>

Haproxy 1.6.8:  
[root@drft001 ~]# haproxy -v  
HA-Proxy version 1.6.8 2016/08/14  
Copyright 2000-2016 Willy Tarreau [willy@haproxy.org](mailto:willy@haproxy.org)

I have the same issue, after  
service haproxy reload  
there are new processes with old one. So I restart the server instead of reloading to avoid this situation.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [October 6, 2016, 9:35am UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/7 "2016-10-06T09:35:44Z")

</div>

Check the PID file. Does it contains the correct PID? Is the file writeable from haproxy?

---

<div class="post-metadata">

**Author:** ![pierrecdn](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.haproxy.org/pierrecdn/32/124_2.png) [@pierrecdn](https://discourse.haproxy.org/u/pierrecdn)\
**Post date:** [October 13, 2016, 3:04pm UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/8 "2016-10-13T15:04:46Z")

</div>

Hi,  
I experiment the same behaviour.  
I’m on 1.6.8 as well, and thought that upgrading from 1.4 or 1.5 could solve the issue. That was not the case.

Here is an example of process tree:

```auto
root 24115 0.0 0.0 46340 1824 ? Ss 14:34 0:00 /usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy 27403 0.2 0.0 89272 20096 ? S 14:49 0:00 \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 27366
haproxy 27450 1.2 0.0 89272 14380 ? Rs 14:49 0:00 | \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 27366
haproxy 27410 0.2 0.0 89272 16008 ? S 14:49 0:00 \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 27366
haproxy 27458 1.2 0.0 89272 14392 ? Ss 14:49 0:00 | \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 27366
haproxy 27626 0.3 0.0 89272 16008 ? S 14:49 0:00 \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 27623
haproxy 27674 1.1 0.0 89272 14380 ? Ss 14:49 0:00 | \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 27623
haproxy 27722 0.2 0.0 89272 16008 ? S 14:49 0:00 \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 27716
haproxy 27762 1.0 0.0 89272 14368 ? Ss 14:49 0:00 | \_ /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds -sf 27716

```

The problem is easily repoducible: just loop over reload, 50 times without sleep for example.

It happens when two reloads are performed in a small amount of time. As a result, there is no ‘back-reference’ in the ‘-sf’ of one haproxy instance to the previous one. It becomes “disconnected” from the others. It is also visible in journalctl output (2 haproxy instance has the same PID reference in ‘-sf’, resulting in one lost).

I had a look at haproxy-systemd-wrapper.c and guess that the PID file is only read and never written by this component.  
To me it seems that a race condition happens and that several instances do not reference the previous one.

Restarting the server is very impacting as you may know, and this is why there was approach like the one in Yelp ([https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html](https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html)) consisting in letting the client do SYN-retries or buffering the SYNs while doing a full restart).  
This becomes impossible in PaaS-like approach where many events occurs and may trigger reloads every seconds.

Maybe you’ve some insights to share before digging into that ?

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [October 13, 2016, 3:47pm UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/9 "2016-10-13T15:47:14Z")

</div>

Please report this to the mailing list, they will be able to help you much better than here on discourse.

edit: Starting with snapshot 20161005 [1] and the unreleased v1.6.10 you will be able to disable SO\_REUSEPORT in the configuration (no-reuseport). That could be something to try …

[1] [http://www.haproxy.org/download/1.6/src/snapshot/haproxy-ss-20161005.tar.gz](http://www.haproxy.org/download/1.6/src/snapshot/haproxy-ss-20161005.tar.gz)

---

<div class="post-metadata">

**Author:** ![pierrecdn](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.haproxy.org/pierrecdn/32/124_2.png) [@pierrecdn](https://discourse.haproxy.org/u/pierrecdn)\
**Post date:** [October 13, 2016, 5:18pm UTC](https://discourse.haproxy.org/t/old-haproxy-processes-accepting-new-connections-after-reload/614/10 "2016-10-13T17:18:45Z")

</div>

Thanks Lukas.  
I just sent my question to the ML : [https://www.mail-archive.com/haproxy@formilux.org/msg23867.html](https://www.mail-archive.com/haproxy@formilux.org/msg23867.html)