# Reloads of haproxy 1.8.3 under SystemD cause "finishing" children to stack-up

**URL:** <https://discourse.haproxy.org/t/reloads-of-haproxy-1-8-3-under-systemd-cause-finishing-children-to-stack-up/1977>\
**Category:** Help!\
**Created:** [January 11, 2018, 9:12am UTC](https://discourse.haproxy.org/t/reloads-of-haproxy-1-8-3-under-systemd-cause-finishing-children-to-stack-up/1977 "2018-01-11T09:12:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rh20i](https://avatars.discourse-cdn.com/v4/letter/r/aca169/32.png) [@rh20i](https://discourse.haproxy.org/u/rh20i)\
**Post date:** [January 11, 2018, 9:12am UTC](https://discourse.haproxy.org/t/reloads-of-haproxy-1-8-3-under-systemd-cause-finishing-children-to-stack-up/1977/1 "2018-01-11T09:12:10Z")

</div>

Hi everyone

I work for a large-ish UK web hosting company and we’re slowly introducing HTTP/2 now that’s it’s in haproxy 1.8.

We’re using haproxy 1.8.3 under SystemD on CentOS 7.4 (3.10.0-693.el7.x86\_64) KVM virtual machines, built with the following make line:

```auto
%{__make} CPU="generic" TARGET=linux2628 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1

```

HTTP/2 is enabled for both of our frontend’s IPv4 and IPv6 “bind” lines:

```auto
frontend httpexternal
    bind *:80
    bind *:443 ssl crt /etc/haproxy/stackcerts/www.stackssl.com.pem ssl crt /etc/haproxy/certs/ alpn h2,http/1.1
    bind :::80
    bind :::443 ssl crt /etc/haproxy/stackcerts/www.stackssl.com.pem ssl crt /etc/haproxy/certs/ alpn h2,http/1.1

```

We have observed behavior where haproxy reloads cause a situation where the “finishing” PID never actually finishes, despite lsof showing that it has no active TCP connections.

If we disable HTTP/2 completely (IPv4 and IPv6) the problem goes away, and the finishing PIDs do indeed go away when they have no more connections.

After lots of lsof’ing later, we noticed that a child we expect to have finished had this line in lsof output:

```auto
haproxy 26670 haproxy 429u sock 0,7 0t0 31631114 protocol: TCPv6

```

but no actual IPv6 connections. We see the same behavior for “finishing” processes that never go away that are for IPv4. It has a high-numbered File Descriptor which makes me think it was from a connection that was being used to serve HTTP requests.

The same behaviour occurs if we’re in single-threaded or multi-threaded mode (modified through config, not recompiling).

We think [http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=4dbce456a223de3d06873828185ba789d5043def](http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=4dbce456a223de3d06873828185ba789d5043def) might be related in some way.

I hope this report helps.

Please let me know if you require any further information.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [January 11, 2018, 6:43pm UTC](https://discourse.haproxy.org/t/reloads-of-haproxy-1-8-3-under-systemd-cause-finishing-children-to-stack-up/1977/2 "2018-01-11T18:43:53Z")

</div>

A few questions:

- can you share your configuration regarding the timeouts?
- are you using systemd in notify mode (starting with -Ws and Type=notify, as per `contrib/systemd/haproxy.service.in`)?
- do you use [hard-stop-after](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#hard-stop-after)? You should set that in any case to something that matches your expections, otherwise a small client or attacker can keep your old processes handing around forever (just to be clear, that doesn’t mean this couldn’t be a bug)

Also there are two important bugs you will hit in a hosting environment with 1.8.3:

- H2 issue in in IE11/Edge browsers: [http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=646d23d1b502bc07a4a846f2ca7d332506b3087e](http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=646d23d1b502bc07a4a846f2ca7d332506b3087e)
- SSL cache failure: [http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=52a80823e8c2d04635cc95e5d0ca9440a53441cf](http://git.haproxy.org/?p=haproxy-1.8.git;a=commit;h=52a80823e8c2d04635cc95e5d0ca9440a53441cf)

Both of those bugs are important in a hosting environment imho, and your issue could be related to the H2 issue with empty data frames from IE11/Edge clients.

I suggest you upgrade to latest 1.8 git tree or, if you prefer a tarball, to yesterdays 1.8 snapshot:  
`http://www.haproxy.org/download/1.8/src/snapshot/haproxy-ss-20180110.tar.gz`

cheers,  
lukas

---

<div class="post-metadata">

**Author:** ![rh20i](https://avatars.discourse-cdn.com/v4/letter/r/aca169/32.png) [@rh20i](https://discourse.haproxy.org/u/rh20i)\
**Post date:** [January 16, 2018, 8:04am UTC](https://discourse.haproxy.org/t/reloads-of-haproxy-1-8-3-under-systemd-cause-finishing-children-to-stack-up/1977/3 "2018-01-16T08:04:16Z")

</div>

Hi Lukas

Timeouts (in global)

```auto
    timeout connect 10s
    timeout queue 30s
    timeout client 1m
    timeout server 5m
    timeout check 10s
    timeout http-request 10s

```

Invocation - I did use the values from `contrib/systemd/haproxy.service.in` but it looks like I missed `Type=notify` but everything else is correct.

We are not using `hard-stop-after`.

I will re-try with latest snapshot, Type=notify and hard-stop-after enabled and get back to you.

Thank you very much.

---

<div class="post-metadata">

**Author:** ![rh20i](https://avatars.discourse-cdn.com/v4/letter/r/aca169/32.png) [@rh20i](https://discourse.haproxy.org/u/rh20i)\
**Post date:** [January 16, 2018, 9:46am UTC](https://discourse.haproxy.org/t/reloads-of-haproxy-1-8-3-under-systemd-cause-finishing-children-to-stack-up/1977/4 "2018-01-16T09:46:02Z")

</div>

I forgot to mention this - we see negative conns in the general process information section sometimes:

```auto
pid = 13087 (process #1, nbproc = 1)
uptime = 0d 0h13m59s
system limits: memmax = unlimited; ulimit-n = 20137
maxsock = 20137; maxconn = 10000; maxpipes = 0
current conns = -1384; current pipes = 0/0; conn rate = 109/sec
Running tasks: 1/1150; idle = 43 %

```

That’s 1 process and no threading.

…

And sometimes it’s fine:

```auto
pid = 15130 (process #1, nbproc = 1)
uptime = 0d 0h02m15s
system limits: memmax = unlimited; ulimit-n = 20137
maxsock = 20137; maxconn = 10000; maxpipes = 0
current conns = 736; current pipes = 0/0; conn rate = 101/sec
Running tasks: 1/1039; idle = 48 %

```

---

<div class="post-metadata">

**Author:** ![rh20i](https://avatars.discourse-cdn.com/v4/letter/r/aca169/32.png) [@rh20i](https://discourse.haproxy.org/u/rh20i)\
**Post date:** [January 17, 2018, 8:43am UTC](https://discourse.haproxy.org/t/reloads-of-haproxy-1-8-3-under-systemd-cause-finishing-children-to-stack-up/1977/5 "2018-01-17T08:43:16Z")

</div>

Hi Lukas

I can confirm that adding `Type=notify` and using the recommended haproxy version did not solve the problem.

Adding `hard-stop-after` did fix it, but we’d prefer not to use that: we’re happy to allow customers to run download sites and we wouldn’t want to ultimately impose a maximum time limit on how long a download can take. That said, we don’t believe it’s download sites causing this issue because the processes that remain don’t have any open connections to the Internet.

Any other ideas?

Thank you for your help so far.