# HaProxy stops working after a period of time

**URL:** <https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756>\
**Category:** Help!\
**Created:** [January 21, 2020, 6:03pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756 "2020-01-21T18:03:20Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![elizabethk](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@elizabethk](https://discourse.haproxy.org/u/elizabethk)\
**Post date:** [January 21, 2020, 6:03pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756/1 "2020-01-21T18:03:20Z")

</div>

I’m having this strange issue where HAProxy randomly stops working for a specific URL/IP and I cannot figure out the reason. A restart of the service will resolve the issue but I need to determine a permanent fix for this rather than a workaround cronjob that restarts the service periodically.

I am proxying a request from a server to an API

When HAProxy is working my logs will look like below (URL replaced with ‘site’ and IP changed for confidentiality):

**/var/log/haproxy/access.log-20200117.gz:**  
haproxy[26037]: 10.10.50.50.:42386 [16/Jan/2020:13:39:14.934] site-uat\_site.com site-uat\_com/site-uat\_com 0/153/272 1104 – 1/1/0/1/0 0/0

and then at some random time, possibly after a request hasn’t been sent to this URL for a while I get the below in my logs:

**/var/log/haproxy/access.log-20200118.gz:**  
haproxy[26037]:10.10.50.10:60657 [17/Jan/2020:12:49:26.065] site-uat\_site.com site-uat\_com/site-uat\_com 0/-1/120008 212 sC 0/0/0/0/3 0/0

Note the 212 error code and sC termination code. I’ve tried doing some research into this:  
s: the server-side time-out expired first.  
C: waiting for CONNECTION to establish on the server. The server might at most have noticed a connection attempt.

I check the log on the backend of the proxy and I don’t see any traffic reaching it.

Any help or ideas would be greatly appreciated, I’m even a little stuck on how to start troubleshooting this.

Config:

```
frontend site-uat_site_com
           bind 10.10.50.50:6443
           mode tcp
           log global
           option tcplog
           option dontlognull
           timeout client 90s
           use_backend site-uat_com

backend site-uat_com
            mode http
            timeout connect 30s
            timeout server 30s
            balance roundrobin
            http-request set-header Host nonprod-site.com
            server site-uat_com nonprod-site.com:443 ssl verify none
```

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [January 21, 2020, 11:32pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756/2 "2020-01-21T23:32:23Z")

</div>

It’s not a 212 error code, it’s 212 bytes read from the server, when it aborted.

Your configuration is very confusing I don’t understand what you are trying to achieve.

You have a HTTP (not HTTPS) request reching port 6443 (why 6 **443** if it isn’t HTTPS), but you don’t use `mode http` here, and then you go through a backend in http mode with SSL on the server.

Can you clarify what it is that you are expecting haproxy todo?

---

<div class="post-metadata">

**Author:** ![elizabethk](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@elizabethk](https://discourse.haproxy.org/u/elizabethk)\
**Post date:** [January 22, 2020, 1:27pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756/3 "2020-01-22T13:27:13Z")

</div>

Right thank you! I had read that it wasn’t a 212 error code but kept hearing that from my team. Changed my subject line accordingly.

We use HAProxy to be able to support TLS1.2 between a very old in house application and the API the traffic is sent out to, since this traffic goes over the Internet. I also didn’t do the initial config so I can’t speak to why they went this direction, but I do know 6443 could very well be 8086 or something to make more sense but wouldn’t change the behaviour.

However, I did restart HAProxy yesterday evening and once again by this morning the traffic has stopped reaching the API. I haven’t confirmed that the traffic isn’t leaving the server but I have confirmed it’s not reaching the API and I know if I restart HAProxy right now this issue will resolve.

Let me read more into the modes, I have no idea why the frontend is tcp, that does seem odd to me too if you’re saying it should probably be http.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [January 22, 2020, 3:28pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756/4 "2020-01-22T15:28:05Z")

</div>

Can you provide the complete configuration and the output of `haproxy -vv`.

I assume the reason for this are missconfigured timeouts.

---

<div class="post-metadata">

**Author:** ![elizabethk](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@elizabethk](https://discourse.haproxy.org/u/elizabethk)\
**Post date:** [January 22, 2020, 4:18pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756/5 "2020-01-22T16:18:16Z")

</div>

Thanks for your assistance Lukas. I’ve included the requested info, I left out the unrelated frontend/backend configs.

**[root@server loc]# haproxy -vv**  
HA-Proxy version 1.7.8 2017/07/07  
Copyright 2000-2017 Willy Tarreau [willy@haproxy.org](mailto:willy@haproxy.org)

Build options :  
TARGET = linux2628  
CPU = generic  
CC = gcc  
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -DTCP\_USER\_TIMEOUT=18  
OPTIONS = USE\_LINUX\_TPROXY=1 USE\_ZLIB=1 USE\_REGPARM=1 USE\_OPENSSL=1 USE\_PCRE=1

Default settings :  
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes  
Built with zlib version : 1.2.7  
Running on zlib version : 1.2.7  
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)  
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013  
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013  
OpenSSL library supports TLS extensions : yes  
OpenSSL library supports SNI : yes  
OpenSSL library supports prefer-server-ciphers : yes  
Built with PCRE version : 8.32 2012-11-30  
Running on PCRE version : 8.32 2012-11-30  
PCRE library supports JIT : no (USE\_PCRE\_JIT not set)  
Built without Lua support  
Built with transparent proxy support using: IP\_TRANSPARENT IPV6\_TRANSPARENT IP\_FREEBIND

Available polling systems :  
epoll : pref=300, test result OK  
poll : pref=200, test result OK  
select : pref=150, test result OK  
Total: 3 (3 usable), will use epoll.

Available filters :  
[COMP] compression  
[TRACE] trace  
[SPOE] spoe

**Global/Default Configs:**

```
global
            maxconn 100000
            stats socket /var/run/haproxy.stat mode 600 level admin
            log 127.0.0.1 local2 debug
            chroot /var/empty
            ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP
            ssl-default-bind-options no-tls-tickets
            ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP
            ssl-default-server-options no-tls-tickets
            tune.ssl.default-dh-param 1024
            daemon

defaults
        timeout client 30s
        timeout connect 5s
        timeout server 5s

frontend dfc-uat_d_com
        bind 10.10.50.50:6443
        mode tcp
        log global
        option tcplog
        option dontlognull
        timeout client 90s
        use_backend dfc-uat_com

backend dfc-uat_com
        mode http
        timeout connect 30s
        timeout server 30s
        balance roundrobin
        http-request set-header Host nonprod-dfc.com

        #New PsPrint LZ URL
        server dfc-uat_com nonprod-dfc.com:443 ssl verify none
```

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [January 22, 2020, 11:00pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756/6 "2020-01-22T23:00:58Z")

</div>

Are you sure you have given me the correct configuration?

As per your error log, the connection timed out after 120 seconds in “timeout connect”:

```
haproxy[26037]:10.10.50.10:60657 [17/Jan/2020:12:49:26.065] site-uat_site.com site-uat_com/site-uat_com 0/-1/120008 212 sC 0/0/0/0/3 0/0

```

But none of the configuration snippets you provided matches a 120 second timeout.

I suggest to move from tcp to http mode and upgrade the logging method to http. This will give us a better picture in the log when you hit that issue.

So instead of:

```
       mode tcp
       log global
       option tcplog

```

you use

```
       mode http
       log global
       option httplog

```

Also, when in this situation, please checkout if you have stale sockets with netstat.

---

<div class="post-metadata">

**Author:** ![elizabethk](https://avatars.discourse-cdn.com/v4/letter/e/41988e/32.png) [@elizabethk](https://discourse.haproxy.org/u/elizabethk)\
**Post date:** [January 25, 2020, 5:44pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756/7 "2020-01-25T17:44:25Z")

</div>

I determined that the problem is with the backend hostname:

server dfc-uat\_com [nonprod-dfc.com:443](http://nonprod-dfc.com:443) ssl verify none

There are two dynamic IPs associated with [nonprod-dfc.com](http://nonprod-dfc.com) and HAProxy requires a reload/restart to update the IP in it’s cache for this hostname.

Could anyone confirm that I could resolve this with:

```
resolvers dns
  nameserver public-0 xx.xx.xx.xx:53
  hold valid 1s

frontend http
  bind *:8000
  default_backend site-backend

backend site-backend
  balance leastconn
  server site sub.example.com:80 resolvers dns check inter 1000
```

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [January 26, 2020, 12:08pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-after-a-period-of-time/4756/8 "2020-01-26T12:08:01Z")

</div>

Ah ok, that makes sense.

I confirm that DNS resolution as you configured should fix the issue, although in this configuration you make a DNS request every second. Consider using 10 or 30 seconds for the `hold valid` timeout.