# Haproxy suddenly stops forwarding

**URL:** <https://discourse.haproxy.org/t/haproxy-suddenly-stops-forwarding/8976>\
**Category:** Help!\
**Created:** [August 28, 2023, 4:27pm UTC](https://discourse.haproxy.org/t/haproxy-suddenly-stops-forwarding/8976 "2023-08-28T16:27:13Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![bladnor](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.haproxy.org/bladnor/32/2023_2.png) [@bladnor](https://discourse.haproxy.org/u/bladnor)\
**Post date:** [August 28, 2023, 4:27pm UTC](https://discourse.haproxy.org/t/haproxy-suddenly-stops-forwarding/8976/1 "2023-08-28T16:27:13Z")

</div>

Out of a suddon (suddenly) haproxy does not forward any more to the backend.  
After a

> systemctl restart haproxy.service

it works again like it should. I can not tell how often this happens. But this time it was the second time within two weeks!

- What would you check on the system when this happens next time?
- Any guesses what the cause could be?
- Any tipps how to work around this?

Thanks for some hints.

Below the relevant config snippet (It is forwarding to a postgres db / There are other rules along this one listening on port 443 with ssl termination):

> global  
> log /dev/log local0  
> chroot /var/lib/haproxy  
> pidfile /var/run/haproxy.pid  
> stats socket /run/haproxy/admin.sock mode 660 level admin  
> stats timeout 30s  
> user haproxy  
> group haproxy  
> daemon
> 
> defaults  
> mode http  
> log global  
> option httpslog  
> timeout connect 240s  
> timeout client 1200s  
> timeout server 1200s
> 
> frontend my-fe  
> bind \*:5432  
> mode tcp  
> option tcplog  
> use\_backend my-backend
> 
> backend my-backend  
> mode tcp  
> server my-servername myip:myport

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 28, 2023, 7:08pm UTC](https://discourse.haproxy.org/t/haproxy-suddenly-stops-forwarding/8976/2 "2023-08-28T19:08:12Z")

</div>

Check CPU load, specifically whether haproxy consumes large amounts of it (or at least one full CPU core). This would indicate a spinning process (which is a bug in haproxy).

CPU usage in the hung condition is likely either very low or very high.

Also check the states of sockets. Does the socket count keep increasing with a lots of old orphaned socket still active?

Run “show info” and “show fd” on the admin socket. The latter produces at lot of outputs.

Absolutely provide the full configuration, not just what you think is relevant. Replace confidential config statements with placeholders, but don’t remove config statements.

And most importantly provide the output of `haproxy -vv`.

---

<div class="post-metadata">

**Author:** ![bladnor](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.haproxy.org/bladnor/32/2023_2.png) [@bladnor](https://discourse.haproxy.org/u/bladnor)\
**Post date:** [August 28, 2023, 8:13pm UTC](https://discourse.haproxy.org/t/haproxy-suddenly-stops-forwarding/8976/3 "2023-08-28T20:13:07Z")

</div>

Below you find the output of haproxy -vv and also the full config.  
Remarks: There is no traffic from real users cause I’m bulding a new system. Only from bots and hackers.  
Thank you very much.

> ~# haproxy -vv  
> HAProxy version 2.6.12-1ppa1~jammy 2023/04/01 - [https://haproxy.org/](https://haproxy.org/)  
> Status: long-term supported branch - will stop receiving fixes around Q2 2027.  
> Known bugs: [http://www.haproxy.org/bugs/bugs-2.6.12.html](http://www.haproxy.org/bugs/bugs-2.6.12.html)  
> Running on: Linux 5.15.0-72-generic #79-Ubuntu SMP Wed Apr 19 08:22:18 UTC 2023 x86\_64  
> Build options :  
> TARGET = linux-glibc  
> CPU = generic  
> CC = cc  
> CFLAGS = -O2 -g -O2 -flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D\_FORTIFY\_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment  
> OPTIONS = USE\_PCRE2=1 USE\_PCRE2\_JIT=1 USE\_OPENSSL=1 USE\_LUA=1 USE\_SLZ=1 USE\_SYSTEMD=1 USE\_OT=1 USE\_PROMEX=1  
> DEBUG = -DDEBUG\_STRICT -DDEBUG\_MEMORY\_POOLS
> 
> Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU\_AFFINITY +CRYPT\_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX\_SPLICE +LINUX\_TPROXY +LUA -MEMORY\_PROFILING +NETFILTER +NS -OBSOLETE\_LINKER +OPENSSL +OT -PCRE +PCRE2 +PCRE2\_JIT -PCRE\_JIT +POLL +PRCTL -PROCCTL +PROMEX -QUIC +RT +SLZ -STATIC\_PCRE -STATIC\_PCRE2 +SYSTEMD +TFO +THREAD +THREAD\_DUMP +TPROXY -WURFL -ZLIB
> 
> Default settings :  
> bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
> 
> Built with multi-threading support (MAX\_THREADS=64, default=1).  
> Built with OpenSSL version : OpenSSL 3.0.2 15 Mar 2022  
> Running on OpenSSL version : OpenSSL 3.0.2 15 Mar 2022  
> OpenSSL library supports TLS extensions : yes  
> OpenSSL library supports SNI : yes  
> OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3  
> OpenSSL providers loaded : default  
> Built with Lua version : Lua 5.3.6  
> Built with the Prometheus exporter as a service  
> Built with network namespace support.  
> Built with OpenTracing support.  
> Support for malloc\_trim() is enabled.  
> Built with libslz for stateless compression.  
> Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)  
> Built with transparent proxy support using: IP\_TRANSPARENT IPV6\_TRANSPARENT IP\_FREEBIND  
> Built with PCRE2 version : 10.39 2021-10-29  
> PCRE2 library supports JIT : yes  
> Encrypted password support via crypt(3): yes  
> Built with gcc compiler version 11.3.0
> 
> Available polling systems :  
> epoll : pref=300, test result OK  
> poll : pref=200, test result OK  
> select : pref=150, test result OK  
> Total: 3 (3 usable), will use epoll.
> 
> Available multiplexer protocols :  
> (protocols marked as cannot be specified using ‘proto’ keyword)  
> h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL\_RISK|NO\_UPG  
> fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL\_RISK|NO\_UPG  
> h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO\_UPG  
> : mode=HTTP side=FE|BE mux=H1 flags=HTX  
> none : mode=TCP side=FE|BE mux=PASS flags=NO\_UPG  
> : mode=TCP side=FE|BE mux=PASS flags=
> 
> Available services : prometheus-exporter  
> Available filters :  
> [CACHE] cache  
> [COMP] compression  
> [FCGI] fcgi-app  
> [OT] opentracing  
> [SPOE] spoe  
> [TRACE] trace

> ~# cat /etc/haproxy/haproxy.cfg  
> global  
> log /dev/log local0  
> chroot /var/lib/haproxy  
> pidfile /var/run/haproxy.pid  
> stats socket /run/haproxy/admin.sock mode 660 level admin  
> stats timeout 30s  
> user haproxy  
> group haproxy  
> daemon
> 
> defaults  
> mode http  
> log global  
> option httpslog  
> timeout connect 240s  
> timeout client 1200s  
> timeout server 1200s
> 
> frontend fe-443  
> bind \*:443 ssl crt /etc/haproxy/certs/  
> acl my-acl hdr(host) -i -m beg [example.org](http://example.org)  
> use\_backend my-backend-1 if my-acl  
> capture request header Host len 50  
> capture request header user-agent len 150
> 
> frontend fe-5432  
> bind \*:5432  
> mode tcp  
> option tcplog  
> use\_backend my-backend-2
> 
> backend my-backend-1  
> mode http  
> server my-server-1 any-ip:any-port
> 
> backend my-backend-2  
> mode tcp  
> server my-server-2 any-ip:any-port

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [August 29, 2023, 10:13am UTC](https://discourse.haproxy.org/t/haproxy-suddenly-stops-forwarding/8976/4 "2023-08-29T10:13:20Z")

</div>

Your not specifying maxconn, so haproxy picks its own value.

You also have huge timeouts (20 minutes), so it could be that connections pile up until all maxconn slots are filled.

---

<div class="post-metadata">

**Author:** ![bladnor](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.haproxy.org/bladnor/32/2023_2.png) [@bladnor](https://discourse.haproxy.org/u/bladnor)\
**Post date:** [August 29, 2023, 4:24pm UTC](https://discourse.haproxy.org/t/haproxy-suddenly-stops-forwarding/8976/5 "2023-08-29T16:24:41Z")

</div>

Thanks. For now I have set the global one to 500 and the per server to 20. I’m curious to see how that will work out.