# Haproxy stops working Connect() failed for backend local address already in use

**URL:** <https://discourse.haproxy.org/t/haproxy-stops-working-connect-failed-for-backend-local-address-already-in-use/10194>\
**Category:** Help!\
**Created:** [September 7, 2024, 10:28pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-connect-failed-for-backend-local-address-already-in-use/10194 "2024-09-07T22:28:41Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![riccardo123](https://avatars.discourse-cdn.com/v4/letter/r/ce73a5/32.png) [@riccardo123](https://discourse.haproxy.org/u/riccardo123)\
**Post date:** [September 7, 2024, 10:28pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-connect-failed-for-backend-local-address-already-in-use/10194/1 "2024-09-07T22:28:41Z")

</div>

I am running HAproxy package in pfsense (HyperV) and I am facing a strange issue.  
Every few days or twice a day haproxy fails to forward o backends. The backend start to go randomly up and down even though are on local lan and have enough resources . Ping is ok and also if i use curl from console to the back end works ok.  
From logs i see this message:  
pfSense01 haproxy[75001]: Connect() failed for backend IIS: local address already in use.

At first i thought it was port exchaustion problem so i added multiple ip to reach backend,but the issue remains.

Restarting HAprox does not fix the problem,only restarting full pfsense works.  
Im using pfsense CE **2.7.2-RELEASE**  
haproxy: 2.8.3-86e043a

And this is my config  
global  
maxconn 120000  
log /var/run/log local0 err  
stats socket /tmp/haproxy.socket level admin expose-fd listeners  
uid 80  
gid 80  
nbthread 8  
hard-stop-after 30s  
chroot /tmp/haproxy\_chroot  
daemon  
server-state-file /tmp/haproxy\_server\_state  
#cpu-map auto:1/1-6 0-5  
ssl-default-bind-options no-sslv3  
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:DHE-DSS-AES128-SHA:DES-CBC3-SHA  
defaults  
balance roundrobin  
log global  
mode http  
option httplog  
option dontlognull  
option forwardfor  
option http-server-close  
option redispatch  
option contstats  
timeout connect 5000  
timeout client 50000  
timeout server 50000  
timeout tunnel 1800s

listen HAProxyLocalStats  
bind 127.0.0.1:2200 name localstats  
mode http  
stats enable  
stats refresh 10  
stats admin if TRUE  
stats show-legends  
stats uri /haproxy/haproxy\_stats.php?haproxystats=1  
timeout client 5000  
timeout connect 5000  
timeout server 5000

frontend Porta\_80-177  
bind WAN:80 name WAN:80  
mode http  
log global  
option socket-stats  
option dontlognull  
option dontlog-normal  
option http-server-close  
option forwardfor  
acl https ssl\_fc  
http-request set-header X-Forwarded-Proto http if !https  
http-request set-header X-Forwarded-Proto https if https  
timeout client 30000  
default\_backend Porta\_80\_ipvANY

frontend RABBITMQ-181  
bind WAN:5672 name WAN:5672 transparent  
mode tcp  
log global  
option dontlognull  
timeout client 30000  
#source 0.0.0.0 usesrc clientip  
default\_backend RABBIT\_MQ\_ipvANY

frontend Porta\_SSL\_ALL  
bind 0.0.0.0:443 name 0.0.0.0:443 ssl crt-list /var/etc/haproxy/Porta\_SSL\_ALL.crt\_list  
mode http  
log global  
option socket-stats  
option dontlognull  
option dontlog-normal  
option log-separate-errors  
option httplog  
option http-keep-alive  
option forwardfor  
acl https ssl\_fc  
http-request set-header X-Forwarded-Proto http if !https  
http-request set-header X-Forwarded-Proto https if https  
timeout client 30000  
acl mycerts var(txn.txnhost) -m end -i [mydomain.com](http://mydomain.com)  
acl mycerts var(txn.txnhost) -m end -i [mydomain1.com](http://mydomain1.com)  
acl mycerts var(txn.txnhost) -m end -i [mydomain2.com](http://mydomain2.com)  
acl aclcrt\_Porta\_SSL\_ALL var(txn.txnhost) -m reg -i ^([^.]_).mydomain1.com(:([0-9]){1,5})?$  
acl aclcrt\_Porta\_SSL\_ALL var(txn.txnhost) -m reg -i ^mydomain1.com(:([0-9]){1,5})?$  
acl aclcrt\_Porta\_SSL\_ALL var(txn.txnhost) -m reg -i ^([^.]_).mydomain2.com(:([0-9]){1,5})?$  
acl aclcrt\_Porta\_SSL\_ALL var(txn.txnhost) -m reg -i ^mydomain2.com(:([0-9]){1,5})?$  
acl aclcrt\_Porta\_SSL\_ALL var(txn.txnhost) -m reg -i ^([^.]\*).mydomain.com(:([0-9]){1,5})?$  
acl aclcrt\_Porta\_SSL\_ALL var(txn.txnhost) -m reg -i ^mydomain.com(:([0-9]){1,5})?$  
http-request set-var(txn.txnhost) hdr(host)  
use\_backend Porta\_80\_ipvANY if mycerts aclcrt\_Porta\_SSL\_ALL

frontend RABBITMQ-local  
bind 192.168.100.100:5672 name 192.168.100.100:5672 transparent  
mode tcp  
log global  
option dontlognull  
timeout client 30000  
#source 0.0.0.0 usesrc clientip  
default\_backend RABBIT\_MQ\_ipvANY

frontend IP-178  
bind wanip2:80 name wanip2:80  
mode http  
log global  
option dontlognull  
option dontlog-normal  
option http-server-close  
option forwardfor  
acl https ssl\_fc  
http-request set-header X-Forwarded-Proto http if !https  
http-request set-header X-Forwarded-Proto https if https  
timeout client 30000  
default\_backend Porta\_80\_ipvANY

backend Porta\_80\_ipvANY  
mode http  
id 100  
log global  
option log-health-checks  
http-check send meth OPTIONS ver HTTP/1.1\r\nHost:\ haproxy.mydomain.com:80\r\nAccept:\ _/_  
balance roundrobin  
timeout connect 30000  
timeout server 30000  
retries 3  
load-server-state-from-file global  
option httpchk  
server IIS2\_1 192.168.100.2:80 id 101 check inter 15000 source 192.168.100.14:1025-65000  
server IIS2\_2 192.168.100.2:80 id 102 check inter 15000 source 192.168.100.15:1025-65000  
server IIS2\_3 192.168.100.2:80 id 103 check inter 15000 source 192.168.100.16:1025-65000

backend RABBIT\_MQ\_ipvANY  
mode tcp  
id 104  
log global  
balance leastconn  
timeout connect 30000  
timeout server 30000  
retries 3  
load-server-state-from-file global  
server rabbit\_01 192.168.100.1:5672 id 105 check inter 5000  
server rabbit\_02 192.168.100.2:5672 id 106 check inter 5000  
server rabbit\_03 192.168.100.3:5672 id 107 check inter 5000

Any help would be appreciated

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [September 8, 2024, 7:51am UTC](https://discourse.haproxy.org/t/haproxy-stops-working-connect-failed-for-backend-local-address-already-in-use/10194/2 "2024-09-08T07:51:27Z")

</div>

> [@riccardo123](#):
>
> At first i thought it was port exchaustion problem so i added multiple ip to reach backend,but the issue remains.

I assume we are talking about `backend Porta_80_ipvANY`.

Leave only a single backend server and make sure you do not specify a source IP and more importantly that you do not specify a source port range. Specifying a soure port range is dangerous and slow in a multi threaded enviroment like this. Source port selection should be done by the kernel instead.

If it happens again, check `netstat -nt` or `ss -s` and `ss -l` output.

---

<div class="post-metadata">

**Author:** ![riccardo123](https://avatars.discourse-cdn.com/v4/letter/r/ce73a5/32.png) [@riccardo123](https://discourse.haproxy.org/u/riccardo123)\
**Post date:** [September 8, 2024, 10:33am UTC](https://discourse.haproxy.org/t/haproxy-stops-working-connect-failed-for-backend-local-address-already-in-use/10194/3 "2024-09-08T10:33:02Z")

</div>

i already had only one back end without source ip and port. but state table reached 65k and we had problems. when i added multiple source the state table increased. but the issue persisted.

we have like 10k-20k active connections(like websockets) and like 200-300req/s

The strange thing is that it happens randomly. it can happen during peak times or at 4 Am when there like 1k connections

We have also a lot of Time\_wait anf Fin2 connections but this also does not explain when it crashes at 4 am

I dissabled carp ip and use only ip of interfaces and see if it fixes anything

I have used haproxy for years (installed in ubuntu) and did not have these problems.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [September 8, 2024, 8:30pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-connect-failed-for-backend-local-address-already-in-use/10194/4 "2024-09-08T20:30:47Z")

</div>

Considering that a haproxy restart does not fix the situation, but you have to reboot, likely you are exhausting fw/pf sessions.

---

<div class="post-metadata">

**Author:** ![riccardo123](https://avatars.discourse-cdn.com/v4/letter/r/ce73a5/32.png) [@riccardo123](https://discourse.haproxy.org/u/riccardo123)\
**Post date:** [September 9, 2024, 8:23am UTC](https://discourse.haproxy.org/t/haproxy-stops-working-connect-failed-for-backend-local-address-already-in-use/10194/5 "2024-09-09T08:23:16Z")

</div>

Pf state table is far less than its limit. when it happens during the night state table is like 14k entries and my pfsense state limit is 2M.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [September 9, 2024, 12:37pm UTC](https://discourse.haproxy.org/t/haproxy-stops-working-connect-failed-for-backend-local-address-already-in-use/10194/6 "2024-09-09T12:37:20Z")

</div>

Well something is exhaustion state on the OS, if it isn’t pf than it could just be the actual sockets.