Haproxy stops working Connect() failed for backend local address already in use

Help!
1

I am running HAproxy package in pfsense (HyperV) and I am facing a strange issue.
Every few days or twice a day haproxy fails to forward o backends. The backend start to go randomly up and down even though are on local lan and have enough resources . Ping is ok and also if i use curl from console to the back end works ok.
From logs i see this message:
pfSense01 haproxy[75001]: Connect() failed for backend IIS: local address already in use.

At first i thought it was port exchaustion problem so i added multiple ip to reach backend,but the issue remains.

Restarting HAprox does not fix the problem,only restarting full pfsense works.
Im using pfsense CE 2.7.2-RELEASE
haproxy: 2.8.3-86e043a

And this is my config
global
maxconn 120000
log /var/run/log local0 err
stats socket /tmp/haproxy.socket level admin expose-fd listeners
uid 80
gid 80
nbthread 8
hard-stop-after 30s
chroot /tmp/haproxy_chroot
daemon
server-state-file /tmp/haproxy_server_state
#cpu-map auto:1/1-6 0-5
ssl-default-bind-options no-sslv3
ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES128-SHA256:AES128-SHA:AES256-GCM-SHA384:AES256-SHA256:AES256-SHA:DHE-DSS-AES128-SHA:DES-CBC3-SHA
defaults
balance roundrobin
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
option redispatch
option contstats
timeout connect 5000
timeout client 50000
timeout server 50000
timeout tunnel 1800s

listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats refresh 10
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000

frontend Porta_80-177
bind WAN:80 name WAN:80
mode http
log global
option socket-stats
option dontlognull
option dontlog-normal
option http-server-close
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
default_backend Porta_80_ipvANY

frontend RABBITMQ-181
bind WAN:5672 name WAN:5672 transparent
mode tcp
log global
option dontlognull
timeout client 30000
#source 0.0.0.0 usesrc clientip
default_backend RABBIT_MQ_ipvANY

frontend Porta_SSL_ALL
bind 0.0.0.0:443 name 0.0.0.0:443 ssl crt-list /var/etc/haproxy/Porta_SSL_ALL.crt_list
mode http
log global
option socket-stats
option dontlognull
option dontlog-normal
option log-separate-errors
option httplog
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl mycerts var(txn.txnhost) -m end -i mydomain.com
acl mycerts var(txn.txnhost) -m end -i mydomain1.com
acl mycerts var(txn.txnhost) -m end -i mydomain2.com
acl aclcrt_Porta_SSL_ALL var(txn.txnhost) -m reg -i ^([^.]).mydomain1.com(:([0-9]){1,5})?$
acl aclcrt_Porta_SSL_ALL var(txn.txnhost) -m reg -i ^mydomain1.com(:([0-9]){1,5})?$
acl aclcrt_Porta_SSL_ALL var(txn.txnhost) -m reg -i ^([^.]).mydomain2.com(:([0-9]){1,5})?$
acl aclcrt_Porta_SSL_ALL var(txn.txnhost) -m reg -i ^mydomain2.com(:([0-9]){1,5})?$
acl aclcrt_Porta_SSL_ALL var(txn.txnhost) -m reg -i ^([^.]*).mydomain.com(:([0-9]){1,5})?$
acl aclcrt_Porta_SSL_ALL var(txn.txnhost) -m reg -i ^mydomain.com(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
use_backend Porta_80_ipvANY if mycerts aclcrt_Porta_SSL_ALL

frontend RABBITMQ-local
bind 192.168.100.100:5672 name 192.168.100.100:5672 transparent
mode tcp
log global
option dontlognull
timeout client 30000
#source 0.0.0.0 usesrc clientip
default_backend RABBIT_MQ_ipvANY

frontend IP-178
bind wanip2:80 name wanip2:80
mode http
log global
option dontlognull
option dontlog-normal
option http-server-close
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
default_backend Porta_80_ipvANY

backend Porta_80_ipvANY
mode http
id 100
log global
option log-health-checks
http-check send meth OPTIONS ver HTTP/1.1\r\nHost:\ haproxy.mydomain.com:80\r\nAccept:\ /
balance roundrobin
timeout connect 30000
timeout server 30000
retries 3
load-server-state-from-file global
option httpchk
server IIS2_1 192.168.100.2:80 id 101 check inter 15000 source 192.168.100.14:1025-65000
server IIS2_2 192.168.100.2:80 id 102 check inter 15000 source 192.168.100.15:1025-65000
server IIS2_3 192.168.100.2:80 id 103 check inter 15000 source 192.168.100.16:1025-65000

backend RABBIT_MQ_ipvANY
mode tcp
id 104
log global
balance leastconn
timeout connect 30000
timeout server 30000
retries 3
load-server-state-from-file global
server rabbit_01 192.168.100.1:5672 id 105 check inter 5000
server rabbit_02 192.168.100.2:5672 id 106 check inter 5000
server rabbit_03 192.168.100.3:5672 id 107 check inter 5000

Any help would be appreciated

2

I assume we are talking about backend Porta_80_ipvANY.

Leave only a single backend server and make sure you do not specify a source IP and more importantly that you do not specify a source port range. Specifying a soure port range is dangerous and slow in a multi threaded enviroment like this. Source port selection should be done by the kernel instead.

If it happens again, check netstat -nt or ss -s and ss -l output.

3

i already had only one back end without source ip and port. but state table reached 65k and we had problems. when i added multiple source the state table increased. but the issue persisted.

we have like 10k-20k active connections(like websockets) and like 200-300req/s

The strange thing is that it happens randomly. it can happen during peak times or at 4 Am when there like 1k connections

We have also a lot of Time_wait anf Fin2 connections but this also does not explain when it crashes at 4 am

I dissabled carp ip and use only ip of interfaces and see if it fixes anything

I have used haproxy for years (installed in ubuntu) and did not have these problems.

4

Considering that a haproxy restart does not fix the situation, but you have to reboot, likely you are exhausting fw/pf sessions.

5

Pf state table is far less than its limit. when it happens during the night state table is like 14k entries and my pfsense state limit is 2M.

6

Well something is exhaustion state on the OS, if it isn’t pf than it could just be the actual sockets.