HAProxy backends is flapping DOWN and UP

Help!
1

I configured the haproxy with 2 backends for ports 80 and 443.
Here is the part for these:-
defaults
mode tcp

========================
frontend secure
bind *:443
default_backend secure

frontend insecure
bind *:80
default_backend insecure

=================
And Backends:-
backend secure
balance source
server <IP_Address>:443 check
server <IP_Address>:443 check

backend insecure
balance source
server <IP_Address>:80 check
server <IP_Address>:80 check
==============================

[root@ocplbnext01 ~]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; preset: disabled)
Active: active (running) since Wed 2024-08-21 15:30:16 EEST; 12h ago
Process: 100653 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -f $CFGDIR -c -q $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 100657 (haproxy)
Tasks: 9 (limit: 203835)
Memory: 14.2M
CPU: 7min 44.875s
CGroup: /system.slice/haproxy.service
├─100657 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid
└─100660 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d -p /run/haproxy.pid
Aug 22 03:28:34 ocplbnext01 haproxy[100660]: [WARNING] (100660) : Server insecure/ is DOWN, reason: Layer4 connection problem, info: “Connection refused”, check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining>
Aug 22 03:28:34 ocplbnext01 haproxy[100660]: [ALERT] (100660) : backend ‘insecure’ has no server available!
Aug 22 03:28:45 ocplbnext01 haproxy[100660]: [WARNING] (100660) : Server secure/ is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Aug 22 03:28:46 ocplbnext01 haproxy[100660]: [WARNING] (100660) : Server insecure/ is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Aug 22 03:32:53 ocplbnext01 haproxy[100660]: [WARNING] (100660) : Server secure/ is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Aug 22 03:32:54 ocplbnext01 haproxy[100660]: [WARNING] (100660) : Server insecure/ is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Aug 22 03:33:33 ocplbnext01 haproxy[100660]: [WARNING] (100660) : Server secure/ is DOWN, reason: Layer4 connection problem, info: “Connection refused”, check duration: 0ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining i>
Aug 22 03:33:34 ocplbnext01 haproxy[100660]: [WARNING] (100660) : Server insecure/ is DOWN, reason: Layer4 connection problem, info: “Connection refused”, check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining>
Aug 22 03:33:46 ocplbnext01 haproxy[100660]: [WARNING] (100660) : Server secure/ is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Aug 22 03:33:46 ocplbnext01 haproxy[100660]: [WARNING] (100660) : Server insecure/ is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
~
~

2

Well since the log says connection is refused to the server, I’d suggest looking at the server, perhaps it will tell you something more about this.

What makes you believe the issue could be on the haproxy side?

3

Hello,
Actually since I am a fresh to HAProxy so I am trying to get info about the error. I expected it is related to haproxy since I tried to other nodes under secure and insecure backends and same error is returning.

4

Well the logs are saying there is a LAYER 4 problem. (That refers to the OSI layer 4)
In turn that refers to the transport layer. In short, it means not all tcp traffic is reliably reaching your server.

I can only guess what the problem is and I’m no network or HAPROXY expert. but it sounds to me like there might be a bad cable / ethernet card or switch having issues.

But not knowing anything about your infrastructure, I can only guess.
If it happens pretty often you can verify on the haproxy if a normal ping to the server will reliably reach the server, if that fails too, you can leave haproxy out of the equation and start troubleshooting the network.