I’ve used HAProxy for a long time for a simple load balancer in front of client web servers with zero unsolvable issues through multiple upgrades and previous hardware replacement. I am in the process of making significant infrastructure upgrades at a client and moving off of older CentOS versions and onto new hardware and (in this case) RHEL 9, with HAProxy 2.4.22 out of the RHEL repos.
The config check passes on my config file. The service starts and is running, with netstat showing the ports bound to haproxy and listening. I’ve modified the firewall to listen on 80 and 443. I (believe) I’ve resolved all SELINUX issues out of the audit log.
But for some reason, when I flip my firewall to forward packets to the new box, it’s as if they are not picked up properly and balanced across the backend. Packet trace on the firewall shows the packets to be forwarded, just like they do for the old IP, but nothing ever comes back. I haven’t dug into a tcpdump fully yet, but it doesn’t look to me like the packets are showing up (I’m having to learn tcpdump as I go here).
Here’s where it gets mind-blowing to me. If I use curl to hit the frontend IP from another server on the same network, I get a response back from the backend Apache hosts, just like I expect. Somewhere between the firewall and the HAProxy service on the new host, something is getting lost it seems to me, but I for the life of me cannot figure out where, why or how, so I thought I’d ask here. I’ve removed all complexity from the network config (no bonding), the fact I can hit the IP internally and get the content says the switch has the right vlan assigned, and hitting it with curl says the config file is at least workable if not yet maybe optimized. I’m just kind of boggled where to go next.
Is there any SELINUX magic that now must be done on RHEL 9 with HAProxy 2.4 that might not be caught in the audit log? Is there something the multitude of tutorials for RHEL 8/9 out there that just basically all say “dnf install haproxy and build your config file” are missing?
Is there anyone out there with some wisdom to share on troubleshooting this further? I really need to get off this older hardware and versions for obvious security and longevity purposes, but can’t until I get the service listening and responding properly on the new box.
Sorry if this is TLDR.