webber
November 2, 2016, 2:00pm
1
I have set up a single haproxy server to load balance hundreds of sites running on two IIS servers. It works but after a while some client ip’s have long delays connecting to sites. If I clear arp on both of the IIS servers with “arp -d *” the issue goes away for a while.
I have set the ArpCacheSize to 8000 (hex) in the registry on both servers and that has helped but there is still an issue. Is there a garbage collecting timeout that also needs to be reduced in the registry? I have also tuned the linux kernel on the haproxy server for raising the arp-related settings:
net.ipv4.route.gc_thresh = 524288
Any ideas on a permanent fix besides flushing arp on the WIndows 2012 backend servers?
Thanks.
This is not related to HAproxy, but what I would assume is a network miss configuration.
How many hosts do you have in your LAN? Do you see remote-hosts (like remote clients from the Internet) in your ARP list?
You have to properly configure all servers and hosts with a default-gateway (a next-hop), DON’T YOU EVER point a default-route to a interface without specifying a next-hop when using regular Ethernet interfaces (as opposed to point-to-point interfaces like ATM or PPP).
This is valid for all Operating Systems.
Otherwise your kernel will ask for the MAC address of every single destination IP via ARP request, and some network vendors (Cisco) by default will reply with their MAC, hiding this bogus configuration.
webber
November 4, 2016, 9:45pm
3
Thanks for the info. We do not have very many hosts in the LAN but we are using transparent client ip in Haproxy. The backend IIS servers are using the haproxy server as their default gateway and needed for transparency to work. The haproxy server has a firewall in front of it that it uses as its default gateway.
So I am not certain what complication the transparency adds but we do not see many entries in the IIS servers’ arp tables even though the arp clearing on them helps the problem.
When you have the problem, please provide from an administrative prompt the following outputs (Windows):
route print
ipconfig /all
arp -a
and Linux:
route -n
arp -n
webber
November 7, 2016, 5:03pm
5
Thanks Lucas,
We are using the haproxy server with only one ethernet connection (single arm) and we are now considering if we need to split the connections to have one for the frontend network and one for the backend. Maybe having all the traffic through one physical connection causes the problem?
It should not cause any issues. If you want to find out and troubleshoot further, you will have to provide the informations I requested in my last post.
webber
November 7, 2016, 8:05pm
7
Today I switched the haproxy server to a two-arm approach keeping client ip transparency and it looks like the issue has been corrected. I agree that the single-arm config should have worked but there was a problem that was not obvious.
Thanks.
webber
November 8, 2016, 5:21pm
8
So the two-arm approach does not seem to work through backend Windows server reboots. I have switched back to single-arm haproxy and recreated the issue. With transparent proxying some ip addresses are prevented access to the sites but others are not. Thanks for any help you can provide.
HAPROXY SERVER:
[root@nsproxy]# arp -n
ONE OF THE BACKEND IIS SERVERS:
arp -a
Interface: 192.168.4.3 — 0xf
route print
===========================================================================#2
IPv4 Route Table
Active Routes:
Persistent Routes:
IPv6 Route Table
Active Routes:
Persistent Routes:
ipconfig /all
Windows IP Configuration
Host Name . . . . . . . . . . . . : Web01example.com example.com
Ethernet adapter Ethernet 2:
Connection-specific DNS Suffix . :#2
(MANY MORE ADDRESSES IN .5, and .7)
Default Gateway . . . . . . . . . : 2604:f80:1:4::1
Tunnel adapter isatap.{FC3BFB19-E047-44E6-AA55-F58972CA673A}:
Media State . . . . . . . . . . . : Media disconnected