Proxied IIS site delay issues


#1

I have set up a single haproxy server to load balance hundreds of sites running on two IIS servers. It works but after a while some client ip’s have long delays connecting to sites. If I clear arp on both of the IIS servers with “arp -d *” the issue goes away for a while.

I have set the ArpCacheSize to 8000 (hex) in the registry on both servers and that has helped but there is still an issue. Is there a garbage collecting timeout that also needs to be reduced in the registry? I have also tuned the linux kernel on the haproxy server for raising the arp-related settings:

net.ipv4.route.gc_thresh = 524288
net.ipv4.neigh.default.gc_thresh1 = 8192
net.ipv4.neigh.default.gc_thresh2 = 16384
net.ipv4.neigh.default.gc_thresh3 = 32768

Any ideas on a permanent fix besides flushing arp on the WIndows 2012 backend servers?

Thanks.


#2

This is not related to HAproxy, but what I would assume is a network miss configuration.

How many hosts do you have in your LAN? Do you see remote-hosts (like remote clients from the Internet) in your ARP list?

You have to properly configure all servers and hosts with a default-gateway (a next-hop), DON’T YOU EVER point a default-route to a interface without specifying a next-hop when using regular Ethernet interfaces (as opposed to point-to-point interfaces like ATM or PPP).

This is valid for all Operating Systems.

Otherwise your kernel will ask for the MAC address of every single destination IP via ARP request, and some network vendors (Cisco) by default will reply with their MAC, hiding this bogus configuration.


#3

Thanks for the info. We do not have very many hosts in the LAN but we are using transparent client ip in Haproxy. The backend IIS servers are using the haproxy server as their default gateway and needed for transparency to work. The haproxy server has a firewall in front of it that it uses as its default gateway.

So I am not certain what complication the transparency adds but we do not see many entries in the IIS servers’ arp tables even though the arp clearing on them helps the problem.


#4

When you have the problem, please provide from an administrative prompt the following outputs (Windows):

route print
ipconfig /all
arp -a

and Linux:

route -n
arp -n

#5

Thanks Lucas,
What I did today was remove transparency off all the sites and kernel settings and the haproxy server worked perfectly. It looks like an issue with the iptables mangling of the packets. I have followed the documented way to do transparency and that worked but it leads to the issue we are having.

We are using the haproxy server with only one ethernet connection (single arm) and we are now considering if we need to split the connections to have one for the frontend network and one for the backend. Maybe having all the traffic through one physical connection causes the problem?


#6

It should not cause any issues. If you want to find out and troubleshoot further, you will have to provide the informations I requested in my last post.


#7

Today I switched the haproxy server to a two-arm approach keeping client ip transparency and it looks like the issue has been corrected. I agree that the single-arm config should have worked but there was a problem that was not obvious.

Thanks.


#8

So the two-arm approach does not seem to work through backend Windows server reboots. I have switched back to single-arm haproxy and recreated the issue. With transparent proxying some ip addresses are prevented access to the sites but others are not. Thanks for any help you can provide.

HAPROXY SERVER:

[root@nsproxy]# arp -n
Address HWtype HWaddress Flags Mask Iface
192.168.5.17 ether 00:15:5d:02:7d:03 C eth0
192.168.4.3 ether 00:15:5d:02:7d:03 C eth0
192.168.4.4 ether 00:15:5d:02:7d:07 C eth0
192.168.4.17 ether 00:15:5d:02:7d:07 C eth0
192.168.4.1 ether 2c:21:72:c6:39:8e C eth0
[root@nsproxy]# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
169.254.0.0 0.0.0.0 255.255.0.0 U 1002 0 0 eth0
192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 eth0
0.0.0.0 192.168.4.1 0.0.0.0 UG 0 0 0 eth0

ONE OF THE BACKEND IIS SERVERS:

arp -a

Interface: 192.168.4.3 — 0xf
Internet Address Physical Address Type
192.168.4.1 2c-21-72-c6-39-8e dynamic
192.168.4.5 00-15-5d-02-7d-02 dynamic
192.168.4.7 00-15-5d-02-7d-01 dynamic
192.168.4.9 00-15-5d-02-7d-04 dynamic
192.168.4.254 00-15-5d-01-75-03 dynamic
192.168.5.255 ff-ff-ff-ff-ff-ff static
224.0.0.22 01-00-5e-00-00-16 static
224.0.0.252 01-00-5e-00-00-fc static

route print

===========================================================================
Interface List
15…00 15 5d 02 7d 03 …Microsoft Hyper-V Network Adapter #2
1…Software Loopback Interface 1
13…00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter

IPv4 Route Table

Active Routes:
Network Destination Netmask Gateway Interface Metric
0.0.0.0 0.0.0.0 192.168.4.254 192.168.4.3 261
127.0.0.0 255.0.0.0 On-link 127.0.0.1 306
127.0.0.1 255.255.255.255 On-link 127.0.0.1 306
127.255.255.255 255.255.255.255 On-link 127.0.0.1 306
192.168.4.0 255.255.254.0 On-link 192.168.4.3 261
192.168.4.3 255.255.255.255 On-link 192.168.4.3 261
192.168.4.18 255.255.255.255 On-link 192.168.4.3 261
192.168.4.20 255.255.255.255 On-link 192.168.4.3 261
192.168.4.21 255.255.255.255 On-link 192.168.4.3 261
192.168.4.23 255.255.255.255 On-link 192.168.4.3 261
192.168.4.27 255.255.255.255 On-link 192.168.4.3 261
192.168.4.117 255.255.255.255 On-link 192.168.4.3 261
192.168.5.17 255.255.255.255 On-link 192.168.4.3 261
192.168.5.25 255.255.255.255 On-link 192.168.4.3 261
192.168.5.27 255.255.255.255 On-link 192.168.4.3 261
192.168.5.29 255.255.255.255 On-link 192.168.4.3 261
192.168.5.31 255.255.255.255 On-link 192.168.4.3 261
192.168.5.33 255.255.255.255 On-link 192.168.4.3 261
192.168.5.35 255.255.255.255 On-link 192.168.4.3 261
192.168.5.37 255.255.255.255 On-link 192.168.4.3 261
192.168.5.39 255.255.255.255 On-link 192.168.4.3 261
192.168.5.41 255.255.255.255 On-link 192.168.4.3 261
192.168.5.43 255.255.255.255 On-link 192.168.4.3 261
192.168.5.45 255.255.255.255 On-link 192.168.4.3 261
192.168.5.47 255.255.255.255 On-link 192.168.4.3 261
192.168.5.49 255.255.255.255 On-link 192.168.4.3 261
192.168.5.51 255.255.255.255 On-link 192.168.4.3 261
192.168.5.53 255.255.255.255 On-link 192.168.4.3 261
192.168.5.55 255.255.255.255 On-link 192.168.4.3 261
192.168.5.57 255.255.255.255 On-link 192.168.4.3 261
192.168.5.59 255.255.255.255 On-link 192.168.4.3 261
192.168.5.61 255.255.255.255 On-link 192.168.4.3 261
192.168.5.63 255.255.255.255 On-link 192.168.4.3 261
192.168.5.65 255.255.255.255 On-link 192.168.4.3 261
192.168.5.67 255.255.255.255 On-link 192.168.4.3 261
192.168.5.69 255.255.255.255 On-link 192.168.4.3 261
192.168.5.71 255.255.255.255 On-link 192.168.4.3 261
192.168.5.73 255.255.255.255 On-link 192.168.4.3 261
192.168.5.75 255.255.255.255 On-link 192.168.4.3 261
192.168.5.77 255.255.255.255 On-link 192.168.4.3 261
192.168.5.79 255.255.255.255 On-link 192.168.4.3 261
192.168.5.81 255.255.255.255 On-link 192.168.4.3 261
192.168.5.83 255.255.255.255 On-link 192.168.4.3 261
192.168.5.85 255.255.255.255 On-link 192.168.4.3 261
192.168.5.87 255.255.255.255 On-link 192.168.4.3 261
192.168.5.89 255.255.255.255 On-link 192.168.4.3 261
192.168.5.91 255.255.255.255 On-link 192.168.4.3 261
192.168.5.93 255.255.255.255 On-link 192.168.4.3 261
192.168.5.95 255.255.255.255 On-link 192.168.4.3 261
192.168.5.97 255.255.255.255 On-link 192.168.4.3 261
192.168.5.99 255.255.255.255 On-link 192.168.4.3 261
192.168.5.101 255.255.255.255 On-link 192.168.4.3 261
192.168.5.103 255.255.255.255 On-link 192.168.4.3 261
192.168.5.105 255.255.255.255 On-link 192.168.4.3 261
192.168.5.107 255.255.255.255 On-link 192.168.4.3 261
192.168.5.109 255.255.255.255 On-link 192.168.4.3 261
192.168.5.111 255.255.255.255 On-link 192.168.4.3 261
192.168.5.113 255.255.255.255 On-link 192.168.4.3 261
192.168.5.115 255.255.255.255 On-link 192.168.4.3 261
192.168.5.117 255.255.255.255 On-link 192.168.4.3 261
192.168.5.119 255.255.255.255 On-link 192.168.4.3 261
192.168.5.121 255.255.255.255 On-link 192.168.4.3 261
192.168.5.123 255.255.255.255 On-link 192.168.4.3 261
192.168.5.125 255.255.255.255 On-link 192.168.4.3 261
192.168.5.127 255.255.255.255 On-link 192.168.4.3 261
192.168.5.129 255.255.255.255 On-link 192.168.4.3 261
192.168.5.131 255.255.255.255 On-link 192.168.4.3 261
192.168.5.133 255.255.255.255 On-link 192.168.4.3 261
192.168.5.135 255.255.255.255 On-link 192.168.4.3 261
192.168.5.137 255.255.255.255 On-link 192.168.4.3 261
192.168.5.139 255.255.255.255 On-link 192.168.4.3 261
192.168.5.141 255.255.255.255 On-link 192.168.4.3 261
192.168.5.143 255.255.255.255 On-link 192.168.4.3 261
192.168.5.145 255.255.255.255 On-link 192.168.4.3 261
192.168.5.147 255.255.255.255 On-link 192.168.4.3 261
192.168.5.149 255.255.255.255 On-link 192.168.4.3 261
192.168.5.151 255.255.255.255 On-link 192.168.4.3 261
192.168.5.153 255.255.255.255 On-link 192.168.4.3 261
192.168.5.155 255.255.255.255 On-link 192.168.4.3 261
192.168.5.157 255.255.255.255 On-link 192.168.4.3 261
192.168.5.159 255.255.255.255 On-link 192.168.4.3 261
192.168.5.161 255.255.255.255 On-link 192.168.4.3 261
192.168.5.163 255.255.255.255 On-link 192.168.4.3 261
192.168.5.165 255.255.255.255 On-link 192.168.4.3 261
192.168.5.167 255.255.255.255 On-link 192.168.4.3 261
192.168.5.169 255.255.255.255 On-link 192.168.4.3 261
192.168.5.171 255.255.255.255 On-link 192.168.4.3 261
192.168.5.173 255.255.255.255 On-link 192.168.4.3 261
192.168.5.175 255.255.255.255 On-link 192.168.4.3 261
192.168.5.177 255.255.255.255 On-link 192.168.4.3 261
192.168.5.179 255.255.255.255 On-link 192.168.4.3 261
192.168.5.181 255.255.255.255 On-link 192.168.4.3 261
192.168.5.183 255.255.255.255 On-link 192.168.4.3 261
192.168.5.185 255.255.255.255 On-link 192.168.4.3 261
192.168.5.187 255.255.255.255 On-link 192.168.4.3 261
192.168.5.189 255.255.255.255 On-link 192.168.4.3 261
192.168.5.191 255.255.255.255 On-link 192.168.4.3 261
192.168.5.193 255.255.255.255 On-link 192.168.4.3 261
192.168.5.195 255.255.255.255 On-link 192.168.4.3 261
192.168.5.197 255.255.255.255 On-link 192.168.4.3 261
192.168.5.199 255.255.255.255 On-link 192.168.4.3 261
192.168.5.201 255.255.255.255 On-link 192.168.4.3 261
192.168.5.203 255.255.255.255 On-link 192.168.4.3 261
192.168.5.205 255.255.255.255 On-link 192.168.4.3 261
192.168.5.207 255.255.255.255 On-link 192.168.4.3 261
192.168.5.209 255.255.255.255 On-link 192.168.4.3 261
192.168.5.211 255.255.255.255 On-link 192.168.4.3 261
192.168.5.213 255.255.255.255 On-link 192.168.4.3 261
192.168.5.215 255.255.255.255 On-link 192.168.4.3 261
192.168.5.217 255.255.255.255 On-link 192.168.4.3 261
192.168.5.219 255.255.255.255 On-link 192.168.4.3 261
192.168.5.221 255.255.255.255 On-link 192.168.4.3 261
192.168.5.223 255.255.255.255 On-link 192.168.4.3 261
192.168.5.225 255.255.255.255 On-link 192.168.4.3 261
192.168.5.227 255.255.255.255 On-link 192.168.4.3 261
192.168.5.229 255.255.255.255 On-link 192.168.4.3 261
192.168.5.231 255.255.255.255 On-link 192.168.4.3 261
192.168.5.233 255.255.255.255 On-link 192.168.4.3 261
192.168.5.235 255.255.255.255 On-link 192.168.4.3 261
192.168.5.237 255.255.255.255 On-link 192.168.4.3 261
192.168.5.239 255.255.255.255 On-link 192.168.4.3 261
192.168.5.241 255.255.255.255 On-link 192.168.4.3 261
192.168.5.243 255.255.255.255 On-link 192.168.4.3 261
192.168.5.245 255.255.255.255 On-link 192.168.4.3 261
192.168.5.247 255.255.255.255 On-link 192.168.4.3 261
192.168.5.249 255.255.255.255 On-link 192.168.4.3 261
192.168.5.251 255.255.255.255 On-link 192.168.4.3 261
192.168.5.253 255.255.255.255 On-link 192.168.4.3 261
192.168.5.255 255.255.255.255 On-link 192.168.4.3 261
192.168.6.0 255.255.254.0 On-link 192.168.4.3 261
192.168.7.29 255.255.255.255 On-link 192.168.4.3 261
192.168.7.31 255.255.255.255 On-link 192.168.4.3 261
192.168.7.33 255.255.255.255 On-link 192.168.4.3 261
192.168.7.35 255.255.255.255 On-link 192.168.4.3 261
192.168.7.37 255.255.255.255 On-link 192.168.4.3 261
192.168.7.39 255.255.255.255 On-link 192.168.4.3 261
192.168.7.41 255.255.255.255 On-link 192.168.4.3 261
192.168.7.43 255.255.255.255 On-link 192.168.4.3 261
192.168.7.45 255.255.255.255 On-link 192.168.4.3 261
192.168.7.47 255.255.255.255 On-link 192.168.4.3 261
192.168.7.49 255.255.255.255 On-link 192.168.4.3 261
192.168.7.51 255.255.255.255 On-link 192.168.4.3 261
192.168.7.53 255.255.255.255 On-link 192.168.4.3 261
192.168.7.55 255.255.255.255 On-link 192.168.4.3 261
192.168.7.57 255.255.255.255 On-link 192.168.4.3 261
192.168.7.59 255.255.255.255 On-link 192.168.4.3 261
192.168.7.61 255.255.255.255 On-link 192.168.4.3 261
192.168.7.63 255.255.255.255 On-link 192.168.4.3 261
192.168.7.65 255.255.255.255 On-link 192.168.4.3 261
192.168.7.67 255.255.255.255 On-link 192.168.4.3 261
192.168.7.69 255.255.255.255 On-link 192.168.4.3 261
192.168.7.71 255.255.255.255 On-link 192.168.4.3 261
192.168.7.73 255.255.255.255 On-link 192.168.4.3 261
192.168.7.75 255.255.255.255 On-link 192.168.4.3 261
192.168.7.255 255.255.255.255 On-link 192.168.4.3 261
224.0.0.0 240.0.0.0 On-link 127.0.0.1 306
224.0.0.0 240.0.0.0 On-link 192.168.4.3 261
255.255.255.255 255.255.255.255 On-link 127.0.0.1 306
255.255.255.255 255.255.255.255 On-link 192.168.4.3 261

Persistent Routes:
Network Address Netmask Gateway Address Metric
0.0.0.0 0.0.0.0 192.168.4.254 Default
0.0.0.0 0.0.0.0 192.168.4.254 Default

IPv6 Route Table

Active Routes:
If Metric Network Destination Gateway
15 261 ::/0 2604:f80:1:4::1
1 306 ::1/128 On-link
15 261 2604:f80:1:4::/64 On-link
15 261 2604:f80:1:4::3/128 On-link
15 261 fe80::/64 On-link
15 261 fe80::11c4:2817:967b:e0dc/128
On-link
1 306 ff00::/8 On-link
15 261 ff00::/8 On-link

Persistent Routes:
If Metric Network Destination Gateway
0 4294967295 ::/0 2604:f80:1:4::1
0 4294967295 ::/0 2604:f80:1:4::1

ipconfig /all

Windows IP Configuration

Host Name . . . . . . . . . . . . : Web01
Primary Dns Suffix . . . . . . . : example.com
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : example.com

Ethernet adapter Ethernet 2:

Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Microsoft Hyper-V Network Adapter #2
Physical Address. . . . . . . . . : 00-15-5D-02-7D-03
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
IPv6 Address. . . . . . . . . . . : 2604:f80:1:4::3(Preferred)
Link-local IPv6 Address . . . . . : fe80::11c4:2817:967b:e0dc%15(Preferred)
IPv4 Address. . . . . . . . . . . : 192.168.4.3(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.254.0
IPv4 Address. . . . . . . . . . . : 192.168.4.18(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.254.0
IPv4 Address. . . . . . . . . . . : 192.168.4.20(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.254.0
IPv4 Address. . . . . . . . . . . : 192.168.5.17(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.254.0
IPv4 Address. . . . . . . . . . . : 192.168.5.25(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.254.0

(MANY MORE ADDRESSES IN .5, and .7)

Default Gateway . . . . . . . . . : 2604:f80:1:4::1
192.168.4.254
DHCPv6 IAID . . . . . . . . . . . : 352327005
DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-1D-FC-05-58-00-15-5D-83-67-01
DNS Servers . . . . . . . . . . . : 2604:f80:1:4::9
192.168.4.9
8.8.8.8
NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{FC3BFB19-E047-44E6-AA55-F58972CA673A}:

Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Description . . . . . . . . . . . : Microsoft ISATAP Adapter
Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes