jbuk
February 7, 2019, 9:37pm
1
Hi, I’m running haproxy 1.5.18 on CentOS and it is load balancing a couple of Windows Server 2016 machines. I’m hitting an issue whereby if I try and run a vulnerability scan e.g. from Qualys, after a while the Windows Server becomes inaccessible to the HAProxy. I can’t ping it or access websites from the haproxy but connections to it are available from other devices. Then, when the scans stop, and after a while the connectivity returns. The config is similar to this:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 100000
user hap
group hap
daemon
stats socket /var/lib/haproxy/stats
defaults
log global
option dontlognull
option http-server-close
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 60m
timeout server 60m
timeout http-keep-alive 10s
timeout check 10s
maxconn 5000
frontend 1.2.3.4-http
bind 1.2.3.4:80
reqadd X-Forwarded-Proto:\ http
mode http
option forwardfor except 127.0.0.1/8
option httplog
option http-server-close
timeout http-request 10s
timeout http-keep-alive 10s
use_backend scan-test
frontend 1.2.3.4-https
bind 1.2.3.4:443 ssl crt /etc/certs/cert1.pem
reqadd X-Forwarded-Proto:\ https
mode http
option forwardfor except 127.0.0.1/8
option httplog
option http-server-close
timeout http-request 10s
timeout http-keep-alive 10s
use_backend scan-test
backend scan-test
balance leastconn
cookie serverid insert
mode http
option httpchk GET /pagemonitor.txt
http-check expect rstatus ([2-3][0-9][0-9])
server server1_http 192.168.1.101:80 cookie 1 check inter 30000
server server2_http 192.168.1.102:80 cookie 2 check inter 30000
Has anyone experienced this issue before?
jbuk
February 7, 2019, 10:13pm
3
Feb 7 21:52:26 localhost haproxy[15100]: 4.3.2.1:52142 [07/Feb/2019:21:52:20.044] 1.2.3.4-https-scantest/1: SSL handshake failure
Your backend server stops responding when 50 sessions are active, the issue is there:
Feb 7 21:54:54 localhost haproxy[15100]: Server scan-test/web01_http is DOWN, reason: Layer4 timeout, check duration: 10000ms. 0 active and 0 backup servers left. 50 sessions active, 0 requeued, 0 remaining in queue.
Feb 7 21:54:54 localhost haproxy[15100]: backend scan-test has no server available!
You will have to figure out why your backend server can only handle 50 sessions.
jbuk
February 8, 2019, 7:16pm
5
Thanks for the response. I also have another haproxy which is connecting to the same set of Windows servers and I’m seeing an extra 200 sessions from that one (there are more backends on that haproxy). So thats a max of around 250 sessions. Does that seem low to you? It’s odd because when the haproxy that I test with the scan hits the limit, the other one is still serving requests without issue.
I suggest you try to access the backend server from the box where haproxy runs while it doesn’t work. You may have some intermediate devices (Firewalls) or the server itself that rejects the IP address of haproxy due to excessive queries.
I can only guess at this point, but for haproxy, the server is down.
jbuk
February 8, 2019, 8:17pm
7
I have tried this and it’s strange as there is no connectivity there at all. I cannot ping the backend server from the haproxy when it sees it as down (would this be expected?!). If I run a continuous ping it takes 10 minutes to start responding again each time which must be a significant figure. I’m trying to work out where this 10 minute timeout exists.
It probably means that some anti DDoS protection is kicking in somewhere. Maybe on the server, maybe on intermediate devices.