[SOLVED] HAProxy 1.8 check failing 50% of the times

Help!
1

Hi,
Last weekend we had an outage due to the backend for one VIP going down. Everything I could see in the logs is that there were no backends available, and in Fortigate I can see that 50% of the connection ( the healthchecks) where refused.

image

So, on the left side you have a failed healthcheck and on the right side a good one.

Any idea why HAProxy could had start failing without any configuration changed being applied? After a machine reboot it came back fine. It was also happening on the failover instance of HAProxy.

I realized that sni ssl_fc_sni was not configured in the backend , but I am surprissed that we didn’t ran into problems for many weeks and suddenly it happened.

Here is the backend config:

server web1 10.11.1.1:443 maxconn 8 check inter 5s fall 4 rise 3 ssl verify none

Thanks.

2

This is from the error log

local0.alert: Jun 29 18:32:52 haproxy[88975]: Server ext_web_ssl_back/web1 is DOWN, reason: Layer4 connection problem, info: "Connection error during SSL handshake (Connection refused)", check duration: 0ms. 1 active and 0 backup servers left. 5 sessions active, 0 requeued, 0 remaining in queue.
local0.alert: Jun 29 18:32:58 haproxy[88975]: Server ext_web_ssl_back/web1 is DOWN, reason: Layer4 connection problem, info: "Connection error during SSL handshake (Connection refused)", check duration: 0ms. 0 active and 0 backup servers left. 8 sessions active, 0 requeued, 0 remaining in queue.
local0.emerg: Jun 29 18:32:58 haproxy[88975]: backend ext_web_ssl_back has no server available!

But as I said, Those backends where working for another LB with the same setup, almost (Except sni ssl_fc_sni)

3

Why do you think you need that?

Meaning you have a problem between the Fortigate and your backend servers. Haproxy is not involved, instead the Fortigate never got any response.

4

Regarding sni ssl_fc_sni I was using it in the other LB because the backends are serving multiple different websites.

And about the Fortinet, the network guy said that the firewall is not blocking anything it is just not receiving the reply back from the server, but this is weird since the same servers are reporting properly to a different LB.

5

But the Fortigate Firewall is between haproxy and the backend server, right? So if the Fortigate Firewall does not receive a reply from the backend servers, haproxy cannot either.

This is an indication that the backend servers are blocking a specific load-balancer IP, imho. Maybe you have some iptables/conntrack issues or something like that on the backend servers?

6

we found the problem… The backends had fail2ban enabled and were adding rules that blocked the response back to the firewall.

Thanks for your help!

1 Like