Backend servers are down due to Layer4 connection problem

Help!
1

I have HA LB configured with 3 backend servers . From the logs, i see that one of the nodes are going down frequently and coming up . Error message is like below.

eb 16 08:38:36 localhost haproxy[4303]: Server xxxxxxxx is DOWN, reason: Layer4 connection problem, info: “Connection refused”, check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

haproxy version is as below.

HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau willy@haproxy.org

backend configuration is as below,

backend nodes
mode http
balance roundrobin #Balance algorithm
option forwardfor

option httpchk HEAD / HTTP/1.1\r\nHost:\ localhost #Check the server application is up and healty - 200 status code

server servr1 xx.xx.xx.xx:30010 check                 #Nginx1
server server2 xx.xx.xx.xx:30010 check                 #Nckend nodes
server server2 xx.xx.xx.xx:30010 check

Please help to why these connection refused and time out are occurring… I have tested on backend servers and i can access the app without any issues directly.

2

It would be useful if you could perform some simple, low level tests on the host running HAProxy, towards the backend servers. Pings, ncat connection tests, traceroute etc. This could relate to any number of many network issues that should be discounted before looking at HAProxy. Comparisons of output between when things are working and not would also be useful.

3

thanks for the reply… below is the telnet result for one of backend server… It is weird …different results to continuous test…

says sometimes connected and some times no route to the host.

4

Hey, you’re welcome.

It very much looks like you have an underlying network issue, that or a very unstable backend host (or application). Assuming the latter isn’t true as you’ve suggested you don’t have issues when connecting ‘on-host’ you need to take a look at everything along the network path between the two, starting with the network interface on the host running HAProxy. Moving onto routing and whatever network devices are sat between the two. It might also be worth checking there is nothing rate limiting or similar (iptables, in the app wherever).

It’s hard to be more precise with no view of what your infrastructure looks like.