On a backend server restart, it always gives 503 error

Help!
1

Hi,
I have set up where there are multiple back end servers. On restart of any backend server, on accessing the page , it gives 503 error as below:-

===========

503 Service Unavailable

No server is available to handle this request.

On haproxy restart it works once again. This does not seem to be an appropriate behavior. I would like to know how HAProxy keeps track of the backend servers. on restart of backend server, how it will know if the service has become up.

Can anybody kindly help?

regards
tridev

2

Hi,

It’s hard to help without some configuration shared. Can you please provide the configuration for an affected backend?

3

Hello Jerome,
Thanks for responding.
Following is my haproxy.cfg. I have tried to restart backend “otntomcat” and “ntsm_serv” and it gives the same problem with both.
Request your help.
thanks
Tridev

global
log stdout format raw local0 debug
defaults
mode http
log global
option http-server-close
option httplog
option dontlognull
option redispatch
option forwardfor
retries 3
backlog 4096
timeout client 20s
timeout client-fin 20s
timeout connect 5s
timeout server 20s
timeout tunnel 15m
timeout http-keep-alive 30s
timeout http-request 10s
timeout queue 5s
timeout tarpit 60s
default-server inter 3s rise 2 fall 3

frontend web_gateway
bind *:7777 ssl crt /usr/local/etc/haproxy/cert.crt
log 127.0.0.1 local0 debug
redirect scheme https if !{ ssl_fc }
http-request add-header X-Forwarded-Proto https
http-request set-query %[query,regsub(:7777,g)]
http-request set-uri %[url,regsub(otntomcat,"$ipv4addr",g)]
acl otntomcat_path path_beg -i /oms1350
acl dojoroot_path path_beg -i /dojoroot
acl p1 path_beg -i /jointroot
acl p2 path_beg -i /jquery
acl p3 path_beg -i /lodash
acl p4 path_beg -i /openlayersroot
acl p5 path_beg -i /search
acl p6 path_beg -i /svgutilsroot
acl p7 path_beg -i /d3root
acl p8 path_beg -i /backbone
acl p9 path_beg -i /Cpb
use_backend otntomcat if otntomcat_path || dojoroot_path || p1 || p2 || p3 || p4 || p5 || p6 || p7 || p8 || p9
acl cas_path path_beg -i /cas
acl session_path path_beg -i /session-manager
use_backend nspos_serv if cas_path || session_path
acl fault_path path_beg -i /FaultManagement
use_backend nspos_serv1 if fault_path
acl int_path path_beg -i /internal
use_backend nspos_serv if int_path
acl ntsm_path path_beg -i /ntsm
use_backend ntsm_serv if ntsm_path
acl systemmonitor_path path_beg -i /systemmonitor
use_backend systemmonitor_serv if systemmonitor_path
default_backend nspos_serv

backend otntomcat
mode http
balance roundrobin
log stdout local0 debug
server serv01 otntomcat:8443 check ssl verify none
http-response replace-header Location (.)(\botntomcat\b)(.)$ \1"ipv4addr"\3 http-response replace-header Location (.*)(\b443\b)(.*) \17777\3
http-response replace-header Location (.)(\b8544\b)(.) \17777\3 http-response replace-header Location (.*)(\b8545\b)(.*) \17777\3
http-response replace-header Location (.)(\b8443\b)(.)$ \17777\3
http-response replace-header Location (.)(\b/oms1350/pages/otn\b)(.) \1:7777/oms1350/pages/otn\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend nspos_serv
mode http
balance roundrobin
log stdout local0 debug
acl p1 path_beg -i /session-manager
acl p2 path_beg -i /cas
acl p3 path_beg -i /oms1350
acl c1 hdr(Location) urlp(service)
server srv01 nspos:443 check ssl verify none
http-response replace-header Location (.)(\botntomcat\b)(.)$ \1"ipv4addr"\3 http-response replace-header Location (.*)(\b\/cas\/login\b)(.*) \1:7777\/cas\/login\3 http-response replace-header Location (.*)(\b\/oms1350\/login\b)(.*) \1:7777\/oms1350\/login\3 http-response replace-header Location (.*)(\b\/session-manager\/login\b)(.*) \1:7777\/session-manager\/login\3 http-response replace-header Location (.*)(\b:7777:7777\/session-manager\/login\b)(.*) \1:7777\/session-manager\/login\3 http-response replace-header Location (.*)(\b8443\b)(.*) \17777\3
http-response replace-header Location (.)(\b/cas/logout\b)(.) \1:7777/cas/logout\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend nspos_serv1
mode http
balance roundrobin
log 127.0.0.1 local0 debug
server srv01 nspos:8544 check ssl verify none
http-response replace-header Location (.)(\b8544\b)(.)$ \17777\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend ntsm_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 ntsm:8546 check ssl verify none
http-response replace-header Location (.)(\b8546\b)(.)$ \17777\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend systemmonitor_serv
mode http
balance roundrobin
log stdout local0 debug
server srv01 mnc-admin:8547 check ssl verify none
http-response replace-header Location (.)(\b8547\b)(.) \17777\3 http-response replace-header Location (.*)(\b8548\b)(.*) \17777\3
http-request add-header X-Forwarded-Proto https if { ssl_fc }

listen stats
bind *:1974
stats enable

4

if you use names for server addresses on your server lines, and the IP address associated to the name changes, haproxy can’t know until you restart it if you don’t use a resolvers section as detailed here

I’m guessing that’s what happens here since you fix it with an haproxy restart.

5

Thank You very much.
after adding a resolver section it started working. appreciate your support.

regards

Tridev