503 Service Unavailable - Rundeck

Help!
1

03:27:41 # service haproxy restart
Stopping haproxy: [ OK ]
Starting haproxy: [WARNING] 218/032810 (8549) : parsing [/etc/haproxy/haproxy.cfg:22] : ‘log-format’ : format variable ‘r’ is reserved for HTTP mode.
[WARNING] 218/032810 (8549) : parsing [/etc/haproxy/haproxy.cfg:22] : ‘log-format’ : format variable ‘Tr’ is reserved for HTTP mode.
[WARNING] 218/032810 (8549) : Setting tune.ssl.default-dh-param to 1024 by default, if your workload permits it you should set it to at least 2048. Please set a value >= 1024 to make this warning disappear.
[ OK ]
root@cernrdeckhaproxy101.cernerasp.com:~ ( cernrdeckhaproxy101.cernerasp.com )
03:28:10 #
Message from syslogd@localhost at Aug 6 03:28:10 …
haproxy[8549]: backend rundeck has no server available!

Message from syslogd@localhost at Aug 6 03:28:12 …
haproxy[8550]: backend mysql has no server available!

When i load the page i get 503 service unavailable. Any help will me much appreciated.

2

Please elaborate.

From your post we know very little about the situation and nothing at all about your setup.

3

We have 2 DB nodes and 2 App nodes behind 2 haproxy nodes.

We had patched the nodes recently. and the Java and rundeck version on app nodes was upgraded and we downgraded it to the previous version.

Later noticed that rundeck URL is not browsing giving 503 error.

while restarting the haproxy service i get this error.

Message from syslogd@localhost at Aug 6 03:28:10 …
haproxy[8549]: backend rundeck has no server available!

Message from syslogd@localhost at Aug 6 03:28:12 …
haproxy[8550]: backend mysql has no server available!

4

So since you worked on your backend servers, haproxy returns 503 messages and restarting haproxy unsurprisingly does not help.

Please start by actually sharing your haproxy configuration.

Most likely your haproxy health checks return negative results. It’s possible we need your to send Host headers or SNI to your backend, after the changes.

5

Below is the haproxy configuration.

global
user haproxy
group haproxy
daemon
log 127.0.0.1 len 2000 local1 debug
stats socket /var/run/haproxy.sock mode 600
stats timeout 2m

defaults
log global
mode http
retries 3
maxconn 2000
timeout connect 5000ms
timeout client 600000ms
timeout server 600000ms
option dontlognull
option redispatch
log-format %{+Q}o\ %{-Q}ci\ -\ -\ [%T]\ %r\ %ST\ %B\ %hrl\ %Tr\ %U\ %B\ %b\ %f

frontend stats_interface
capture request header Referer len 2000
capture request header User-Agent len 250
capture request header Host len 100
capture request header X-Forwarded-For len 50
capture request header Correlation-Id len 50
http-request deny if METH_TRACE OR METH_OPTIONS
bind *:8080
default_backend stats_interface

frontend http-in
capture request header Referer len 2000
capture request header User-Agent len 250
capture request header Host len 100
capture request header X-Forwarded-For len 50
capture request header Cerner-Correlation-Id len 50
http-request deny if METH_TRACE OR METH_OPTIONS
bind *:80
redirect scheme https if !{ ssl_fc }
default_backend rundeck

frontend https-in
capture request header Referer len 2000
capture request header User-Agent len 250
capture request header Host len 100
capture request header X-Forwarded-For len 50
capture request header Cerner-Correlation-Id len 50
http-request deny if METH_TRACE OR METH_OPTIONS
bind *:443 ssl crt /etc/ssl/private/rundeck-sandbox.asp.com.pem
default_backend rundeck

frontend mysql-in
capture request header Referer len 2000
capture request header User-Agent len 250
capture request header Host len 100
capture request header X-Forwarded-For len 50
capture request header Correlation-Id len 50
http-request deny if METH_TRACE OR METH_OPTIONS
bind *:3306
default_backend mysql
mode tcp

peers rundeck_proxy
peer rdeckhaproxy101.asp.c ip-address:1024
peer rdeckhaproxy102.asp.c ip-address:1024

backend stats_interface
server local 0.0.0.0:8080
stats enable
stats uri /
stats realm Strictly\ Private

backend rundeck
server rdeckapp101.asp.com ip-address:4440 check observe layer4 error-limit 2 on-error mark-down
server rdeckapp102.asp.com ip-address:4440 check observe layer4 error-limit 2 on-error mark-down backup
balance source
mode http
stick-table type ip size 100k expire 24h peers rundeck_proxy
stick on src
option httpchk

backend mysql
server rdeckmsql101.asp.com ip-address:3306 check port 9200 inter 2000 rise 6 fall 3 downinter 5000
server rdeckmsql102.asp.com ip-address:3306 check port 9200 inter 2000 rise 6 fall 3 downinter 5000
server rdeckmsql103.asp.com ip-address:3306 check port 9200 inter 2000 rise 6 fall 3 downinter 5000
balance source
mode tcp
stick-table type ip size 100k expire 24h peers rundeck_proxy
stick on src
option httpchk

6

So neither the backend rundeck nor the backend mysql works and haproxy health check declares them down.

If you doubt haproxy health checks, capture the health check traffic and analyze it. But first, try if you can actually reach your backends from your box where haproxy runs: curl -vv http://ip-address:4440/ for the rundeck backend and curl -vv http://ip-address:9200/ for the mysql backend all from the same VM where haproxy runs.

7

root@rdeckhaproxy101.asp.com:~ ( rdeckhaproxy101.asp.com )
01:06:16 # curl -vv http://ip-address:4440

  • About to connect() to ip-address port 4440 (#0)
  • Trying ip-address… connected
  • Connected to ip-address (ip-address) port 4440 (#0)

GET / HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.36 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: ip-address:4440
Accept: /

< HTTP/1.1 503 Service Unavailable
< Cache-Control: must-revalidate,no-cache,no-store
< Content-Type: text/html;charset=ISO-8859-1
< Content-Length: 1287
< Server: Jetty(7.6.0.v20120127)
<

Error 503 Service Unavailable

HTTP ERROR: 503

Problem accessing /. Reason: 

    Service Unavailable

Powered by Jetty:// * Connection #0 to host ip-address left intact * Closing connection #0
8

root@rdeckhaproxy101.asp.com:~ ( rdeckhaproxy101.asp.com )
01:08:13 # curl -vv http://ip-address:9200/

  • About to connect() to ip-address port 9200 (#0)
  • Trying ip-address… Connection refused
  • couldn’t connect to host
  • Closing connection #0
    curl: (7) couldn’t connect to host
9

There you go: your backend servers are not available. Either they refuse the connection (port 9200) or they answer with a 503 Service Unavailable response.

That is way you see them down in haproxy.

10

Hi Lukas,

How to fix the issue?
Your assistance would be of great help.

Thanks.

11

I have no idea why your backends are not working, and won’t be able to help you with that.

12

On my haproxy node i am unable to access mysql
root@haproxy101.asp.com:~ ( haproxy101.asp.com )

06:57:53 # mysql -u user -h msql101.asp.com -p

-bash: mysql: command not found