HAProxy 1.5.18 giving error 503 when proxying Tomcat?

Help!
1

I’ve got HAProxy running as a reverse proxy on a CentOS 7.4 box, with JIRA (Tomcat server) as the backend application and it proxies from https 443 to the non-SSL port on the JIRA server without issue.

However, on three other servers that are also running Tomcat apps on the backend, setting up the exact same reverse proxy and HAProxy haproxy.cfg (changing hosts and ports, of course) and they return 503 Service Unavailable errors. I can confirm that the Tomcat servers are up and running and accessible on their non-SSL ports so the servers are behaving exactly like the working one. This seems to be HAProxy related?
cfg is below. Any ideas?
#---------------------------------------------------------------------

Global settings

#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the ‘-r’ option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2

chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats

#---------------------------------------------------------------------

common defaults that all the ‘listen’ and ‘backend’ sections will

use if not designated in their block

#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000

#---------------------------------------------------------------------

main frontend which proxys to the backends

Tells HAProxy to start listening on for HTTPS requests. It uses the SSL key

and certificate found within the corp.company.com.pem file. All requests will

be routed to the confluence_http_backend

#---------------------------------------------------------------------
frontend confluence_http_frontend
bind *:443 ssl crt /etc/pki/tls/keystore/corp.company.com.pem
default_backend confluence_http_backend

#---------------------------------------------------------------------

static backend for serving up images, stylesheets and such

The confluence_http_backend simpl forwards all requests onto http://dev-wiki.corp.xperi.com:8090/.

It will only allow 1000 concurrent connections to the server at once.

#---------------------------------------------------------------------
backend confluence_http_backend
mode http
option httplog
option forwardfor

option httpchk

server server01 server.corp.mydomain.com:8090 maxconn 1000

balance roundrobin

#---------------------------------------------------------------------

round robin balancing between the various backends

#---------------------------------------------------------------------
#backend app

balance roundrobin

server app1 127.0.0.1:5001 check

server app2 127.0.0.1:5002 check

server app3 127.0.0.1:5003 check

server app4 127.0.0.1:5004 check

2

Well, it’s kind of hard to read your configuration like this (try using the “Preformatted text” option in the editor for this kind of stuff), but you seem to have one backend with 2 different sets of servers and nothing in your configuration that tells HAProxy how to choose between them, so it just does a roundrobin over all 5 servers. You probably want to start by fixing this. Ref:
http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4-use-server

Also, have you checked what the HAProxy log line says about requests coming to this frontend? That would tell you if HAProxy is giving the 503 response on its own (you would then see confluence_http_backend/<NOSRV> on that line with the default log format), because it thinks that there are no backend servers available. Or if you see e.g. confluence_http_backend/app1 then that would mean the backend server app1 is giving the 503 response.

Finally, you can check via the stats unix socket you already have set up what HAProxy thinks is the status of your backend servers. Ref:
http://cbonte.github.io/haproxy-dconv/1.5/configuration.html#9.2

or for something simpler to use, set up the web-based stats page, e.g.:

listen stats
        bind :8181
        stats enable
        stats uri /
        stats show-legends