HTTP 503 after changing port number on backend

Help!
1

Hi,

I have inherited an existing reverse proxy configuration I’m trying to change that seems simple enough, but something isn’t working. The configuration that works uses port 9000 on the backend. I’m trying to change that to 9443, but when I do I am getting the following in the Chrome console:

"Failed to load the resource: the server responded with a status of 503 (Service Unavailable)"

I’ve used tcpdump -A -s 0 ‘tcp port 9443’ to try to verify that the server is seeing the request, but none appears.

My servers are using HAProxy 1.5.18 on CentOS 7.

The proxy server’s firewall accepts HTTPS on port 443.

The server’s firewall accepts TCP connections from the proxy server on ports 443 and 9000 (or 9443 as necessary).

The app’s functionality works fine in this configuration.

I’m not sure what to try next and would appreciate any help.

Thanks,
Scott

Here’s my HAProxy configuration:

#---------------------------------------------------------------------

Global settings

#---------------------------------------------------------------------
global
#Set the protocol
ssl-default-bind-options no-sslv3 force-tlsv12

#set the acceptable ciphers
ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
#debug
log         127.0.0.1 local2

chroot      /var/lib/haproxy/haproxy    pidfile     /var/run/haproxy.pidfile
# max per-process number of connections
maxconn     256

# process's user and group
user        haproxy    group       haproxy
# make the process fork into background
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats
tune.ssl.default-dh-param 2048

#---------------------------------------------------------------------

common defaults that all the ‘listen’ and ‘backend’ sections will

use if not designated in their block

#---------------------------------------------------------------------
defaults mode http
log global # the following enables logging of HTTP requests
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
# allow SSE and WebSocket connections to stay open for longer
timeout tunnel 8h
maxconn 3000

#---------------------------------------------------------------------

main frontend which proxys to the backends

#---------------------------------------------------------------------
frontend main bind *:80
bind *:443 ssl crt /usr/local/glads/ssl/web.glads.stk.com.bundle.pem ca-file /usr/local/glads/ssl/ca-chain.cert.pem verify required crl-file /usr/local/glads/crl/combined.crl.pem
# allow access to the keycloak admin console to a select few ips or subnets.
acl network_allowed src 192.168.100.10/32
acl restricted_page path_beg -m beg -i /auth/admin/master/console http-request deny if restricted_page !network_allowed
http-request set-header X-SSL-Client-Cert %{+Q}[ssl_c_der,base64]
http-request set-header X-SSL-Issuer %{+Q}[ssl_c_i_dn]

#for logging purposes STIG V-69385
capture request header referer len 64
capture request header user-agent len 64
capture request header host len 100
capture request header connect len 20

#log-format %Tl\ %ci:%cp\ requestHeader=%hrl\ httpRequest=%redispatch
# Force SSL only
redirect scheme https if !{ ssl_fc }

acl acl-glads-id            path_beg        /identity /auth/admin/master/console    acl acl-glads-app           path_beg        /favicon.ico /glads-web    acl acl-glads-app           path_beg        /glads    acl acl-glads-app           path_beg        /help
acl acl-glads-wx            path_beg        /geoserver /geowebcache
acl acl-terrain             path_beg        /stk-terrain    acl acl-chat                path_beg        /http-bind
use_backend glads-id        if acl-glads-id    use_backend glads-app       if acl-glads-app    use_backend stk-terrain     if acl-terrain    use_backend glads-wx        if acl-glads-wx    use_backend chat-srv        if acl-chat

#---------------------------------------------------------------------

glads-id backend for serving identity

#---------------------------------------------------------------------
backend glads-id server glads_id c0007150-00.stk.com:443 ssl ca-file /usr/local/glads/ssl/ca-chain.cert.pem
#---------------------------------------------------------------------

glads-web backend for serving all things GLADS

#---------------------------------------------------------------------
backend glads-app server glads_app c0007151-00.stk.com:443 ssl ca-file /usr/local/glads/ssl/ca-chain.cert.pem
#---------------------------------------------------------------------

backend for serving terrain

#---------------------------------------------------------------------
backend stk-terrain server glads_terrain assets.agi.com:443 ssl ca-file /usr/local/glads/ssl/ca-chain.cert.pem
#---------------------------------------------------------------------

glads-wx backend for serving WMTS

#---------------------------------------------------------------------
backend glads-wx server glads_wx c0007151-00.stk.com:8443 ssl ca-file /usr/local/glads/ssl/ca-chain.cert.pem
#---------------------------------------------------------------------

chat-srv backend for serving EjabberD

#---------------------------------------------------------------------
backend chat-srv server ejabberd c0007151-00.stk.com:9443 ssl ca-file /usr/local/glads/ssl/ca-chain.cert.pem timeout server 90s

2

I assume this is simply a reachability problem, I can see that you checked the firewalls but it doesn’t look like you actually tried it, so I’d suggest to check it from the proxy with something like this:

$ curl -v --with-ca-bundle=/usr/local/glads/ssl/ca-chain.cert.pem http://c0007151-00.stk.com:9443/

I’ve guessed the hostname here, you’d obviously need to replace it with whatever backend server you are having problems with.

Btw, every hostname above is recognized as URL, so your post technically contains 5 URLs. Thats why discourse rejected your post earlier. You’d have to put the entire configuration in code blocks (the </> button above), makes it more readable and avoid misinterpretation of hostnames with URLs.

3

Checking from the proxy worked.
curl -v --cacert /usr/local/glads/ssl/ca-chain.cert.pem https://c0007151-00.stk.com:9443/http-bind/

Through the proxy failed with the 503.
`curl -v --cacert /usr/local/glads/ssl/ca-chain.cert.pem --cert /opt/GLADSCerts/glads.operator.0000000102.crt --key /opt/GLADSCerts/glads.operator.0000000102.nopass.key https://c0007152-00.stk.com:443/http-bind/

  • About to connect() to c0007152-00.stk.com port 443 (#0)
  • Trying 172.16.145.116…
  • Connected to c0007152-00.stk.com (172.16.145.116) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /usr/local/glads/ssl/ca-chain.cert.pem
    CApath: none
  • NSS: client certificate from file
  •   subject: E=glads.operator.0000000102@glads.agi.com,CN=glads.operator.0000000102,OU=GLADS,O="Analytical Graphics, Inc.",ST=PA,C=US

  •   start date: May 01 00:00:00 2017 GMT

  •   expire date: Dec 31 23:59:59 2018 GMT

  •   common name: glads.operator.0000000102

  •   issuer: CN=Nimbus Intermediate CA,OU=Cirrus Certificate Authority,O="Analytical Graphics, Inc.",ST=Pennsylvania,C=US
  • SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  • Server certificate:
  •   subject: CN=*.stk.com,OU=Development,O="Analytical Graphics, Inc.",ST=PA,C=US

  •   start date: Jun 12 19:04:26 2017 GMT

  •   expire date: Dec 31 23:59:59 2019 GMT

  •   common name: *.stk.com

  •   issuer: CN=Stratus Intermediate CA,OU=Cirrus Certificate Authority,O="Analytical Graphics, Inc.",ST=Pennsylvania,C=US

GET /http-bind/ HTTP/1.1
User-Agent: curl/7.29.0
Host: c0007152-00.stk.com
Accept: /

  • HTTP 1.0, assume close after body
    < HTTP/1.0 503 Service Unavailable
    < Cache-Control: no-cache
    < Connection: close
    < Content-Type: text/html
    <

503 Service Unavailable

No server is available to handle this request. * Closing connection 0`
4

Capture the failing port 9443 traffic on the proxy and checkout what happens in the capture. There must be something that your backend server doesn’t like.

Also try the direct curl command with the IP address instead of the hostname and see if that fails. It’s possible your backend requires SNI, Host header or both.

5

Here’s the line from the proxy log:

<Aug 17 12:33:54 localhost haproxy[18196]: 172.16.145.116:44208 [17/Aug/2018:12:33:54.018] main~ chat-srv/ejabberd 180/0/-1/-1/180 503 212 - - SC-- 0/0/0/0/3 0/0 {|curl/7.29.0|c0007152-00.stk.com|} “GET /http-bind/ HTTP/1.1”/>

Direct to server using IP address fails:

< curl -v --cacert /usr/local/glads/ssl/ca-chain.cert.pem --cert /opt/GLADSCerts/glads.operator.0000000102.crt --key /opt/GLADSCerts/glads.operator.0000000102.nopass.key https://172.16.145.115:9443/http-bind/

  • About to connect() to 172.16.145.115 port 9443 (#0)
  • Trying 172.16.145.115…
  • Connected to 172.16.145.115 (172.16.145.115) port 9443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /usr/local/glads/ssl/ca-chain.cert.pem
    CApath: none
  • Server certificate:
  •   subject: CN=*.stk.com,OU=Development,O="Analytical Graphics, Inc.",ST=PA,C=US

  •   start date: Jun 12 19:04:12 2017 GMT

  •   expire date: Dec 31 23:59:59 2019 GMT

  •   common name: *.stk.com

  •   issuer: CN=Stratus Intermediate CA,OU=Cirrus Certificate Authority,O="Analytical Graphics, Inc.",ST=Pennsylvania,C=US
  • NSS error -12276 (SSL_ERROR_BAD_CERT_DOMAIN)
  • Unable to communicate securely with peer: requested domain name does not match the server’s certificate.
  • Closing connection 0
    curl: (51) Unable to communicate securely with peer: requested domain name does not match the server’s certificate./>

Unfortunately, much of this is new to me. Could this be the SNI scenario you mentioned?

6

Can you add the -k parameter to the curl call with the ip address?

7

Adding -k option resulted in success.

8

Then it’s neither SNI nor the Host header and we have nothing.

Capturing the 9443 traffic generated by haproxy is the only way:

9

Here’s the log entry using curl through the proxy:

Aug 23 18:21:36 localhost haproxy[10696]: 172.16.145.116:34384 [23/Aug/2018:18:21:36.137] main~ chat-srv/ejabberd 318/0/-1/-1/319 503 212 - - SC-- 0/0/0/0/3 0/0 {|curl/7.29.0|c0007152-00.stk.com|} "GET /http-bind/ HTTP/1.1"

Which, if I read this correctly, says;

  1. The TCP session was unexpectedly closed by the server, or the server explicitly refused it.
  2. The proxy was waiting for the CONNECTION to establish on the server. The server might at most have noticed a connection attempt.
10

We more or less knew that already, what we need is a packet capture.

11 
tcpdump -i eth0 port 9443

yields nothing on the server and I do see the 503 in the HAProxy log.

Is there some other packet capture I could do (this is not something I’ve done before)?

12

You’d need to capture on the haproxy server, not the backend server.

Also, don’t truncate the packets (-s0) and write it to a cap file (-w capture.cap).

13

I’ve got the capture. Upload wouldn’t allow .pcap file. Should I just rename it?

14

Are you sure you captured this on the haproxy box, not on the destination server?

15

I thought so, but will do it again.

16

The reason I am asking is that there is no port 9443 traffic at all.

17

That capture is from the correct system. The application’s traffic all comes in on 443 and I’m trying to route paths that begin with /http-bind/ to 9443. No traffic to 9443 is consistent with what I’m seeing on the application server.

18

Ok, we need to isolate this and run it through debug and strace, because otherwise, I don’t see how we would be able to make progress in this troubleshooting…

Create a new, separate configuration file (without the other backends) on the haproxy server, have it listening on a different port (so you don’t influence the existing haproxy instance) run haproxy directly from CLI in a debug mode, and then through strace -tt.

Config would look like this (different ports, pidfiles, stats socket, as to not interfere), single chat backend:

global
    #Set the protocol
    ssl-default-bind-options no-sslv3 force-tlsv12

    #set the acceptable ciphers
    ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
    #debug
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy/haproxy
    pidfile     /var/run/haproxydebug.pidfile
    # max per-process number of connections
    maxconn     256

    # process's user and group
    user        haproxy    group       haproxy
    # make the process fork into background
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/statsdebug
    tune.ssl.default-dh-param 2048

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults    mode                    http
    log                     global
    # the following enables logging of HTTP requests
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    # allow SSE and WebSocket connections to stay open for longer
    timeout tunnel          8h
    maxconn                 3000

frontend  main
    bind *:1080
    bind *:10443 ssl crt /usr/local/glads/ssl/web.glads.stk.com.bundle.pem ca-file /usr/local/glads/ssl/ca-chain.cert.pem verify required crl-file /usr/local/glads/crl/combined.crl.pem
    # allow access to the keycloak admin console to a select few ips or subnets.
    acl network_allowed         src                 192.168.100.10/32
    acl restricted_page         path_beg            -m beg -i /auth/admin/master/console
    http-request deny           if restricted_page  !network_allowed
    http-request set-header X-SSL-Client-Cert           %{+Q}[ssl_c_der,base64]
    http-request set-header X-SSL-Issuer                %{+Q}[ssl_c_i_dn]

    #for logging purposes STIG V-69385
    capture request header referer len 64
    capture request header user-agent len 64
    capture request header host len 100
    capture request header connect len 20

    #log-format %Tl\ %ci:%cp\ requestHeader=%hrl\ httpRequest=%redispatch
    # Force SSL only
    redirect scheme https if !{ ssl_fc }

    default_backend chat-srv

backend chat-srv
 server ejabberd c0007151-00.stk.com:9443 ssl ca-file /usr/local/glads/ssl/ca-chain.cert.pem
 timeout server 90s

Then:

  • run it through debug mode (-d) and capture the output (haproxy -f /etc/new-debug-config-file -d)
  • run it through strace (and without going into background mode with -db) and capture the output (strace -tt haproxy -f /etc/new-debug-config-file -db)

Each time, fire a request through the test instance of haproxy, remember to use the different port. I assume this would look something like:

curl -v --cacert /usr/local/glads/ssl/ca-chain.cert.pem --cert /opt/GLADSCerts/glads.operator.0000000102.crt \
--key /opt/GLADSCerts/glads.operator.0000000102.nopass.key \
https://c0007152-00.stk.com:10443/http-bind/

Then provide the entire output of haproxy in debug mode, and stracing haproxy. If the output is short enough, you can post it in this forum by using the preformatted text format (the </> button above), but at least strace output will certainly be very long, so either upload it to something like pastbin (preferred) or use the upload link you previously used for the pcap trace.

Please make sure newlines are correctly maintained, the configuration you posted in the first post has a lot of missing newlines, which makes it even harder to read.

19

How do I fix this error?

[ALERT] 241/180525 (14823) : [haproxy.main()] Cannot chroot(/var/lib/haproxy/haproxy).
20

Debug config worked.

	[root@c0007152-00 tmp]# haproxy -f /tmp/haproxy_debug.cfg -d
Available polling systems :
	  epoll : pref=300,  test result OK       poll : pref=200,  test result OK     select : pref=150,  test result OKTotal: 3 (3 usable), will use epoll.
Using epoll() as the polling mechanism.
00000000:main.accept(0006)=0008 from [172.16.145.116:49020]
00000000:main.clireq[0008:ffffffff]: GET /http-bind/ HTTP/1.1
00000000:main.clihdr[0008:ffffffff]: User-Agent: curl/7.29.0
00000000:main.clihdr[0008:ffffffff]: Host: c0007152-00.stk.com:10443
00000000:main.clihdr[0008:ffffffff]: Accept: */*
00000000:chat-srv.srvrep[0008:0009]: HTTP/1.1 200 OK00000000:chat-srv.srvhdr[0008:0009]: Connection: close
00000000:chat-srv.srvhdr[0008:0009]: Content-Length: 9363
00000000:chat-srv.srvhdr[0008:0009]: Content-Type: text/xml; charset=utf-8
00000000:chat-srv.srvhdr[0008:0009]: Access-Control-Allow-Origin: *
00000000:chat-srv.srvhdr[0008:0009]: Access-Control-Allow-Headers: Content-Type00000001:main.clicls[0008:ffffffff]
00000001:main.closed[0008:ffffffff]

The only visible difference I see in the config, is the routing “default_backend chat-srv” instead of the ACL I’m using. Could the hyphen in the path, “/http-bind/” be causing a problem?