Error 503 on frontend despite backend servers up

I’ve just set up HAProxy on my pfSense router, and all of my requests are returning error 503: no server is available. When I check the logs, entries are showing up for the correct frontend with error 503, however the stats page shows the servers are up (and when I bypass HAProxy I have no issues connecting to them), and my frontend has a default backend set, so even if my ACLs are wrong I would expect a 404 in the worst case from sending the request to the wrong server.

I ran a packet capture between HAProxy and my only backend server that’s not on HTTPS to see what was getting sent between them, and the only traffic was the regular health check. That was while making requests from the internet (using my cell phone with wifi turned off). So it seems that requests are not being forwarded to the backend, which makes sense with the 503 error. How can I see what HAProxy is doing with the incoming requests? Is there a way to troubleshoot which ACL is being applied to incoming requests, or other logic?

This is what the log entries look like for each request:

Apr 13 22:48:57 haproxy[25781]: 123.123.123.123:34984 [13/Apr/2020:22:48:57.984] HTTPS-IN~ HTTPS-IN/ -1/-1/-1/-1/0 503 29 - - SC-- 1/1/0/0/0 0/0 “GET /index.html HTTP/1.1”

SC error code means:

https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#8.5

 SC   The server or an equipment between it and haproxy explicitly refused
      the TCP connection (the proxy received a TCP RST or an ICMP message
      in return). Under some circumstances, it can also be the network
      stack telling the proxy that the server is unreachable (e.g. no route,
      or no ARP response on local network). When this happens in HTTP mode,
      the status code is likely a 502 or 503 here.

You really should share your full configuration here (the actual haproxy configuration on the filesystem, not the pfsense webinterface abstraction), for us to be able to help you out.

Thanks for the reply, that’s very interesting. Most of my backend is currently an Nginx server running as a reverse proxy. I don’t think it would reset the TCP connection, as for one thing the health checks are working, and for another I can connect with netcat without a TCP reset. However that sounds like my firewall could be rejecting the connection from HAProxy - I know this isn’t a pfSense forum, but I’ll check my firewall logs again. It might explain why no requests showed up in my packet capture.

Here’s my full configuration file:

# Automaticaly generated, dont edit manually.
# Generated on: 2020-04-13 22:47
global
        maxconn                 500
        log                     /var/run/log    local0  debug
        stats socket /tmp/haproxy.socket level admin  expose-fd listeners
        uid                     80
        gid                     80
        nbproc                  1
        nbthread                        1
        hard-stop-after         15m
        chroot                          /tmp/haproxy_chroot
        daemon
        tune.ssl.default-dh-param       2048
        server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
        bind 127.0.0.1:2200 name localstats
        mode http
        stats enable
        stats admin if TRUE
        stats show-legends
        stats uri /haproxy/haproxy_stats.php?haproxystats=1
        timeout client 5000
        timeout connect 5000
        timeout server 5000

resolvers globalresolvers
        nameserver pfsense 127.0.0.1:53
        resolve_retries 3
        timeout retry 1s
        timeout resolve 10s

frontend HTTPS-IN
        bind                    55.55.55.55:443 name 55.55.55.55:443   ssl crt-list /var/etc/haproxy/HTTPS-IN.crt_list
        mode                    http
        log                     global
        option                  socket-stats
        option                  httplog
        option                  http-keep-alive
        option                  forwardfor
        acl https ssl_fc
        http-request set-header         X-Forwarded-Proto http if !https
        http-request set-header         X-Forwarded-Proto https if https
        timeout client          30000
        errorfile                       503 /var/etc/haproxy/errorfile_HTTPS-IN_503_Default
        acl                     Webui   var(txn.txnhost) -m str -i pfsense
        acl                     Unraid  var(txn.txnhost) -m str -i tower
        acl                     homeassistant   var(txn.txnhost) -m str -i homeassistant
        acl                     aclcrt_HTTPS-IN var(txn.txnhost) -m reg -i ^mydomain\.ca(:([0-9]){1,5})?$
        http-request set-var(txn.txnhost) hdr(host)
        use_backend Webui_ipvANY  if  Webui aclcrt_HTTPS-IN
        use_backend Unraid_ipvANY  if  Unraid aclcrt_HTTPS-IN
        use_backend Hassio_ipvANY  if  homeassistant aclcrt_HTTPS-IN
        use_backend subdomain_ipvANY  if   aclcrt_HTTPS-IN

frontend ACME
        bind                    55.55.55.55:80 name 55.55.55.55:80
        mode                    http
        log                     global
        option                  http-keep-alive
        option                  forwardfor
        acl https ssl_fc
        http-request set-header         X-Forwarded-Proto http if !https
        http-request set-header         X-Forwarded-Proto https if https
        timeout client          30000
        acl                     ACME    var(txn.txnpath) -m beg -i .well-known/acme-challenge/
        http-request set-var(txn.txnpath) path
        use_backend ACME_ipvANY  if  ACME

backend Webui_ipvANY
        mode                    http
        id                      102
        log                     global
        timeout connect         30000
        timeout server          30000
        retries                 3
        option                  httpchk OPTIONS /
        server                  pfSense 127.0.0.1:1443 id 103 ssl check inter 1000  verify none resolvers globalresolvers

backend Unraid_ipvANY
        mode                    http
        id                      106
        log                     global
        timeout connect         30000
        timeout server          30000
        retries                 3
        option                  httpchk OPTIONS /
        server                  unraid-webgui 192.168.1.3:80 id 107 check inter 1000  resolvers globalresolvers

backend Hassio_ipvANY
        mode                    http
        id                      111
        log                     global
        timeout connect         30000
        timeout server          30000
        retries                 3
        server                  hassio 192.168.1.4:8123 id 112 check inter 1000  resolvers globalresolvers

backend subdomain_ipvANY
        mode                    http
        id                      100
        log                     global
        option                  log-health-checks
        http-response set-header Strict-Transport-Security max-age=10;
        timeout connect         30000
        timeout server          30000
        retries                 3
        option                  httpchk GET /
        server                  subdomain.mydomain.ca 192.168.1.3:443 id 101 ssl check inter 1000  verify none resolvers globalresolvers

backend ACME_ipvANY
        mode                    http
        id                      104
        log                     global
        timeout connect         30000
        timeout server          30000
        retries                 3
        server                  ACME-BACKEND 127.0.0.1:4002 id 105  resolvers globalresolvers

Okay, I figured it out and wanted to report back here. The issue was not with HAProxy at all, and neither was it with my backend. I had configured my ACME cert manager improperly, and forgot to include a wildcard certificate. When I fixed the settings and reissued the certificate, the error immediately disappeared.