# Error 503 on frontend despite backend servers up

**URL:** <https://discourse.haproxy.org/t/error-503-on-frontend-despite-backend-servers-up/5065>\
**Category:** Help!\
**Created:** [April 14, 2020, 5:51am UTC](https://discourse.haproxy.org/t/error-503-on-frontend-despite-backend-servers-up/5065 "2020-04-14T05:51:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![bdben](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bdben](https://discourse.haproxy.org/u/bdben)\
**Post date:** [April 14, 2020, 5:51am UTC](https://discourse.haproxy.org/t/error-503-on-frontend-despite-backend-servers-up/5065/1 "2020-04-14T05:51:52Z")

</div>

I’ve just set up HAProxy on my pfSense router, and all of my requests are returning error 503: no server is available. When I check the logs, entries are showing up for the correct frontend with error 503, however the stats page shows the servers are up (and when I bypass HAProxy I have no issues connecting to them), and my frontend has a default backend set, so even if my ACLs are wrong I would expect a 404 in the worst case from sending the request to the wrong server.

I ran a packet capture between HAProxy and my only backend server that’s not on HTTPS to see what was getting sent between them, and the only traffic was the regular health check. That was while making requests from the internet (using my cell phone with wifi turned off). So it seems that requests are not being forwarded to the backend, which makes sense with the 503 error. How can I see what HAProxy is doing with the incoming requests? Is there a way to troubleshoot which ACL is being applied to incoming requests, or other logic?

This is what the log entries look like for each request:

> Apr 13 22:48:57 haproxy[25781]: 123.123.123.123:34984 [13/Apr/2020:22:48:57.984] HTTPS-IN~ HTTPS-IN/ -1/-1/-1/-1/0 503 29 - - SC-- 1/1/0/0/0 0/0 “GET /index.html HTTP/1.1”

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [April 14, 2020, 8:40am UTC](https://discourse.haproxy.org/t/error-503-on-frontend-despite-backend-servers-up/5065/2 "2020-04-14T08:40:31Z")

</div>

`SC` error code means:

[https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#8.5](https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#8.5)

```
 SC The server or an equipment between it and haproxy explicitly refused
      the TCP connection (the proxy received a TCP RST or an ICMP message
      in return). Under some circumstances, it can also be the network
      stack telling the proxy that the server is unreachable (e.g. no route,
      or no ARP response on local network). When this happens in HTTP mode,
      the status code is likely a 502 or 503 here.

```

You really should share your full configuration here (the actual haproxy configuration on the filesystem, not the pfsense webinterface abstraction), for us to be able to help you out.

---

<div class="post-metadata">

**Author:** ![bdben](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bdben](https://discourse.haproxy.org/u/bdben)\
**Post date:** [April 14, 2020, 3:57pm UTC](https://discourse.haproxy.org/t/error-503-on-frontend-despite-backend-servers-up/5065/3 "2020-04-14T15:57:46Z")

</div>

Thanks for the reply, that’s very interesting. Most of my backend is currently an Nginx server running as a reverse proxy. I don’t think it would reset the TCP connection, as for one thing the health checks are working, and for another I can connect with netcat without a TCP reset. However that sounds like my firewall could be rejecting the connection from HAProxy - I know this isn’t a pfSense forum, but I’ll check my firewall logs again. It might explain why no requests showed up in my packet capture.

Here’s my full configuration file:

```
# Automaticaly generated, dont edit manually.
# Generated on: 2020-04-13 22:47
global
        maxconn 500
        log /var/run/log local0 debug
        stats socket /tmp/haproxy.socket level admin expose-fd listeners
        uid 80
        gid 80
        nbproc 1
        nbthread 1
        hard-stop-after 15m
        chroot /tmp/haproxy_chroot
        daemon
        tune.ssl.default-dh-param 2048
        server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
        bind 127.0.0.1:2200 name localstats
        mode http
        stats enable
        stats admin if TRUE
        stats show-legends
        stats uri /haproxy/haproxy_stats.php?haproxystats=1
        timeout client 5000
        timeout connect 5000
        timeout server 5000

resolvers globalresolvers
        nameserver pfsense 127.0.0.1:53
        resolve_retries 3
        timeout retry 1s
        timeout resolve 10s

frontend HTTPS-IN
        bind 55.55.55.55:443 name 55.55.55.55:443 ssl crt-list /var/etc/haproxy/HTTPS-IN.crt_list
        mode http
        log global
        option socket-stats
        option httplog
        option http-keep-alive
        option forwardfor
        acl https ssl_fc
        http-request set-header X-Forwarded-Proto http if !https
        http-request set-header X-Forwarded-Proto https if https
        timeout client 30000
        errorfile 503 /var/etc/haproxy/errorfile_HTTPS-IN_503_Default
        acl Webui var(txn.txnhost) -m str -i pfsense
        acl Unraid var(txn.txnhost) -m str -i tower
        acl homeassistant var(txn.txnhost) -m str -i homeassistant
        acl aclcrt_HTTPS-IN var(txn.txnhost) -m reg -i ^mydomain\.ca(:([0-9]){1,5})?$
        http-request set-var(txn.txnhost) hdr(host)
        use_backend Webui_ipvANY if Webui aclcrt_HTTPS-IN
        use_backend Unraid_ipvANY if Unraid aclcrt_HTTPS-IN
        use_backend Hassio_ipvANY if homeassistant aclcrt_HTTPS-IN
        use_backend subdomain_ipvANY if aclcrt_HTTPS-IN

frontend ACME
        bind 55.55.55.55:80 name 55.55.55.55:80
        mode http
        log global
        option http-keep-alive
        option forwardfor
        acl https ssl_fc
        http-request set-header X-Forwarded-Proto http if !https
        http-request set-header X-Forwarded-Proto https if https
        timeout client 30000
        acl ACME var(txn.txnpath) -m beg -i .well-known/acme-challenge/
        http-request set-var(txn.txnpath) path
        use_backend ACME_ipvANY if ACME

backend Webui_ipvANY
        mode http
        id 102
        log global
        timeout connect 30000
        timeout server 30000
        retries 3
        option httpchk OPTIONS /
        server pfSense 127.0.0.1:1443 id 103 ssl check inter 1000 verify none resolvers globalresolvers

backend Unraid_ipvANY
        mode http
        id 106
        log global
        timeout connect 30000
        timeout server 30000
        retries 3
        option httpchk OPTIONS /
        server unraid-webgui 192.168.1.3:80 id 107 check inter 1000 resolvers globalresolvers

backend Hassio_ipvANY
        mode http
        id 111
        log global
        timeout connect 30000
        timeout server 30000
        retries 3
        server hassio 192.168.1.4:8123 id 112 check inter 1000 resolvers globalresolvers

backend subdomain_ipvANY
        mode http
        id 100
        log global
        option log-health-checks
        http-response set-header Strict-Transport-Security max-age=10;
        timeout connect 30000
        timeout server 30000
        retries 3
        option httpchk GET /
        server subdomain.mydomain.ca 192.168.1.3:443 id 101 ssl check inter 1000 verify none resolvers globalresolvers

backend ACME_ipvANY
        mode http
        id 104
        log global
        timeout connect 30000
        timeout server 30000
        retries 3
        server ACME-BACKEND 127.0.0.1:4002 id 105 resolvers globalresolvers
```

---

<div class="post-metadata">

**Author:** ![bdben](https://avatars.discourse-cdn.com/v4/letter/b/65b543/32.png) [@bdben](https://discourse.haproxy.org/u/bdben)\
**Post date:** [April 14, 2020, 6:12pm UTC](https://discourse.haproxy.org/t/error-503-on-frontend-despite-backend-servers-up/5065/4 "2020-04-14T18:12:18Z")

</div>

Okay, I figured it out and wanted to report back here. The issue was not with HAProxy at all, and neither was it with my backend. I had configured my ACME cert manager improperly, and forgot to include a wildcard certificate. When I fixed the settings and reissued the certificate, the error immediately disappeared.