I’ve just set up HAProxy on my pfSense router, and all of my requests are returning error 503: no server is available. When I check the logs, entries are showing up for the correct frontend with error 503, however the stats page shows the servers are up (and when I bypass HAProxy I have no issues connecting to them), and my frontend has a default backend set, so even if my ACLs are wrong I would expect a 404 in the worst case from sending the request to the wrong server.
I ran a packet capture between HAProxy and my only backend server that’s not on HTTPS to see what was getting sent between them, and the only traffic was the regular health check. That was while making requests from the internet (using my cell phone with wifi turned off). So it seems that requests are not being forwarded to the backend, which makes sense with the 503 error. How can I see what HAProxy is doing with the incoming requests? Is there a way to troubleshoot which ACL is being applied to incoming requests, or other logic?
This is what the log entries look like for each request:
SC The server or an equipment between it and haproxy explicitly refused
the TCP connection (the proxy received a TCP RST or an ICMP message
in return). Under some circumstances, it can also be the network
stack telling the proxy that the server is unreachable (e.g. no route,
or no ARP response on local network). When this happens in HTTP mode,
the status code is likely a 502 or 503 here.
You really should share your full configuration here (the actual haproxy configuration on the filesystem, not the pfsense webinterface abstraction), for us to be able to help you out.
Thanks for the reply, that’s very interesting. Most of my backend is currently an Nginx server running as a reverse proxy. I don’t think it would reset the TCP connection, as for one thing the health checks are working, and for another I can connect with netcat without a TCP reset. However that sounds like my firewall could be rejecting the connection from HAProxy - I know this isn’t a pfSense forum, but I’ll check my firewall logs again. It might explain why no requests showed up in my packet capture.
Here’s my full configuration file:
# Automaticaly generated, dont edit manually.
# Generated on: 2020-04-13 22:47
global
maxconn 500
log /var/run/log local0 debug
stats socket /tmp/haproxy.socket level admin expose-fd listeners
uid 80
gid 80
nbproc 1
nbthread 1
hard-stop-after 15m
chroot /tmp/haproxy_chroot
daemon
tune.ssl.default-dh-param 2048
server-state-file /tmp/haproxy_server_state
listen HAProxyLocalStats
bind 127.0.0.1:2200 name localstats
mode http
stats enable
stats admin if TRUE
stats show-legends
stats uri /haproxy/haproxy_stats.php?haproxystats=1
timeout client 5000
timeout connect 5000
timeout server 5000
resolvers globalresolvers
nameserver pfsense 127.0.0.1:53
resolve_retries 3
timeout retry 1s
timeout resolve 10s
frontend HTTPS-IN
bind 55.55.55.55:443 name 55.55.55.55:443 ssl crt-list /var/etc/haproxy/HTTPS-IN.crt_list
mode http
log global
option socket-stats
option httplog
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
errorfile 503 /var/etc/haproxy/errorfile_HTTPS-IN_503_Default
acl Webui var(txn.txnhost) -m str -i pfsense
acl Unraid var(txn.txnhost) -m str -i tower
acl homeassistant var(txn.txnhost) -m str -i homeassistant
acl aclcrt_HTTPS-IN var(txn.txnhost) -m reg -i ^mydomain\.ca(:([0-9]){1,5})?$
http-request set-var(txn.txnhost) hdr(host)
use_backend Webui_ipvANY if Webui aclcrt_HTTPS-IN
use_backend Unraid_ipvANY if Unraid aclcrt_HTTPS-IN
use_backend Hassio_ipvANY if homeassistant aclcrt_HTTPS-IN
use_backend subdomain_ipvANY if aclcrt_HTTPS-IN
frontend ACME
bind 55.55.55.55:80 name 55.55.55.55:80
mode http
log global
option http-keep-alive
option forwardfor
acl https ssl_fc
http-request set-header X-Forwarded-Proto http if !https
http-request set-header X-Forwarded-Proto https if https
timeout client 30000
acl ACME var(txn.txnpath) -m beg -i .well-known/acme-challenge/
http-request set-var(txn.txnpath) path
use_backend ACME_ipvANY if ACME
backend Webui_ipvANY
mode http
id 102
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server pfSense 127.0.0.1:1443 id 103 ssl check inter 1000 verify none resolvers globalresolvers
backend Unraid_ipvANY
mode http
id 106
log global
timeout connect 30000
timeout server 30000
retries 3
option httpchk OPTIONS /
server unraid-webgui 192.168.1.3:80 id 107 check inter 1000 resolvers globalresolvers
backend Hassio_ipvANY
mode http
id 111
log global
timeout connect 30000
timeout server 30000
retries 3
server hassio 192.168.1.4:8123 id 112 check inter 1000 resolvers globalresolvers
backend subdomain_ipvANY
mode http
id 100
log global
option log-health-checks
http-response set-header Strict-Transport-Security max-age=10;
timeout connect 30000
timeout server 30000
retries 3
option httpchk GET /
server subdomain.mydomain.ca 192.168.1.3:443 id 101 ssl check inter 1000 verify none resolvers globalresolvers
backend ACME_ipvANY
mode http
id 104
log global
timeout connect 30000
timeout server 30000
retries 3
server ACME-BACKEND 127.0.0.1:4002 id 105 resolvers globalresolvers
Okay, I figured it out and wanted to report back here. The issue was not with HAProxy at all, and neither was it with my backend. I had configured my ACME cert manager improperly, and forgot to include a wildcard certificate. When I fixed the settings and reissued the certificate, the error immediately disappeared.