503 Service Unavailable - but the server is there

I have three NextCloud servers, all connected via pfSense/HAProxy. One of them isn’t working properly—I get a “503 Service Unavailable” error when trying to access it externally. However, I can still reach it using the local IP from within the internal LAN. The other two NextCloud servers work fine through both the external address and the internal IP.

The haproxy.cfg has the same configuration for all three servers. What could be causing this issue?

A lot of possibilities, but what does your stats page say? It usually tells you why the connection failed and on what layer.

So it might provide some basic information.

The pfSense dashboard shows my third Nextcloud server as “DOWN,” while the others display “0/100.” The haproxy.cfg file has identical settings for all three servers, and they all function properly when accessed via their local IP addresses within the LAN.
image

[NOTICE] (50313) : haproxy version is 2.8.3-86e043a
[NOTICE] (50313) : path to executable is /usr/local/sbin/haproxy
[WARNING] (50313) : config : Server third.server.com_ipvANY/third.server.com is DOWN, changed from server-state after a reload. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

haprocy.cfg

This server is working:
backend second.server.com_ipvANY
mode http
id 110
log global
option log-health-checks
http-check send meth OPTIONS
timeout connect 30000
timeout server 30000
retries 3
load-server-state-from-file global
option httpchk
server second.server.com 192.168.1.24:443 id 111 ssl check inter 1000 verify none

This server is DOWN according to HAPROXY/pfsense… but I can access it local

backend third.server.com_ipvANY
mode http
id 131
log global
http-check send meth OPTIONS
timeout connect 30000
timeout server 30000
retries 3
load-server-state-from-file global
option httpchk
server third.server.com 192.168.1.30:443 id 132 ssl check inter 1000 verify none

I got Stats page working… what does this figures say?

Sessions limit: 100… is this is issue?

general information
pid = 37843 (process #1, nbproc = 1, nbthread = 1)
uptime = 0d 0h09m37s; warnings = 1
system limits: memmax = unlimited; ulimit-n = 2033
maxsock = 2033; maxconn = 1000; reached = 0; maxpipes = 0
current conns = 11; current pipes = 0/0; conn rate = 1/sec; bit rate = 34.804 kbps
Running tasks: 0/57; idle = 99 %

I’m no expert, but I can try to help.

Did you try connecting to the server from the haproxy machine to the server that seems to fail? I’d try a curl

curl -vI https://yourserver.com
  • connect to 192.168.1.30 port 443 from 192.168.1.1 port 54927 failed: Connection refused

  • Failed to connect to 192.168.1.30 port 443 after 0 ms: Couldn’t connect to server

    0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0

  • Closing connection
    curl: (7) Failed to connect to 192.168.1.30 port 443 after 0 ms: Couldn’t connect to server

same towards a working nextcloud server:

  • Connected to 192.168.1.24 (192.168.1.24) port 443

  • ALPN: curl offers h2,http/1.1
    } [5 bytes data]

  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
    } [512 bytes data]

    0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0* TLSv1.3 (IN), TLS handshake, Server hello (2):
    { [122 bytes data]

  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
    { [21 bytes data]

  • TLSv1.3 (IN), TLS handshake, Certificate (11):
    { [820 bytes data]

  • TLSv1.3 (OUT), TLS alert, unknown CA (560):
    } [2 bytes data]

  • SSL certificate problem: self-signed certificate

    0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0

  • Closing connection
    curl: (60) SSL certificate problem: self-signed certificate
    More details here: curl - SSL CA Certificates

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

From another computer in LAN:

root@storage ~# curl -vI https://192.168.1.30

  • Trying 192.168.1.30:443…
  • Connected to 192.168.1.30 (192.168.1.30) port 443 (#0)
  • ALPN: offers h2,http/1.1
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN: server accepted http/1.1
  • Server certificate:
  • subject: CN=storage
  • start date: Aug 5 12:43:46 2024 GMT
  • expire date: Aug 5 12:43:46 2025 GMT
  • subjectAltName does not match 192.168.1.30
  • SSL: no alternative certificate subject name matches target host name ‘192.168.1.30’
  • Closing connection 0
  • TLSv1.3 (OUT), TLS alert, close notify (256):
    curl: (60) SSL: no alternative certificate subject name matches target host name ‘192.168.1.30’
    More details here: curl - SSL CA Certificates

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

The selfsigned cert shouldnt be a problem, as apparently haproxy trusts that one, however the other error that there is no connection possible seems to me that this issue is not related to haproxy at all but something on the server

I got similar problem with another server now… is my pfsense to blame?

haproxy: startup error output!: [NOTICE] (29403) : haproxy version is 2.8.3-86e043a[NOTICE] (29403) : path to executable is /usr/local/sbin/haproxy[WARNING] (29403) : config : Server photoprism_ipvANY/photoprism is DOWN, changed from server-state after a reload. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

same with this server, works fine from LAN but

Curl from pfsense shows the same…

  • connect to 192.168.1.15 port 443 from 192.168.1.1 port 11214 failed: Connection refused
  • Failed to connect to 192.168.1.15 port 443 after 0 ms: Couldn’t connect to server