HTTP 503 after changing port number on backend

No change when I changed the path to /http from /http-bind/. Grasping…

Well, chat-srv/ejabberd was select as backend/server, as evident from the logs, so I don’t think the ACL is the issue.

Does this debug config still work when replacing -d with -db (not debug, just foreground)?
If yes, try with -D (uppercase D) instead, that is daemon mode and will make this haproxy instance go into the background (can’t stop that one with ctrl-c, you’ll have to kill the appropriate PID).

The reason I am asking is that some code path may be different in debug/foreground mode, so it worth trying.

If this all works fine, you’ll have to attach strace -tt -p<PID> to your actual haproxy instance and run a request through it.

Both -db and -D worked. Here’s strace parts that looked relevant.

	[pid 39495] 12:38:45.967564 write(1, "\24\3\3\0\1\1\26\3\3\0(]7\36v\253\215q\223LA\352q\21\26\276\26\r32\201\202"..., 51) = 51
[pid 39495] 12:38:45.967741 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 39495] 12:38:45.967848 read(1, 0x55c1255d58b3, 5) = -1 EAGAIN (Resource temporarily unavailable)
[pid 39495] 12:38:45.967948 epoll_wait(0, [{EPOLLIN, {u32=1, u64=1}}], 200, 1000) = 1
[pid 39495] 12:38:45.968595 read(1, "\27\3\3\0u", 5) = 5
[pid 39495] 12:38:45.968703 read(1, "\0\0\0\0\0\0\0\1\23}\213\31\22\315@\352\320:'%7yH\n\f\3nCqn\353_"..., 117) = 117
[pid 39495] 12:38:45.968887 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 2
[pid 39495] 12:38:45.969067 fcntl(2, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
[pid 39495] 12:38:45.969152 setsockopt(2, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 39495] 12:38:45.969239 connect(2, {sa_family=AF_INET, sin_port=htons(9443), sin_addr=inet_addr("172.16.145.115")}, 16) = -1 EACCES (Permission denied)
[pid 39495] 12:38:45.969690 close(2)    = 0
[pid 39495] 12:38:45.969789 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 2
[pid 39495] 12:38:45.969899 fcntl(2, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
[pid 39495] 12:38:45.970114 setsockopt(2, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 39495] 12:38:45.970210 connect(2, {sa_family=AF_INET, sin_port=htons(9443), sin_addr=inet_addr("172.16.145.115")}, 16) = -1 EACCES (Permission denied)
[pid 39495] 12:38:45.970371 close(2)    = 0
[pid 39495] 12:38:45.970463 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 2
[pid 39495] 12:38:45.970568 fcntl(2, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
[pid 39495] 12:38:45.970666 setsockopt(2, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 39495] 12:38:45.970758 connect(2, {sa_family=AF_INET, sin_port=htons(9443), sin_addr=inet_addr("172.16.145.115")}, 16) = -1 EACCES (Permission denied)
[pid 39495] 12:38:45.970930 close(2)    = 0
[pid 39495] 12:38:45.971074 socket(AF_INET, SOCK_STREAM, IPPROTO_TCP) = 2
[pid 39495] 12:38:45.971186 fcntl(2, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
[pid 39495] 12:38:45.971265 setsockopt(2, SOL_TCP, TCP_NODELAY, [1], 4) = 0
[pid 39495] 12:38:45.971350 connect(2, {sa_family=AF_INET, sin_port=htons(9443), sin_addr=inet_addr("172.16.145.115")}, 16) = -1 EACCES (Permission denied)
[pid 39495] 12:38:45.971512 close(2)    = 0
[pid 39495] 12:38:45.971609 epoll_ctl(0, EPOLL_CTL_DEL, 1, 0x55c115ca30e0) = 0
[pid 39495] 12:38:45.971692 epoll_wait(0, [], 200, 0) = 0
[pid 39495] 12:38:45.971781 write(1, "\27\3\3\0\354]7\36v\253\215q\224\365\35\243\366b\273QOJ\272x\6\3536\\\354\324\372s"..., 241) = 241
[pid 39495] 12:38:45.971928 write(1, "\25\3\3\0\32]7\36v\253\215q\225\3531\360K\223\377@\377b\254\17\27\316:\21=FX", 31) = 31
[pid 39495] 12:38:45.972116 shutdown(1, SHUT_WR) = 0
[pid 39495] 12:38:45.972261 sendto(7, "<150>Aug 31 12:38:45 haproxy[394"..., 219, MSG_DONTWAIT|MSG_NOSIGNAL, {sa_family=AF_INET, sin_port=htons(514), sin_addr=inet_addr("127.0.0.1")}, 16) = 219
[pid 39495] 12:38:45.972687 close(1)    = 0

The kernel denies the connect call with EACCES (Permission denied).

man 2 connect says:

   EACCES, EPERM
          The user tried to connect to a broadcast address without
          having the socket broadcast flag enabled or the connection
          request failed because of a local firewall rule.

Something on your systems denies the connect call to that backend, on a syscall level. I don’t really know where that is coming from. Maybe something like SELinux. You’ll have to dig into your box and understand what security related features and rules are enabled. But this is a kernel level firewall not allowing this particular application to connect to that particular IP:port pair.

Maybe also checkout the haproxy startup script, maybe it is assigned into a particular SELinux context.

I really don’t know SELinux …

Thanks for all your help. I’ll look into the SELinux angle.