Haproxy returns 403 NOSRV error intermittently

For a clean situation, stop all haproxy processes and kill all the rest, and then start it. Check the process table to see a “normal” situation. Reload and restart haproxy a few teams then check the process table again to see a “normal” situation after reload/restarts.

systemctl stop haproxy
killall haproxy
ps auxwf

The reason multiple processes binding to the same port are possible is due to the use of SO_REUSEPORT on the socket. You can disable it with the noreuseport directive, however it’s possible reload/restart performance is impaired.

You can test reproduce this by trying to start haprocesses manually (/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg a few times). It should not be possible when noreuseport is configured.

You may also want to consider hard-stop-after to set a timeout for an old process that still serves connections after a reload situation (although this process would not accept() new connections, so it’s not really related to this case).