I am developing an HAProxy extension in Rust using the Lua API endpoints. The code presents an interstitial page unless the user has a cookie to pass it. This library is deployed in Docker alongside Redis for session management.
This works very well, except sometimes (after several hours of production use and never during testing even with simulated traffic), it starts presenting white pages and eventually 504 errors when the timeout is reached.
I would presume that this is the fault of the backend server, but it can be fixed by restarting the HAProxy Docker, implying that it is HAProxy’s fault. However, I never had this problem with HAProxy before (but I did not use Docker). Once the 504 error starts appearing, it does not go away until I restart Docker. I don’t think it’s the fault of the scripts because the interstitial page always works, even when the errors are being thrown.
I’m really at a loss at what it is. It’s hard to reproduce, only happens in production where there’s a time pressure to just restart it, and I’m not finding any valuable debugging information.
Yes, haproxy will absolutely respond with 504. This happens when the server timeout strikes.
I suggest you analyze haproxy logs and prepare a tcpdump to capture the backend traffic.
AFAIK it is possible to make haproxy misbehave when using unsupported LUA API calls, but I’m not familiar enough with that part to give you something specific to look for.
I don’t think that I am using any strange API calls.
There’s two bugs that I’m experiencing:
Timeouts leading to 504 pages that persist until the proxy is restarted.
Returning blank pages from the backend which only contain headers.
In regards to the second, I am pretty sure this is not the webserver for two reasons. First, it’s happening to static javascript files. Second, the backend headers are being sent, only the content is truncated. This causes haproxy to cache a blank file, and distribute a blank file with caching headers, which causes the clients to cache a blank javascript file. This destroys usability and causes a myriad of issues.