Recently we had an interesting case where our application behind our HAProxies started responding very slowly.
So the cause for the 520 HTTP error(A Cloudflare only error) was quite clear.
Instead of serving a Cloudflare 520 error, I would rather serve my own maintenance/error page, we have one in place in case of a HTTP 503 error which works perfectly. I could do the same for a HTTP 408 code.
To do that I think I need to adjust the timeouts so HAProxy will timeout the server before Cloudflare does.(I believe the Cloudflare HTTP timeout is 15 seconds.)
My current timeouts are like this, however this way we still get the HTTP 520 errors if we ever have a critical error in our web backend.(which has only happened once so far)
We run HAProxy in http mode
timeout connect 6s
timeout client 300s
timeout server 300s
timeout http-request 10s
timeout http-keep-alive 120s
So my initial question is, should I lower the server timeout to get the expected behaviour and serve a custom 408 page?
Or bonus question, can I make it so that if a server does not respond to a request that it gets redispatched? (keeping in mind that some requests should not ever be repeated).