We have a custom backend we use to serve a single static file:
acl is_robots_txt path /robots.txt use_backend robotstxt if is_robots_txt is_cdn backend robotstxt mode http errorfile 503 /etc/haproxy/errors/200robots.http
The file contains:
HTTP/1.0 200 Found Cache-Control: no-cache Connection: close Content-Type: text/plain # Discourse CDN, all crawling is allowed (cdn contains images and text files) User-Agent: * Allow: /
This means that when people hit:
https://cdn.discourse.org/robots.txt well they get the robots file.
This hack works OK, except that now we are logging a bunch of
503 statuses in the logs which in reality are
200s, this in turn makes log analysis a bit annoying.
Is there another hack we can use that allows us to serve a single static file from a backend and retain status 200 in the logs?