Robots.txt file on HAProxy

[brianb]

Hello,

Currently we host multiple sites behind our HAProxy load balancer, and I would like to prevent Robots from indexing and crawling our sites. I do not want to put a robots.txt on every site we host behind our load balancer, but I would rather have the file on the load balancer itself. Does anyone know if this is possible through use of an ACL or other kinds of config?

Thanks.

[lukastribus]

How do I serve a single static file from HAProxy?

We have a custom backend we use to serve a single static file: acl is_robots_txt path /robots.txt use_backend robotstxt if is_robots_txt is_cdn backend robotstxt mode http errorfile 503 /etc/haproxy/errors/200robots.http The file contains: HTTP/1.0 200 Found Cache-Control: no-cache Connection: close Content-Type: text/plain # Discourse CDN, all crawling is allowed (cdn contains images and text files) User-Agent: * Allow: / This means that when people hit: https://cdn.discourse.org/ro…