Hello, Currently we host multiple sites behind our HAProxy load balancer, and I would like to prevent Robots from indexing and crawling our sites. I do not want to put a robots.txt on every site we host behind our load balancer, but I would rather have the file on the load balancer itself. Does an…

[image] How do I serve a single static file from HAProxy? Help! We have a custom backend we use to serve a single static file: acl is_robots_txt path /robots.txt use_backend robotstxt if is_robots_txt is_cdn backend robotstxt mode http errorfile 503 /etc/haproxy/err…

Robots.txt file on HAProxy

Help!

lukastribus March 8, 2017, 7:34pm 2

Topic		Replies	Views
ACL Pathing not working for /robots.txt Help!	0	327	March 25, 2024
How do I serve a single static file from HAProxy? Help!	15	56624	January 5, 2021
.htaccess file for noindex Help!	0	519	June 8, 2022
Howto block badbots, crawlers & scrapers using list file Help!	4	9881	February 7, 2017
HAproxy configuration Help!	6	1771	June 10, 2018

Robots.txt file on HAProxy

Related topics