How do I serve a single static file from HAProxy?

We have a custom backend we use to serve a single static file:

acl is_robots_txt path /robots.txt
use_backend robotstxt if is_robots_txt is_cdn


backend robotstxt
  mode http
  errorfile 503 /etc/haproxy/errors/200robots.http

The file contains:

HTTP/1.0 200 Found
Cache-Control: no-cache
Connection: close
Content-Type: text/plain

# Discourse CDN, all crawling is allowed (cdn contains images and text files)
User-Agent: *
Allow: /

This means that when people hit: https://cdn.discourse.org/robots.txt well they get the robots file.

This hack works OK, except that now we are logging a bunch of 503 statuses in the logs which in reality are 200s, this in turn makes log analysis a bit annoying.

Is there another hack we can use that allows us to serve a single static file from a backend and retain status 200 in the logs?

1 Like

You may disable logging in this backend, which is probably the best thing to do anyway since you don’t care about who retrieves this file :

http-request set-log-level silent

I wanted to have the “return-raw” and “return-html” actions merged into 1.6 but we didn’t have time to do them.

1 Like

I see, ideally I want to keep the logging, can I achieve my desired result with a LUA script running in the backend serving the request?

Hi Sam,

Yes, you may be able to create a lua service which delivers the content of you robots.txt.
Then you may have a 200 in th logs.

Baptiste

1 Like

could it be done for 1.6.4 maybe or does it require bigger changes?

also yay my favorite forum software for my favorite proxy!

time to find out how stable 1.7 is :stuck_out_tongue:

1 Like

Anyone have a good example of how to serve a single static file using lua?
Looks a bit less hacky than above, would like to create a landing page for an ssl verify required scenario, vs. just present a request for cert.

There is a hello world example here, but it doesn’t have much:

Also I saw someone just do:

listen http-webservices
    bind :8080 
    monitor-uri /c 
    errorfile 200 /etc/haproxy/errorfiles/200.http

I got a LUA response working based on the rabbit trail @mattpark started me down. I am using this with HAproxy running on pfSense.

I have several responses in a single lua file and just call them by function name (service at the bottom)

Here is an example from my lua file, just replace the response data with the contents of your robots.txt, indicating new lines by a \n . If you want this to be HTML (or any other file type) just change the Content-Type header as needed.

http_resp = function(applet) 
local response = "Your Content goes here.\n use backslash+n to generate a new line"
applet:add_header("Content-Length", string.len(response)) 
applet:add_header("Content-Type", "text/plain") 
applet:set_status(200) 
applet:start_response() 
applet:send(response) 
end 

core.register_service("http_resp", "http", http_resp)

@sam, I’m sure you found an acceptable solution a long time ago, but I found this thread today and am not using this solution to disallow robots indexing on all of my backends at once with this frontend LUA config.

My specific robots.txt response uses this as the ACL to match to call the function.
Name Expression Value Actions
robots Path ends with: robots.txt

frontend config:

	acl			robots	path_end -i robots.txt
	http-request use-service lua.robots  if  robots aclcrt_SharedFront

LUA file:

robots = function(applet) 
local response = "User-agent: *\nDisallow: /"
applet:add_header("Content-Length", string.len(response)) 
applet:add_header("Content-Type", "text/plain") 
applet:set_status(200) 
applet:start_response() 
applet:send(response) 
end 

core.register_service("robots", "http", robots)
2 Likes

Hey there,
I came up with the following solution for serving files just like a web-server would do with the help of LUA. The following frontend-config sets the HTTP-header X-LUA-LOADFILE-DOCROOT with a local directory path which serves as the document-root. Furthermore, it defines a lua-service called load-file. See:

lua-load /etc/haproxy/lua/load-file.lua

...

backend lua-load-file
  http-request set-header X-LUA-LOADFILE-DOCROOT /etc/haproxy/docroot
  http-request use-service lua.load-file

As you can see I load a lua-file called load-file.lua. The contents of that file are as follows:

core.register_service("load-file", "http", function(applet)
  local docroot
  local location
  local file
  local retval
  local response
  local extension

  if(applet.path == nil or applet.headers["x-lua-loadfile-docroot"] == nil or applet.headers["x-lua-loadfile-docroot"][0] == "") then
    retval = 500
    response = "Internal Server Error"
  else
    docroot = applet.headers["x-lua-loadfile-docroot"][0]
    location = applet.path
    if(location == "" or location == "/") then
      location = "/index.html"
    end
    file = io.open(docroot .. location, "r")
    if(file == nil) then
      retval = 404
      response = "File Not Found"
    else
      retval = 200
      response = file:read("*all")
      file:close()
    end
  end

  extension = string.match(location, ".(%w+)$")
  if       extension == "css"  then applet:add_header("content-type", "text/css")
    elseif extension == "gif"  then applet:add_header("content-type", "image/gif")
    elseif extension == "htm"  then applet:add_header("content-type", "text/html")
    elseif extension == "html" then applet:add_header("content-type", "text/html")
    elseif extension == "ico"  then applet:add_header("content-type", "image/x-icon")
    elseif extension == "jpg"  then applet:add_header("content-type", "image/jpeg")
    elseif extension == "jpeg" then applet:add_header("content-type", "image/jpeg")
    elseif extension == "js"   then applet:add_header("content-type", "application/javascript; charset=UTF-8")
    elseif extension == "json" then applet:add_header("content-type", "application/json")
    elseif extension == "mpeg" then applet:add_header("content-type", "video/mpeg")
    elseif extension == "png"  then applet:add_header("content-type", "image/png")
    elseif extension == "txt"  then applet:add_header("content-type", "text/plain")
    elseif extension == "xml"  then applet:add_header("content-type", "application/xml")
    elseif extension == "zip"  then applet:add_header("content-type", "application/zip")
  end

  applet:set_status(retval)
  if(response ~= nil and response ~= "") then
    applet:add_header("content-length", string.len(response))
  end
  applet:start_response()
  applet:send(response)
end)

Basically, this code reads files from a specified document-root location in the filesystem based on the query string and generates appropriate HTTP-responses just as a normal webserver would do. That is why the script needs a document-root do be configured via the HTTP-header X-LUA-LOADFILE-DOCROOT . Furthermore, the code also does some very basic mimetype handling - extend the list according your needs. I hope someone finds this useful.

cheers!

3 Likes

We’ve finally decided to solve this problem properly by implementing in LUA.

EDIT: link removed due to doing IO in LUA being a terrible idea.

You should never access the file system from LUA. Especially not based on user-triggered requests:

The list of prohibited standard Lua functions during the runtime contains all those that do filesystem access:

  • os.remove()
  • os.rename()
  • os.tmpname()
  • package.*()
  • io.*()
  • file.*()

You CANNOT emulate a webserver with LUA, those are blocking syscalls, they don’t belong in an event driven software like haproxy.

In haproxy 2.2, you can use http-request return if the errorfile workaround disgusts you.

2 Likes

Thanks for the advice - I was feeling a little bit bad about doing IO like that and was thinking that I should be reading it in at start and caching it, but since http-request return does exactly that we’re now trying out HAProxy 2.3.

EDIT: this site now being served by HAProxy 2.3 :heart:

2 Likes

can you please provide an example of how you accomplished this with http-request return?
Thanks,
Steve

For example:

http-request return status 200 content-type "text/plain" file "/static/200robots.txt" hdr "cache-control" "no-cache"