[Partly Solved] Proxy to server determined by external process at layer 6 (non-HTTP)


#1

My understanding of HAProxy is very limited but I think what I am asking here is not possible with HAProxy, and would like confirmation or disconfirmation.

Based on the contents of the first packet, which contains the name of the server, I would map that name to the actual backend servicing the request, possibly starting the server if it is not already started, and then transparently proxying the connection. If my understanding of HAProxy is correct, it cannot do packet inspection on layer 4 proxies (non-HTTP). Is this correct?

If HAProxy cannot do it, is there some other recommendation short of a fully custom proxy which could do this?

Thanks!


#2

That is not necessarily true.

We can already content-switch (select a particular backend based on TCP payload) for SNI in SSL/TLS sessions for example (that is when you don’t decrypt SSL in haproxy, but just plainly forward TCP traffic we know is SSL).

We can fetch arbitrary payload from layer 6 with payload and payload_lv and use that to content switch.

It all depends on what your TCP protocol looks like. If it contains a value you would like to use on a fixed offset, it should be doable with the fetch samples above.

What’s the protocol you are looking at?


#3

This happens to be the Minecraft protocol, and in the handshake packet is basically what I am looking at:

http://wiki.vg/Protocol#Handshake
Basically, the server address is the key bit of information I am looking at (A UTF-8 encoded string). The field before it is technically of variable size, but perhaps I can do something clever?


#4

Well, the Protocol Version can grow until 16363 to stay within the current 2 byte length, so I guess just statically looking at the third byte will work for a long time (protocol number currently is at 340).

However, it would be impossible to support Server List Ping, as it doesn’t contain the Server address field.

If you don’t need Server List ping and you just need the actual Handshake packet, you should be able to do this, by using arbitrary binary matches (payload).

Figuring out what the bytes look like at byte 3 and beyond is something you will have to automate away by other means though, the protobuf style protocol is not that friendly to the human eye.

Take a look at packet captures, and work from there.


#5

So, trying to understand how all of this fits together, it seems like I could extract the desired server using a Lua extension registered as a sample fetcher. Using content-switching, like described here, I could select a different backend based on the results of that sample fetch. However, it’s not clear to me how I would dynamically create a backend to switch to.

Example. Suppose I extract the value “myserver.myhost.com” from the packet. I don’t know what IP that might map to, nor whether that server is even running yet.

One possibility, since all the possible names ARE known ahead of time, is that I have pre-mapped all of them to static internal IPs and put them into the HAProxy config, OR they will DNS resolve internally to the right IP once the server is started. What I don’t understand is how to get this extracted DNS name to the backend (or more strictly, the server entry).

It occurs to me that perhaps an answer might be that when a new server is registered (giving it a new name in the *.myhost.com namespace), I generate a new HAProxy config file which adds the new ACL for the front end (matching the new host name), and a new backend which will be selected by that ACL, with the backend specifying that dns name in the server entry. When such a new server is added, I perform a “reload” operation on HAProxy which if I understand it correctly will more or less gracefully continue existing connections uninterrupted with the new file. This would mean from HAProxy’s standpoint, the config is always a static file, and the only bit I need to do is to have a script (maybe a LUA extension to do the capture.)

This still leaves me with exactly how to start that backend once the request comes in though.

Does this make sense? Am I misunderstanding or missing anything here?


#6

I have made substantial progress on this, so I want to give others who may have a similar issue an idea of what they can do…

Using Lua, its possible to inspect the incoming packet using a tcp-content request <lua.myaction> directive in the Frontend. In my case, this action reads the incoming packet, extracts the address information as well as some state information and then adds a session variable to the session containing that information. Note that in order for this to work, there must be sufficient delay configured on incoming connections to allow the first packet to be read.

Then, use-backend references this variable to extract the desired hostname. The config file already has a list of all possible backends available (whether the servers are up or not), so use-backend will either directly switch to the requested backend and attempt to connect to the desired server (through DNS resolution or static IPs as appropriate to the configuration), or it will failover (ultimately in my case, rejecting the connection.)

The other part of my request was to start the server if the incoming packet is a request for a connection as opposed to a request for status. This information is also available in the packet, so part of the session variable is populated with this. Additionally, if it is a connection request, the Lua script launches a shell script to start the server, passing that script the desired public address (which results in a call to a REST service that launches the server if it is not already launched.)

One has to be careful here about making blocking calls - in particular it doesn’t appear possible to make a complete REST transaction from directly within Lua as when I get to the response reading part, the connection appears to be closed and no response body is available (thus I decided to use an external script instead.) Curiously, if one uses os.execute - which is a blocking call - that will work, but presumably this blocks some internal HAProxy networking threads for the duration, so probably not advisable.

Rather than one use-backend directive I actually have two, and each is conditional on an ACL which determines whether to use the backend which goes to the server, or use the backend which goes to a status service - the latter making the server look like its available even if it isn’t presently running so the server can appear to be online at all times.

So that’s where I am at now.