Adding client IP to the first TCP message in TCP mode

[zeusz4u]

I was wondering if it’s possible to retain an in incoming connection’s public IP to the TCP stream that’s going to be routed to a backend server.

This should apply to the situation that you guys helped me resolve here:

TLS Termination Proxy with both TLS and plain TCP on the same port

We’re considering using HAProxy as a TLS termination proxy, running in front of our TCP server where our clients connect with their front-end apps. I’m wondering if HAProxy is capabale of making distinction between SSL connection and plain connection on the same port in the frontend section (like binding for example on port 80 both the plain and the ssl sockets), and regardless if it’s an encrypted or non-encrypted connection, it will be proxied on to the backend server: if it’s a plain connect…

The backend server is FIX server, so doesn’t know PROXY protocol.

My ultimate goal would be adding the incoming client’s source IP address (which is known by the proxy) to the first incoming message when the connection is established. First message in FIX would be always login, and I would like to add the source IP address delimited by \n in front of the login message (in other words adding a header to a plain TCP text message like FIX logon). The backend’s parser would take care of the rest extracting this info and doing whatever is needed with it. Also this is preferably only needed once when the connection is first established.

Is this possible to achieve with HAProxy?

Concrete example of FIX login message that’s incoming:

8=FIX.4.4|9=126|35=A|49=theBroker.12345|56=CSERVER|34=1|52=20170117- 08:03:04|57=TRADE|50=any_string|98=0|108=30|141=Y|553=12345|554=passw0rd!|10=131|

I want to alter it by appending the following header to it (using haproxy.org’s IP in the example):

51.15.8.218
8=FIX.4.4|9=126|35=A|49=theBroker.12345|56=CSERVER|34=1|52=20170117- 08:03:04|57=TRADE|50=any_string|98=0|108=30|141=Y|553=12345|554=passw0rd!|10=131|

Any other delimiter would be acceptable not only \n, we could also get by adding a fixed length string in front of the message and eliminate it later in the backend (like in the below example IP would reserve xxx.xxx.xxx.xxx - 19 bytes to have headroom to store any IPv4 address in the message):

51.15.8.218    8=FIX.4.4|9=126|35=A|49=theBroker.12345|56=CSERVER|34=1|52=20170117- 08:03:04|57=TRADE|50=any_string|98=0|108=30|141=Y|553=12345|554=passw0rd!|10=131|

[lukastribus]

No, you will have to implement the proxy protocol. Which should be easy enough, it’s just (when IPv4 is used):

PROXY TCP4 <source-ipv4> <destination-ipv4> <source-port> <destination-port>\r\n

[zeusz4u]

Will this only be sent once when the TCP connection is initially established in case of FIX?

The TCP connection will be kept open between the client and server throughout the lifetime of a fix session…

Another alternative I’ve looking into is to log the FIX logon message in plain text in the haproxy logs along with the soruce IP of the client, so that every incoming FIX connection could be related to a public IP and port (something like I described above - but don’t think that’s possible - altough I saw examples of logging every single http request in /var/log/haproxy.log).

[lukastribus]

Yes, only once after connection is established - it wouldn’t work otherwise.

You can use tcp logging. But you cannot use http logging, as you don’t use HTTP.

[zeusz4u]

Thanks a lot, will then implement parsing the PROXY header in the code of the backend FIX server.

I suppose I will also need to add send-proxy/send-proxy-v2 in the backend node config to the server line after check to make this happen?

# main backend server to route to - only working with unencrypted connections
backend fix-backend
  mode tcp
  log global
  option tcplog
  server quickfix 127.0.0.1:9008 check send-proxy-v2

[lukastribus]

Yes, send-proxy is the ASCII protocol that I mentioned above. send-proxy-v2 would be a different, binary protocol. The binary protocol may be simpler to parse if you write in low level languages, but in higher languages you probably prefer ASCII strings.

You can find implementation details about the the proxy protocol in doc/proxy-protocol.txt.

[zeusz4u]

Works like a charm It’s ok to use the ASCII protocol for now, as FIX is also plain-text protocol.

Thanks for all your help!

[zeusz4u]

What would be the setting to immediately cut the connection of the client when he disconnects (terminates the TCP client connection and closes the socket)?

Currently I was running HAProxy withe the provided proxy settings, however for quick disconnect - reconnect cycles, the PROXY header is not always sent. I was suspecting this has to do with my settings for timeouts in HAProxy. As waiting a few more seconds usually ends up in sendint the PROXY V1 header again.

# timeout if backends do not reply
timeout connect 5000ms
# timeout on client side
timeout client 50000ms
# timeout on server side
timeout server 50000ms

This is my current settings. What would be the correct setup in this case for the situation when someone logs out and closes the connection, we want that upon the next connection to receive the proxy header again, even if this happens 1 second apart from the logout. There can also be frequent disconnects that only last a couple of miliseconds, and this would terminate a FIX session with the server and require a new login anyway as the server socket would be closed due to the client losing connection. In this case the next login message we want to have the Proxy header again to correctly identifiy the source IP of the user in the backend.

[zeusz4u]

Also is there a config option to forcefully terminate a TCP connection if there’s no traffic for N seconds coming from the client and log in the haproxy log that the connection was forcefully terminated due to timeout? In FIX protocol you expect to get at least a heartbeat every 30 seconds. The problem is heartbeats are not generated until you log in, but it’s possible that someone connects for example via telnet or sends some other junk data to the server, or simply not send anything and keep the connection open after connecting.

We have already implemented protective mechanism in the backend which terminates the connection after a period of time, but since using the proxy, the original client IP is not visible to our backend in this case, and we have no way to see which users are “abusing” of a TCP connection and not sending anything.

I’d like a way to gather those kicked IPs from logs that were forcefully disconnected to take some security measures later (like blocking them from firewall). Is HAPROXY capable of doing this for me?

[lukastribus]

@zeusz4u

What would be the setting to immediately cut the connection of the client when he disconnects (terminates the TCP client connection and closes the socket)?

That’s what haproxy already does.

@zeusz4u

Currently I was running HAProxy withe the provided proxy settings, however for quick disconnect - reconnect cycles, the PROXY header is not always sent. I was suspecting this has to do with my settings for timeouts in HAProxy.

No, this has absolutely nothing to do with timeouts. When send-proxy is configured, proxy protocol must always be sent. If not, its a bug. However I have some doubts about this being a problem on the haproxy side.

Capture (tcpdump) the traffic between haproxy and your backend to understand whether haproxy doesn’t send the proxy header or if your application does not interpret it.

@zeusz4u

Also is there a config option to forcefully terminate a TCP connection if there’s no traffic for N seconds coming from the client and log in the haproxy log that the connection was forcefully terminated due to timeout?

That’s what timeout client is.

@zeusz4u

The problem is heartbeats are not generated until you log in, but it’s possible that someone connects for example via telnet or sends some other junk data to the server, or simply not send anything and keep the connection open after connecting.

I don’t understand the problem. The proper clients will work just fine and attacker and telnet abusers not sending data for timeout client will be closed.

@zeusz4u

We have already implemented protective mechanism in the backend which terminates the connection after a period of time, but since using the proxy, the original client IP is not visible to our backend in this case

Are you referring to the problem mentioned earlier? Otherwise, the backend should have the client IP already.

@zeusz4u

we have no way to see which users are “abusing” of a TCP connection and not sending anything.

So that is what you are trying to figure out, howto deal with TCP clients that don’t send anything. If haproxy closes to due timeout client firing, you will see that in the haproxy logs.

If your backend closes the connection for whatever reason, haproxy doesn’t know whether it was because of a timeout on the backend or if the transaction has completed successfully. Your backend will have to store the IP address sent at connection establishment via proxy protocol and log that IP address.

You could also check the data before forwarding to the backend server. The FIX protocol implies that the client sends data beginning with 8=FIX, is that correct?

So when you reject clients that don’t send 8=FIX within the first 5 seconds:

tcp-request inspect-delay 5s
tcp-request content accept  if { payload(0,5) -m bin 5661707388 }
tcp-request content reject

You effectively get rid of them, it’s logged in haproxy as such and it never hits your backend servers in the first place.

[zeusz4u]

The last one is exactly what I’ve been looking for. Correct, 8=FIX is what I expect in the first message, this would be proceded by PROXY TCP4 255.255.255.255 255.255.255.255 65535 65535\r\n8=FIX. Our fix parser has implemented this already, and it’s able to extract source address and port from TCP4 and TCP6 addresses upon login and use the information received to identify source IP of the client.

Do you have any ideas why logging in onto the fix server writes the proxy logging with delays in /var/log/haproxy.log? Sometimes it only gets written when the connection is terminated (client disconnected), not immediately after login. I was wondering if I could monitor this log-file in realtime (or as close to real-time logging as possible). I’m using option tcplog already.

Sep 20 10:21:27 localhost haproxy[18691]: Proxy combined started.
Sep 20 10:21:27 localhost haproxy[18691]: Proxy plain_loopback started.
Sep 20 10:21:27 localhost haproxy[18691]: Proxy tls_loopback started.
Sep 20 10:21:27 localhost haproxy[18691]: Proxy fix_tls started.
Sep 20 10:21:27 localhost haproxy[18691]: Proxy fix_plain started.
Sep 20 10:21:27 localhost haproxy[18691]: Proxy fix-backend started.
Sep 20 10:22:52 localhost haproxy[18694]: 192.168.200.73:50382 [20/Sep/2018:10:21:32.606] fix_tls~ fix-backend/fix-backend 53/2/79779 2588 -- 2/1/0/0/0 0/0
Sep 20 10:22:52 localhost haproxy[18694]: 192.168.200.73:50382 [20/Sep/2018:10:21:32.605] combined tls_loopback/loopback-for-tls 1/1/79782 6229 -- 1/1/0/0/0 0/0

As you can see here there’s a 10:21:32 -> 10:22:52 delay in logging, there’s 80 sec delay in logging. It only got logged when I closed the connection from the client. Any way to get these things logged any faster? Why is this delay? Is there a standard buffer which needs to be filled before it gets flushed to logfile?

Also can we log the events when a user is disconnected/closed connection in the same manner? Currently this doesn’t get logged anywehre, only these 2 lines are logged for any logon. I’d like as many debug info as possible. It would be nice if I could even log part of the first message that contains logon, along with the proxy header if possible… not sure if possible though since this is not http, its tcp mode (I’ve only seen logging of http requests so far with haproxy).

Do you recommend using timeout client in conjuction with inspect delay and and payload check? What happens on a TLS-encrypted connection where you’re unable to check the contents of a TCP stream in the first proxy that accepts connetions from the outside? Can i add the inspect delays directly in the backend node? Currently I ended up using multiple proxies to accept both plain and TLS connections as follows:

combined(listen) -> plain_loopback/tls_loopback -> fix_tls/fix_plain -> fix-backend

the plain simply forwards traffic, and in the backend it adds send-proxy, while tls adds decryption/encryption to the loop.

[lukastribus]

@zeusz4u

The last one is exactly what I’ve been looking for. Correct, 8=FIX is what I expect in the first message, this would be proceded by PROXY

Between haproxy and your backend, yes. However between the client and haproxy not, no. And this is where you want to check for the correct header. We don’t care about the connection between haproxy and the backend server here, way may not even establish a connection to the server if the client doesn’t actually speak FIX, right?

@zeusz4u

Do you have any ideas why logging in onto the fix server writes the proxy logging with delays in /var/log/haproxy.log ? Sometimes it only gets written when the connection is terminated (client disconnected), not immediately after login.

It’d say always, logging at disconnection time is default behavior. Add option logasap if you want immediate logs (but I’m not sure if they would be very useful).

@zeusz4u

As you can see here there’s a 10:21:32 -> 10:22:52 delay in logging, there’s 80 sec delay in logging.

No, your TCP connection lasted for 79779 milliseconds, and the log was immediately send at disconnection time.

There are not a lot of interesting information’s to log immediately, all the interesting things like timers are only known at disconnection, which is why that’s the default.

@zeusz4u

Also can we log the events when a user is disconnected/closed connection in the same manner? Currently this doesn’t get logged anywehre, only these 2 lines are logged for any logon.

Those are these 2 logs: sessions state at disconnect time, with all the data (session start at 10:21:32.606, session duration: 79779 ms, etc).

@zeusz4u

It would be nice if I could even log part of the first message that contains logon, along with the proxy header if possible

I don’t think you can do that. And it doesn’t make sense to log the proxy header, as the informations contained therein are already present in the log file.

@zeusz4u

Do you recommend using timeout client in conjuction with inspect delay and and payload check ?

Yes, always use timeouts. Whether the are high or low depends on what works better for you, but never leave them unconfigured (otherwise sessions never close).

@zeusz4u

What happens on a TLS-encrypted connection where you’re unable to check the contents of a TCP stream in the first proxy that accepts connetions from the outside? Can i add the inspect delays directly in the backend node? Currently I ended up using multiple proxies to accept both plain and TLS connections as follows:

Well you need to inspect them after the TLS encryption, but before going to your backend. So in your case, applying the payload check in the fix_plain frontend would be the correct thing to do. For your combined listening frontend, you may also do fix protocol matching for plaintext flix, so you avoid the 2 second inspect-delay we talked about.

[zeusz4u]

Thanks just realized after I wrote why the logging delay (I know I should have read documentation earlier ): https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#8.2.2

It’s just a bit too overwhelming for me. HAProxy seems more complex that I thought at first.

I’ll add incpection delay to both fix_plain (which binds to virtual socket then simply forwards to backend) and in fix_tls (which also binds to virtual socket and adds ssl to it then forwards to backend), before forwarding to backend, this way I should be able to cover both scenarios easily.

Thanks a lot for your help. I’ll get onto some wireshark monitoring to see if the fix backend skips that proxy header somehow and we don’t know about it. What’s strange it’s completely random whether it picks up the proxy header or not, I also tend to think it’s a backend issue.

[lukastribus]

Just make sure you don’t have any old haproxy instances still running. Because the linux kernel may load-balance between old and new haproxy processes, and then you have obviously strange results when those processes are from old configurations.

Stopping haproxy, making sure that no haproxy process is running at all, and then starting haproxy cleanly would guarantee you have a clean start.

Otherwise, yes, a tcpdump capture between haproxy and your backend is what will show what really happens.