HAProxy community

Choosing backend based on tcp payload


My company has multiple development environments based on multiple features being tested. Instead of everyone maintain and point to each other’s respective feature environment, I am thinking of setting up a whitelisted user based routing setup. For example:

This is easier to do for http requests because haproxy / nginx easily support routing http requests to backends based on the value of say a user_id header.
Things get messy when you want to do this for binary protocols like thrift.

Have written a custom router config in haproxy which I wanted to share.

frontend thriftrouter
  bind *:10090
  mode tcp
  tcp-request inspect-delay 100ms
  option tcplog
  log global
  log-format "%ci:%cp -> %fi:%fp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq captured_user:%{+Q}[capture.req.hdr(0)] req.len:%[capture.req.hdr(1)]"

  tcp-request content capture req.payload(52,10) len 10
  tcp-request content capture req.len len 10
  tcp-request content accept if WAIT_END

  acl acl_thrift_call    req.payload(2,2)  -m bin 0001		# Thrift CALL method
  acl acl_magic_field_id req.payload(30,2) -m bin 270f		# Magic field number 9999

  # Define access control list for each user
  acl acl_user_u1 req.payload(0,0),hex -m sub 7573657231  # user1
  acl acl_user_u2 req.payload(0,0),hex -m sub 7573657232  # user2

  # Route based on the user. No default backend so that one always has to set it
  use_backend backend_1 if acl_user_u1 acl_magic_field_id acl_thrift_call
  use_backend backend_2 if acl_user_u2 acl_magic_field_id acl_thrift_call

backend backend_1
  mode tcp
  option tcp-check
  server server1
backend backend_2
  mode tcp
  option tcp-check
  server server2

So now the only missing piece is the java code to generate custom metadata into thrift.

 * Injects custom data into the egress thrift call.
 * @author Gagandeep Singh
public class CustomInjectorProtocol extends TProtocolDecorator {
    private static final short FIELD_ID = 9999; // Magic number - this will be field id of custom data

    private final Function<Void, Map<String, String>> function;
    public CustomInjectorProtocol(TProtocol protocol, Function<Void, Map<String, String>> function) {
        this.function = function;

    public void writeMessageBegin(TMessage tMessage) throws TException {

        Map<String, String> map;
        try {
            map = function.apply(null);
            log.info("Injecting custom data into egress thrift call");
        } catch (Exception e) {
            log.error("Error in applying function: {}", e.getMessage());
            log.debug("Exception: ", e);
            map = new HashMap<>();

        super.writeFieldBegin(new TField("custom_data", TType.MAP, FIELD_ID));
        super.writeMapBegin(new TMap(TType.STRING, TType.STRING, map.size()));
        for (Map.Entry<String, String> entry : map.entrySet()) {

Set the user_id when making the thrift call.

TTransport transport = new TSocket("localhost", 10090, 1000000);

TProtocol protocol = new TBinaryProtocol(transport);
TProtocol spanProtocol = new CustomInjectorProtocol(protocol, (v) -> {
    Map<String, String> map = Maps.newHashMap();
    map.put("userid", "|user1|");
    return map;
client = new Service.Client(spanProtocol);