HAProxy and TCP Mode not working correctly on Layer 7

Help!
1

For some reason, I have to use tcp mode even though I’m making all calls over HTTP (read: Docker Engine API)

I got into a very specific problem, and after hours of debugging, I found out that it seems some TCP connections are forwarded as it is by HAProxy to a backend without going through frontend block. To support this, a lot of times I don’t even see logs in TCP mode when I make new server requests. Here’s a working config which works absolutely fine:

defaults
    log     global
    mode    http
    option tcplog
    timeout connect 5s
    timeout client 5s
    timeout server 5s

frontend www
    bind :2376
    tcp-request inspect-delay 100ms
    acl headertest hdr(sessionid) -i test
    tcp-request content accept if headertest
    tcp-request content reject
    default_backend servers

This works fine if I want to block any request with sessionid header other than ‘test’. Now consider the tcp mode config file:

defaults
    log     global
    mode    tcp
    option tcplog
    timeout connect 5s
    timeout client 5s
    timeout server 5s

frontend www
    bind :2376
    tcp-request inspect-delay 100ms
    acl headertest hdr(sessionid) -i test
    tcp-request content accept if headertest
    tcp-request content reject
    default_backend servers

Reproducing this bug:

  1. Switch it to TCP
  2. First I give an HTTP request with sessionid ‘test1’ which fails would fail because of condition.
  3. Next I give it HTTP request with sessionid ‘test’ which passes, as expected
  4. Finally give it a HTTP request ‘test1’ again, it passes! It shows the webpage, and no log entry is generated, although I can verify that the request was received on backend.

Check this:

Screenshot 2020-02-15 at 7.25.12 PM
Screenshot 2020-02-15 at 7.25.12 PM1792×1914 466 KB

ALSO: NO LOG ENTRY IS GENERATED FOR THE 3rd request, hence, I believe it never passes through the frontend block.

Why is that? I’ve spent a lot of hours reading the docs, and this should not be the behavior. Clearly, it states that it is possible to read HTTP data in TCP mode in docs (HAProxy version 2.0.28 - Configuration Manual) as the docs say and I quote:

It is perfectly possible to match layer 7 contents with “tcp-request content” rules, since HTTP-specific ACL matches are able to preliminarily parse the contents of a buffer before extracting the required data.