TCP fast open is not always applied

To accelerate L4 performance between origin and end-user, I’ve utilized TCP fast open feature on HAProxy.

My expectation is that all requests are accelerated by TCP fast open. But on the result, all is not always accelerated like image.

I want to know how TCP fast open is always applied to all requests.
Are there any other configuration for doing it ?

My test environment is as follow. And I’ve configured TCP fast open at Edge and Relay.


Client(Google VM on Singapore) -> Edge (AWS VM on Singapore) -> Parent (AWS VM on KR) -> Origin(AWS VM on KR)

Test program was run periodically on the client. ex) curl http://www.foo.com:90/10K.bin

Linux version and TCP fast open of Edge & Parent server.


# uname -a
Linux edge_sg_01 4.14.177-139.254.amzn2.x86_64 #1 SMP Thu May 7 18:48:23 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
# cat /proc/sys/net/ipv4/tcp_fastopen
3

Haproxy -vv


HA-Proxy version 2.1.7 2020/06/09 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2021.
Known bugs: http://www.haproxy.org/bugs/bugs-2.1.7.html
Running on: Linux 4.14.177-139.254.amzn2.x86_64 #1 SMP Thu May 7 18:48:23 UTC 2020 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE=1 USE_THREAD=1 USE_LIBCRYPT=1 USE_OPENSSL=1 USE_ZLIB=1 USE_TFO=1 USE_NS=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +BACKTRACE +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as  cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2
            fcgi : mode=HTTP       side=BE        mux=FCGI
        : mode=HTTP       side=FE|BE     mux=H1
        : mode=TCP        side=FE|BE     mux=PASS

Available services : none

Available filters :
	[SPOE] spoe
	[CACHE] cache
	[FCGI] fcgi-app
	[TRACE] trace
	[COMP] compression

Configuration on Parent


frontend tcp_90
    mode     tcp
    bind *:90 tfo
   default_backend web_svr_90

backend tcp_svr_90
    mode   tcp
    server  kr_origin_90  xxx.xxx.xxx.xxx:90

Configuration on Edge


frontend tcp_90
    mode     tcp
    bind *:90
    default_backend tcp_svr_90

backend tcp_svr_90
    mode      tcp
   retry-on conn-failure empty-response response-timeout
    server  kr_parent_tcp_00  xxx.xxx.xxx.xxx:90 tfo

That’s not how TCP fast open works, it will never accelerated everything.

Thanks for your reply.

Is there any additional option or tuning point for applying TCP fast open into more TCP connections ? I just used only “tfo option” on “bind *:90 tfo”.

No, there is not.

What you should really be focusing on is not closing TCP connections in the first place, and reusing them. Checkout http keepalive and http-reuse functionality, also haproxy 2.2 (just released) will reuse existing connections across threads.

Thanks.
Actually, I want to accelerate TCP-based application not HTTP and HTTPS-based. So, I’m searching whether TCP fast open or TCP connection pool is supported or not on HAProxy.

TCP connection pooling is impossible by definition. You can only use a single TCP session for multiple transactions, if you understand the protocol (like HTTP).

TCP Fast Open should work in TCP only mode. But like I said you will never achieve 100% coverage, because that is not what TCP Fast Open does.