Iperf3 and haproxy timeout/keepalives

Help!
1

I’ve been struggling trying to figure out why my iperf3 benchmark going through an HAProxy frontend keeps getting killed at the configured timeout value of 2 minutes even though it’s pushing over 2 gigabits/sec on each of 10 threads. If I raise the timeout values to beyond the iperf3 test duration, the benchmark runs with no issues. I guess I have 2 main questions. Why are the timeouts being applied by HAProxy even though the connections are clearly not idle? Why am I not capturing any TCP keepalive packets with tcpdump when I have the options in haproxy.cfg?

haproxy.cfg (3.1 version):
global
master-worker
log stdout format raw local0 debug
pidfile /var/run/haproxy.pid
user haproxy
group haproxy
daemon

nbthread 4
maxconn 100000
hard-stop-after 30s

stats socket /var/lib/haproxy/stats mode 660 level admin expose-fd listeners

defaults
mode tcp
timeout client 120s
timeout server 120s
timeout tunnel 120s

frontend iperf3
bind 0.0.0.0:8500
option clitcpka
default_backend iperf3.8080

backend iperf3.8080
option srvtcpka
server iperf3 10.2.3.4:8080

sysctl keepalive settings (server, proxy, and client):
net.ipv4.tcp_keepalive_intvl = 5
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_time = 10

iperf3 server command line: iperf3 -s -p 8080
iperf3 client command line: iperf3 -c $HAPROXY_IP -p 8500 -P 10 -t 180
tcpdump command: tcpdump -i eth0 -n ‘tcp[tcpflags] & (tcp-ack) != 0 and tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) == 0 and tcp[32:4] = tcp[36:4] - 1’

I feel like there has to be something really basic that I’m overlooking here. Any insight is appreciated.

2

I think I found the first answer here: Timeout client in TCP mode
One of the iperf3 threads is the control socket, and it does stop sending traffic after the other threads are up and running. So is there anything missing here that is preventing HAProxy from sending keepalive packets to that socket?

3

Haproxy timeouts are about TCP payload. TCP keepalives is a kernel based keepalive mechanism that applications (like haproxy) will not even see.

TCP keepalives help with NATs and Firewalls, closing idle connections. It does not help with haproxy timeouts and it wouldn’t make sense.

Raise your haproxy timeouts to values that make sense for you.

You can then use TCP keepalives to make sure that connections are closed after a few minutes of idle time to counteract the effect that high timeouts may have (which is having a number of connections on the proxy that are not relevant anymore).