'mode tcp' performace degradation with two different backends

lukastribus · June 10, 2026, 11:45am

You have already filed a bug on Github about this:

'mode tcp' performace degradation with two different backends

opened 01:37PM - 08 Jun 26 UTC

type: bug status: needs-triage

### Detailed Description of the Problem I’m trying to test new connections rate… in 'mode tcp' with the following config ``` frontend default_fe mode tcp bind *:443 default_backend default_be backend default_be mode tcp balance roundrobin # three autonomous apache servers server s3_v4 127.0.0.1:8443 server s1_v4 172.16.37.1:8443 server s2_v4 172.16.37.2:8443 ``` using and command: `wrk -t12 -c2500 -d10s -H “Connection: close” https://my_site:443/` and the problem is: - when I’m using the single backend (either local or remote), RPS is about 10-11k - when I’m using two or more different backend (in any combination) the performance degrades to 2-3k RPS Backends performance is ok - when I’m running three simultaneous wrk to all three backends, every of them gives ~10-11k RPS The problem appears with either `leastconn` or `roundrobin` or even `random` algo and not present with `source` or `first` - so as soon as haproxy needs to balance between two and more different destinations, it degrades. **Examples** Single backend: ``` root@px:~# wrk -t12 -c2500 -d10s -H "Connection: close" https://xxxxxxxxxxx.com:443/ Running 10s test @ https://xxxxxxxxxxx.com:443/ 12 threads and 2500 connections Thread Stats Avg Stdev Max +/- Stdev Latency 24.96ms 36.87ms 1.91s 95.62% Req/Sec 0.90k 293.26 1.58k 73.94% 100147 requests in 10.09s, 42.98MB read Requests/sec: 9926.99 Transfer/sec: 4.26MB ``` Two backends: ``` root@px:~# wrk -t12 -c2500 -d10s -H "Connection: close" https://xxxxxxxxxxx.com:443/ Running 10s test @ https://xxxxxxxxxxx.com:443/ 12 threads and 2500 connections Thread Stats Avg Stdev Max +/- Stdev Latency 93.16ms 122.31ms 1.09s 87.70% Req/Sec 353.96 548.95 2.07k 84.66% 24908 requests in 10.17s, 10.69MB read Socket errors: connect 0, read 0, write 0, timeout 1 Requests/sec: 2449.70 Transfer/sec: 1.05MB ``` ### Expected Behavior Regardless of set of backends, haproxy performance does not degrade. ### Steps to Reproduce the Behavior 1. Create config like "main test" section in the config below 2. Use wrk to test the scenarios: - single backend - two different backends 3. Check the difference in the output ### Do you have any idea what may have caused this? _No response_ ### Do you have an idea how to solve the issue? _No response_ ### What is your configuration? ```haproxy global hard-stop-after 1m log /dev/log local0 chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners stats timeout 30s user haproxy group haproxy daemon # see also `systemctl edit haproxy` maxconn 256000 cpu-policy performance # update shards below in listener to == nbthread nbthread 16 cpu-map 1/1 4 cpu-map 1/2 32 cpu-map 1/3 5 cpu-map 1/4 33 cpu-map 1/5 6 cpu-map 1/6 34 cpu-map 1/7 7 cpu-map 1/8 35 cpu-map 1/9 8 cpu-map 1/10 36 cpu-map 1/11 9 cpu-map 1/12 37 cpu-map 1/13 10 cpu-map 1/14 38 cpu-map 1/15 11 cpu-map 1/16 39 # Default SSL material locations ca-base /etc/ssl/certs crt-base /etc/ssl/private #--- Generic recommendation are at https://ssl-config.mozilla.org #--- Performance-ficused tuning ssl-default-bind-curves X25519:prime256v1:secp384r1 # TLS 1.2: Removed DHE (slow), priority: ECDSA+AES128 -> RSA+AES128 -> ChaCha -> AES256 ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SH # TLS 1.3: AES-128 (fastest for AES-NI) -> ChaCha (mobiles w/o AES-NI) -> AES-256 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_256_GCM_SHA384 ssl-default-bind-options ssl-min-ver TLSv1.2 # Backend-side TLS defaults ssl-default-server-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256 ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384 ssl-default-server-options ssl-min-ver TLSv1.2 defaults option dontlognull mode http http-reuse always backlog 65536 retries 3 option redispatch option nolinger timeout connect 5s timeout client 35s timeout client-fin 40s timeout server 35s timeout server-fin 40s timeout http-request 5s timeout http-keep-alive 15s frontend ident_fe mode tcp bind *:443 shards 16 default_backend default_ident_be backend default_ident_be mode tcp fullconn 120000 option tcp-check default-server check inter 3s rise 2 fall 2 balance roundrobin server ident3_v4 127.0.0.1:13443 server ident1_v4 172.16.37.1:13443 server ident2_v4 172.16.37.2:13443 ``` ### Output of `haproxy -vv` ```plain HAProxy version 3.2.19-0+ha32+ubuntu24.04u1 2026/05/11 - https://haproxy.org/ Status: long-term supported branch - will stop receiving fixes around Q2 2030. Known bugs: http://www.haproxy.org/bugs/bugs-3.2.19.html Running on: Linux 6.8.0-124-generic #124-Ubuntu SMP PREEMPT_DYNAMIC Tue May 26 13:00:45 UTC 2026 x86_64 Build options : TARGET = linux-glibc CC = x86_64-linux-gnu-gcc CFLAGS = -O2 -g -fwrapv -fvect-cost-model=very-cheap -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/builds/haproxy-ce/deb-haproxy-awslc/debian/output/source_dir=/usr/src/haproxy-awslc-3.2.19-0+ha32+ubuntu24.04u1 -Wdate-time -D_FORTIFY_SOURCE=3 OPTIONS = USE_OPENSSL=1 USE_OPENSSL_AWSLC=1 USE_LUA=1 USE_SLZ=1 USE_OT=1 USE_QUIC=1 USE_PROMEX=1 USE_MEMORY_PROFILING=1 USE_PCRE2=1 USE_PCRE2_JIT=1 DEBUG = Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH +MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_AWSLC -OPENSSL_WOLFSSL +OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT +RT +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB +ACME Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_TGROUPS=32, MAX_THREADS=1024, default=56). Built with SSL library version : OpenSSL 1.1.1 (compatible; AWS-LC 3.3.0) Running on SSL library version : AWS-LC 3.3.0 SSL library supports TLS extensions : yes SSL library supports SNI : yes SSL library FIPS mode : no SSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 QUIC: connection socket-owner mode support : yes QUIC: GSO emission support : yes Built with Lua version : Lua 5.4.6 Built with the Prometheus exporter as a service Built with network namespace support. Built with OpenTracing support. Built with libslz for stateless compression. Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with PCRE2 version : 10.42 2022-12-11 PCRE2 library supports JIT : yes Encrypted password support via crypt(3): yes Built with gcc compiler version 13.3.0 Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG <default> : mode=HTTP side=FE|BE mux=H1 flags=HTX fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG spop : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG <default> : mode=SPOP side=BE mux=SPOP flags=HOL_RISK|NO_UPG none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG <default> : mode=TCP side=FE|BE mux=PASS flags= Available services : prometheus-exporter Available filters : [BWLIM] bwlim-in [BWLIM] bwlim-out [CACHE] cache [COMP] compression [FCGI] fcgi-app [ OT] opentracing [SPOE] spoe [TRACE] trace ``` ### Last Outputs and Backtraces ```plain ``` ### Additional Information Ubuntu 24.04.4 LTS (Noble Numbat) Haproxy is the official Community Performance package (https://www.haproxy.com/downloads) but tested with Vincent Bernat's PPA packages with the same result

Let’s use a single support channel, not multiple. Thank you.

Topic		Replies	Views
Haproxy session rate slower than single web server Help!	8	2511	January 5, 2018
Haproxy Remote Desktop services 2016 not balancing Help!	4	1648	February 1, 2020
Only first server on backend getting used Help!	3	924	February 10, 2023
HAProxy performance issue when proxying to gRPC Help!	12	8243	July 16, 2019
Loadbalancing TCP requests Help!	2	580	September 3, 2019

'mode tcp' performace degradation with two different backends

Related topics