Hi,
During a load we are observing when the TCP connections are high haproxy is sending traffic to invalid backend but from the logs i don’t have proof to show. In our setup, we run nginx server on the same node to render static files/responses. Within haproxy config we have backend rule to route to the localhost nginx server & other backend rules to the proper app backend. Under the load instead of sending traffic to app backend, few requests are incorrectly routed to nginx & resulting in 404 responses. Did anyone else notice this issue?
We are running haproxy, version info here…
HA-Proxy version 1.9.4 2019/02/06 - https://haproxy.org/
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_LIBCRYPT=1 USE_ZLIB=1 USE_DL=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
<default> : mode=HTX side=FE|BE
<default> : mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Here is config snippet
daemon
log 127.0.0.1:5432 local0 info
log 127.0.0.1:5433 local0 emerg notice
log /dev/log local0 info
log /dev/log local0 emerg notice
maxconn 200000
stats socket /var/run/haproxy/socket mode 600 level admin
stats socket /var/run/haproxy/stats mode 640 group telegraf level operator
stats timeout 2m
server-state-file global
server-state-base /var/state/haproxy/
tune.bufsize 32768
defaults
load-server-state-from-file global
mode http
log global
retries 3
backlog 10000
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 135s
timeout server 135s
timeout tunnel 3600s
timeout http-keep-alive 10s
timeout check 10s
timeout tarpit 60s
option httplog
option dontlognull
option http-server-close
option http-ignore-probes
option redispatch
maxconn 120000
frontend http-in
bind *:80
acl url_mypath path_reg /path
use_backend app-backend if url_mypath
default_backend nginx-local
backend nginx-local
server nginxlocal 127.0.0.1:81 check inter 1m
backend app-backend
server myapp myapp-lb:80 check inter 1m