Haproxy routing to invalid backend under the load

Help!
1

Hi,

During a load we are observing when the TCP connections are high haproxy is sending traffic to invalid backend but from the logs i don’t have proof to show. In our setup, we run nginx server on the same node to render static files/responses. Within haproxy config we have backend rule to route to the localhost nginx server & other backend rules to the proper app backend. Under the load instead of sending traffic to app backend, few requests are incorrectly routed to nginx & resulting in 404 responses. Did anyone else notice this issue?

We are running haproxy, version info here…

HA-Proxy version 1.9.4 2019/02/06 - https://haproxy.org/
Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_LIBCRYPT=1 USE_ZLIB=1 USE_DL=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE
              h2 : mode=HTTP       side=FE
       <default> : mode=HTX        side=FE|BE
       <default> : mode=TCP|HTTP   side=FE|BE

Available filters :
	[SPOE] spoe
	[COMP] compression
	[CACHE] cache
	[TRACE] trace

Here is config snippet

  daemon
  log 127.0.0.1:5432 local0 info
  log 127.0.0.1:5433 local0 emerg notice
  log /dev/log local0 info
  log /dev/log local0 emerg notice
  maxconn 200000
  stats socket /var/run/haproxy/socket mode 600 level admin
  stats socket /var/run/haproxy/stats mode 640 group telegraf level operator
  stats timeout 2m
  server-state-file global
  server-state-base /var/state/haproxy/
  tune.bufsize 32768

defaults
   load-server-state-from-file global
   mode http
   log global
   retries 3
   backlog 10000
   timeout http-request 10s
   timeout queue 1m
   timeout connect 10s
   timeout client 135s
   timeout server 135s
   timeout tunnel 3600s
   timeout http-keep-alive 10s
   timeout check 10s
   timeout tarpit 60s
   option httplog
   option dontlognull
   option http-server-close
   option http-ignore-probes
   option redispatch
   maxconn 120000

frontend http-in
  bind *:80

  acl url_mypath path_reg /path
  use_backend app-backend if url_mypath
  default_backend nginx-local

backend nginx-local
  server nginxlocal 127.0.0.1:81 check inter 1m

backend app-backend
 server myapp myapp-lb:80 check inter 1m
2

@lukastribus @willy can you pls suggest

3

Can you try with nbthread 1 ?

4

@lukastribus Thanks for your suggestion, I will test more with this config change & update .