HAProxy community

Servers number limit for balancing algorithms

We use HAProxy to balance the load between hundreds of servers, however, there is an issue which is new for us and unexpected (works in any HAProxy version). When number of servers in some backend exceeds 100, all weights for active servers are set to zero (marked as SOFT STOPPED) and backend goes down. The only way to make backend UP again is to set server count to 100 or less. We may need to run hundreds of servers under one backed and distribute the load. How we can adjust this number?

Share the output of haproxy -vv, the configuration and explain how you add/remove servers here (are you adjusting the configuration and reloading haproxy, are you using DNS discovery, etc).

Okay, so output of haproxy -vv

HA-Proxy version 2.2.1 2020/07/23 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.2.1.html
Running on: Linux 4.9.0-11-amd64 #1 SMP Debian 4.9.189-3+deb9u2 (2019-11-11) x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1

Feature list : +EPOLL -KQUEUE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
Running on OpenSSL version : OpenSSL 1.1.1d  10 Sep 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.32 2018-09-10
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.3.0
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
              h2 : mode=HTTP       side=FE|BE     mux=H2
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services :
	prometheus-exporter

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace
	[CACHE] cache
	[FCGI] fcgi-app

A configuration example is the next:

global
  maxconn 100000
  stats socket /var/run/haproxy.sock mode 660 level admin
  stats timeout 30s
  debug

defaults
  log	global
  mode	http
  option	httplog
  option	dontlognull
  option  forwardfor
  timeout connect 5000
  timeout client  50000000
  timeout server  50000000

resolvers localdns
  nameserver dns1 <ip>:53
  accepted_payload_size 8192
  hold timeout         600s
  hold refused         600s

frontend http-in
  bind *:8080 accept-proxy
  mode http

  redirect scheme https code 301 if !{ ssl_fc }

frontend https-in
  bind *:8433 accept-proxy
  mode http

  # Define hosts
  acl host_acl hdr(host) -i <address>

  # Figure out which one to use
  use_backend host if host_acl

backend host
  http-request set-header X-Real-IP %[src];
  http-request set-header X-Forwarded-For %[src];
  http-request set-header X-Forwarded-Proto %[src];
  http-request set-header Connection "upgrade"
  http-request set-header Host %[src]

  balance leastconn
  stick-table type string len 80 size 1m expire 8h
  option tcp-check
  stick on url_param(mrid)
  server-template srv 100 _http._tcp.<pod>.<ns>.svc.cluster.local resolvers localdns check inter 1000

As you can see, we are using DNS discovery.

Any logging happening at the same time? Could it be related to the dns response hitting the accepted_payload_size threshold?

No valuable logs. Will try to debug further today. Servers are being updated even if we have 500 of them, but with 0 weight. So, problem is definitely not in DNS Payload size, moreover 8k is the maximum here.

What usually happened when new server is added (1-100):

[WARNING] 212/080533 (7) : Server host/srv48 is UP, reason: Layer4 check passed, check duration: 0ms. 48 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

Here what if happened when I add server number 101:

[ALERT] 212/080658 (7) : backend 'host' has no server available!
[WARNING] 212/080658 (7) : host/srv101 changed its IP from  to <IP> by DNS additional recrd.
[WARNING] 212/080700 (7) : Server host/srv101 is UP, reason: Layer4 check passed, check duration: 0ms. 0 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

After that in 1 minute worker dies :frowning:

[ALERT] 212/080908 (1) : Current worker #1 (7) exited with code 139 (Segmentation fault)

8192 bytes, not servers, and with this numbers it’s not completely unlikely that we hit it. Crossing the threshold doesn’t necessarily mean that no servers are updated, especially if we hit a bug.

A Haproxy worker is crashing here, so this is clearly a bug.

Could you file a bug: