Hi there,
We are experiencing a short downtime on every reload since about a month. I’m currently running HAProxy 2.7.8 version. Currently not upgrading to latest version since it gives a few seconds of downtime and the issue also occured on the previous (2.7.7) version
haproxy -vv
HAProxy version 2.7.8-1ppa1~focal 2023/05/06 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2024.
Known bugs: http://www.haproxy.org/bugs/bugs-2.7.8.html
Running on: Linux 5.15.0-1020-azure #25~20.04.1-Ubuntu SMP Thu Sep 1 19:20:56 UTC 2022 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = cc
CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-8xgY6F/haproxy-2.7.8=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_PROMEX=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=4).
Built with OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020
Running on OpenSSL version : OpenSSL 1.1.1f 31 Mar 2020
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.34 2019-11-21
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 9.4.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
I’ve provided a heavily sanitized log and config(sorry). I’ve removed some backend and not all acls might line up. The config throws no errors in haproxy usually.
What I see happening, is that when the backend is being reloaded. All health checks fail on layer 6. Though we have layer 7 health checks(which normally succeed in 5-20ms)
Also we load the previous serverstate so the state of any server should have been up.
I’ve check the permissions on the currentserverstate to make sure it can be read by haproxy. Now the only real difference is that we have much longer keepalive than before so on a normal day we have about 60/70.000 concurrent connections. With a connection rate of about 50/sec max.
I cannot for the life of me figure out what the issue is, so would really like some suggestions.
global
master-worker
maxconn 100000
tune.ssl.cachesize 1000000
tune.ssl.lifetime 300s
log /dev/log local0
chroot /var/lib/haproxy
stats socket /var/run/haproxy/dyna.sock mode 600 level operator user dtuser
#DYNATRACE.SKIP
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners user SomeAdmin
stats timeout 30s
user haproxy
group haproxy
daemon
# option to load the current server state from a file
server-state-file /etc/haproxy/currentservers.state
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
mode http
option httpslog
option dontlognull
option dontlog-normal
option redispatch
option tcp-smart-accept
option tcp-smart-connect
timeout connect 6s
timeout client 300s
timeout server 300s
timeout http-request 10s
timeout http-keep-alive 120s
load-server-state-from-file global
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend fe_stats
maxconn 5
bind *:8080
option http-server-close
stats enable
stats show-legends
stats scope fe_main
stats scope be_demo
stats auth admin:whZu0Ad0LNBMBEiyeLzwBsjWp5VaXIznOW0duKBs
stats admin if TRUE
stats uri /stats
stats refresh 15s
stats admin if LOCALHOST
frontend metrics
bind :8404
mode http
stats enable
stats uri /
stats refresh 5s
http-request use-service prometheus-exporter if { path /metrics }
frontend fe_main
maxconn 60000
bind *:443 ssl crt /etc/haproxy/certs ssl-min-ver TLSv1.2 alpn h2,http/1.1 accept-proxy
bind *:80 accept-proxy
rate-limit sessions 1500
capture cookie .Identity len 63
timeout http-keep-alive 120s
stick-table type ip size 100k expire 1m store http_req_rate(1m)
## Acls ##
acl adminpage path -i -m str /admin
acl cf_officeip req.hdr(cf-connecting-ip) 89.146.58.82
acl officeip src 89.146.58.82
## End ACLs ##
http-request redirect scheme https code 301 unless { ssl_fc }
# Set Forwarding headers
# log suspicious agents
http-request set-log-level notice if is_suspicious_agent
http-request capture req.fhdr(User-Agent) len 100 if is_suspicious_agent
# We remove some reply headers that provide unnecessary information
http-response del-header server
http-response del-header x-powered-by
# If queueing occurs, prefer GET over POST
http-request set-priority-class int(100) if METH_POST
http-response set-header Referrer-Policy no-referrer-when-downgrade
#URL based backend selection
use_backend %[req.hdr(host),lower,map(/etc/haproxy/maps/urlbackendselect.map)] if { req.hdr(host),lower,map(/etc/haproxy/maps/urlbackendselect.map) -m found }
# Default backend if nothing selected
default_backend be_webaspcore
backend be_webaspcore
option httpchk
http-check send meth GET hdr Host portal.somecompany.online
default-server ssl ca-file /etc/ssl/certs/somecompanyroot-2041.cer check-sni portal.somecompany.online verify required check sni str(portal.somecompany.online) slowstart 120s on-marked-down shutdown-sessions check-alpn http/1.1 weight 100 maxconn 100 maxqueue 100 alpn h2
retry-on conn-failure
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header X-Forwarded-Port 443
http-request set-header X-Forwarded-Proto https
http-request set-header Host portal.somecompany.online
http-response set-header Content-Security-Policy "script-src * 'unsafe-inline' 'unsafe-eval'"
cookie XServerX insert maxlife 9h maxidle 1h nocache
server XX-IG-WEB001 10.0.11.4:443 cookie 601
server XX-IG-WEB002 10.0.11.5:443 cookie 602
server XX-IG-WEB003 10.0.11.6:443 cookie 603
server XX-IG-WEB004 10.0.11.7:443 cookie 604
server XX-IG-WEB006 10.0.11.9:443 cookie 606
server XX-IG-WEB007 10.0.11.10:443 cookie 607
server XX-IG-WEB008 10.0.11.11:443 cookie 608
server XX-IG-WEB009 10.0.11.12:443 cookie 609
server XX-IG-WEB010 10.0.11.13:443 cookie 610
server XX-IG-WEB011 10.0.11.14:443 cookie 611
server XX-IG-WEB012 10.0.11.15:443 cookie 612
server XX-IG-WEB013 10.0.11.16:443 cookie 613
server XX-IG-WEB014 10.0.11.17:443 cookie 614
server XX-IG-WEB015 10.0.11.18:443 cookie 615
server XX-IG-WEB016 10.0.11.19:443 cookie 616
server XX-IG-WEB017 10.0.11.20:443 cookie 617
server XX-IG-WEB018 10.0.11.21:443 cookie 618
server XX-IG-WEB019 10.0.11.22:443 cookie 619
server XX-IG-WEB020 10.0.11.23:443 cookie 620
Haproxy log
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy fe_stats stopped (cumulated conns: FE: 83, BE: 0).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy metrics stopped (cumulated conns: FE: 3, BE: 0).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy fe_main stopped (cumulated conns: FE: 32544008, BE: 0).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy be_webaspcore stopped (cumulated conns: FE: 0, BE: 242733935).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy be_webhooks stopped (cumulated conns: FE: 0, BE: 23706281).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy be_scimapi stopped (cumulated conns: FE: 0, BE: 167753).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy be_demo stopped (cumulated conns: FE: 0, BE: 95549).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy be_webhooksdemo stopped (cumulated conns: FE: 0, BE: 3925).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy be_scimdemoapi stopped (cumulated conns: FE: 0, BE: 10202).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy be_octopus stopped (cumulated conns: FE: 0, BE: 2790).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy be_octopus2 stopped (cumulated conns: FE: 0, BE: 4799).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy static_security stopped (cumulated conns: FE: 0, BE: 36).
Jun 6 11:27:43 XX-IG-HAPROXY001 haproxy[3213129]: Proxy <HTTPCLIENT> stopped (cumulated conns: FE: 0, BE: 0).
Jun 6 11:27:57 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB007 is DOWN, reason: Layer6 timeout, check duration: 2014ms. 14 active and 0 backup servers left. 100 sessions active, 86 requeued, 0 remaining in queue.
Jun 6 11:27:57 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB002 is DOWN, reason: Layer6 timeout, check duration: 2000ms. 13 active and 0 backup servers left. 100 sessions active, 67 requeued, 0 remaining in queue.
Jun 6 11:27:57 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB004 is DOWN, reason: Layer6 timeout, check duration: 2000ms. 12 active and 0 backup servers left. 100 sessions active, 54 requeued, 0 remaining in queue.
Jun 6 11:27:57 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB003 is DOWN, reason: Layer6 timeout, check duration: 2000ms. 11 active and 0 backup servers left. 100 sessions active, 99 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB001 is DOWN, reason: Layer6 timeout, check duration: 2001ms. 10 active and 0 backup servers left. 100 sessions active, 58 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB008 is DOWN, reason: Layer6 timeout, check duration: 2005ms. 9 active and 0 backup servers left. 100 sessions active, 89 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB006 is DOWN, reason: Layer6 timeout, check duration: 2005ms. 8 active and 0 backup servers left. 100 sessions active, 80 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB009 is DOWN, reason: Layer6 timeout, check duration: 2001ms. 7 active and 0 backup servers left. 100 sessions active, 97 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB010 is DOWN, reason: Layer6 timeout, check duration: 2001ms. 6 active and 0 backup servers left. 100 sessions active, 89 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB011 is DOWN, reason: Layer6 timeout, check duration: 2001ms. 5 active and 0 backup servers left. 100 sessions active, 74 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB012 is DOWN, reason: Layer6 timeout, check duration: 2003ms. 4 active and 0 backup servers left. 100 sessions active, 60 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB013 is DOWN, reason: Layer6 timeout, check duration: 2018ms. 3 active and 0 backup servers left. 98 sessions active, 71 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB014 is DOWN, reason: Layer6 timeout, check duration: 2000ms. 2 active and 0 backup servers left. 99 sessions active, 85 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB016 is DOWN, reason: Layer6 timeout, check duration: 2007ms. 1 active and 0 backup servers left. 100 sessions active, 74 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB018 is DOWN, reason: Layer6 timeout, check duration: 2006ms. 0 active and 0 backup servers left. 100 sessions active, 63 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: backend be_webaspcore has no server available!
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooks/XX-IG-WH001 is DOWN, reason: Layer6 timeout, check duration: 2012ms. 1 active and 0 backup servers left. 20 sessions active, 0 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooks/XX-IG-WH002 is DOWN, reason: Layer6 timeout, check duration: 2014ms. 0 active and 0 backup servers left. 20 sessions active, 0 requeued, 0 remaining in queue.
Jun 6 11:27:58 XX-IG-HAPROXY001 haproxy[65792]: backend be_webhooks has no server available!
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB003 is UP, reason: Layer7 check passed, code: 200, check duration: 13ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB004 is UP, reason: Layer7 check passed, code: 200, check duration: 18ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB002 is UP, reason: Layer7 check passed, code: 200, check duration: 25ms. 3 active and 0 backup servers online. 7 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB006 is UP, reason: Layer7 check passed, code: 200, check duration: 26ms. 4 active and 0 backup servers online. 3 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_octopus/XX-IG-OCTO is UP, reason: Layer7 check passed, code: 302, check duration: 24ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB010 is UP, reason: Layer7 check passed, code: 200, check duration: 14ms. 5 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB014 is UP, reason: Layer7 check passed, code: 200, check duration: 12ms. 6 active and 0 backup servers online. 6 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_demo/XX-IG-WEB002 is UP, reason: Layer7 check passed, code: 200, check duration: 20ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_demo/XX-IG-WEB001 is UP, reason: Layer7 check passed, code: 200, check duration: 17ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooksdemo/XX-IG-WH001 is UP, reason: Layer7 check passed, code: 200, check duration: 11ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB009 is UP, reason: Layer7 check passed, code: 200, check duration: 25ms. 7 active and 0 backup servers online. 6 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_scimdemoapi/XX-IG-SCIM02 is UP, reason: Layer7 check passed, code: 403, check duration: 16ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB011 is UP, reason: Layer7 check passed, code: 200, check duration: 12ms. 8 active and 0 backup servers online. 6 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB016 is UP, reason: Layer7 check passed, code: 200, check duration: 14ms. 9 active and 0 backup servers online. 6 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooks/XX-IG-WH002 is UP, reason: Layer7 check passed, code: 200, check duration: 10ms. 1 active and 0 backup servers online. 2 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB007 is UP, reason: Layer7 check passed, code: 200, check duration: 22ms. 10 active and 0 backup servers online. 7 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooks/XX-IG-WH001 is UP, reason: Layer7 check passed, code: 200, check duration: 22ms. 2 active and 0 backup servers online. 2 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB008 is UP, reason: Layer7 check passed, code: 200, check duration: 26ms. 11 active and 0 backup servers online. 6 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB001 is UP, reason: Layer7 check passed, code: 200, check duration: 25ms. 12 active and 0 backup servers online. 6 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_scimapi/XX-IG-SCIM02 is UP, reason: Layer7 check passed, code: 403, check duration: 14ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_scimapi/XX-IG-SCIM01 is UP, reason: Layer7 check passed, code: 403, check duration: 32ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB012 is UP, reason: Layer7 check passed, code: 200, check duration: 15ms. 13 active and 0 backup servers online. 6 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooksdemo/XX-IG-WH002 is UP, reason: Layer7 check passed, code: 200, check duration: 17ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_scimdemoapi/XX-IG-SCIM01 is UP, reason: Layer7 check passed, code: 403, check duration: 29ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB013 is UP, reason: Layer7 check passed, code: 200, check duration: 37ms. 14 active and 0 backup servers online. 6 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_octopus2/XX-IG-OCTO02 is UP, reason: Layer7 check passed, code: 302, check duration: 27ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:28:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB018 is UP, reason: Layer7 check passed, code: 200, check duration: 10ms. 15 active and 0 backup servers online. 6 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_demo/XX-IG-WEB002 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_demo/XX-IG-WEB001 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooksdemo/XX-IG-WH001 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_scimdemoapi/XX-IG-SCIM02 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooks/XX-IG-WH002 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooks/XX-IG-WH001 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_scimapi/XX-IG-SCIM02 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_scimapi/XX-IG-SCIM01 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webhooksdemo/XX-IG-WH002 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:29:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_scimdemoapi/XX-IG-SCIM01 is UP. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB003 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB004 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB002 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB006 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB010 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB014 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB009 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB011 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB016 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB007 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB008 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB001 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB012 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB013 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Jun 6 11:30:06 XX-IG-HAPROXY001 haproxy[65792]: Server be_webaspcore/XX-IG-WEB018 is UP. 15 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
edit:
I’ve checked we do not run out of memory on the server, during the reload there is about 1.63GB memory free and 5.34 available