Getting 503 errors with <NOSRV> in backend and Layer6 timeout error log

Help!
1

Hi
We are using HA Proxy v2.6.15 on Ubuntu 18.04.6 LTS and getting 503 errors in API hits. This behaviour is occurring on when loading the HA Proxy load balancer with some 100 tps load.

While checking the logs, it shows below errors:

Apr 18 06:54:08 haproxy-server-1 haproxy[29108]: [WARNING] (29112) : Server backend-server3:8081-28c6a60e is UP, reason: Layer6 check passed, check duration: 14ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Apr 18 06:54:08 haproxy-server-1 haproxy[29108]: [WARNING] (29112) : Server backend-server1:8081-28c6a60e is UP, reason: Layer6 check passed, check duration: 15ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Apr 18 06:54:08 haproxy-server-1 haproxy[29108]: [WARNING] (29112) : Server backend-server2:8081-28c6a60e is UP, reason: Layer6 check passed, check duration: 14ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
^[[CApr 18 06:54:45 haproxy-server-1 haproxy[29108]: [WARNING] (29112) : Server backend-server3:8081-28c6a60e is DOWN, reason: Layer6 timeout, check duration: 2002ms. 2 active and 0 backup servers left. 265 sessions active, 0 requeued, 0 remaining in queue.
Apr 18 06:54:45 haproxy-server-1 haproxy[29108]: [WARNING] (29112) : Server backend-server1:8081-28c6a60e is DOWN, reason: Layer6 timeout, check duration: 2003ms. 1 active and 0 backup servers left. 267 sessions active, 0 requeued, 0 remaining in queue.
Apr 18 06:54:46 haproxy-server-1 haproxy[29108]: [WARNING] (29112) : Server backend-server2:8081-28c6a60e is DOWN, reason: Layer6 timeout, check duration: 2006ms. 0 active and 0 backup servers left. 271 sessions active, 0 requeued, 0 remaining in queue.
Apr 18 06:54:46 haproxy-server-1 haproxy[29108]: [ALERT] (29112) : backend ‘bck_nodes’ has no server available!
Apr 18 06:55:03 haproxy-server-1 haproxy[29108]: [WARNING] (29112) : Server backend-server2:8081-28c6a60e is UP, reason: Layer6 check passed, check duration: 15ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Apr 18 06:55:03 haproxy-server-1 haproxy[29108]: [WARNING] (29112) : Server backend-server1:8081-28c6a60e is UP, reason: Layer6 check passed, check duration: 16ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Apr 18 06:55:03 haproxy-server-1 haproxy[29108]: [WARNING] (29112) : Server backend-server3:8081-28c6a60e is UP, reason: Layer6 check passed, check duration: 17ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

haproxy -vv Output:

HAProxy version 2.6.15-446b02c 2023/08/09 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.15.html
Running on: Linux 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021 x86_64
Build options :
TARGET = generic
CPU = generic
CC = cc
CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_PCRE=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_SYSTEMD=1 USE_PROMEX=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES -ACCEPT4 -BACKTRACE -CLOSEFROM -CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ENGINE -EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBCRYPT -LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OT +PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PROCCTL +PROMEX -QUIC -RT +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO -THREAD -THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1 11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Support for malloc_trim() is enabled.
Built without multi-threading support (USE_THREAD not set).
Built with libslz for stateless compression.
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): no
Built with gcc compiler version 7.5.0

Available polling systems :
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 2 (2 usable), will use poll.

Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
: mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
: mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG

Available services : prometheus-exporter
Available filters :
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace

haproxy.cfg

global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 30000
user haproxy
group haproxy
daemon
stats socket /tmp/haproxy.sock mode 600 level user
tune.ssl.default-dh-param 1024

ssl-dh-param-file /etc/haproxy/dhparam

ssl-default-bind-ciphers ALL:!aNULL:-ECDHE-RSA-RC4-SHA:-RC4-SHA:-RC4-MD5:-EDH-RSA-DES-CBC-SHA:-DES-CBC-SHA:-AECDH-AES256-SHA:-AECDH-AES128-SHA
ssl-default-bind-options prefer-client-ciphers no-sslv3 no-tls-tickets

ssl-default-server-ciphers ALL:!aNULL:-ECDHE-RSA-RC4-SHA:-RC4-SHA:-RC4-MD5:-EDH-RSA-DES-CBC-SHA:-DES-CBC-SHA:-AECDH-AES256-SHA:-AECDH-AES128-SHA
ssl-default-server-options no-sslv3 no-tls-tickets

defaults
log global
compression algo gzip
compression type text/html text/plain text/css text/javascript
maxconn 30000
fullconn 9000
timeout connect 60s
timeout client 600s
timeout server 600s
option dontlognull
option redispatch
log-format ‘{“client_ip”: “%ci”,“client_port”: %cp,“date_time”: “%t”,“frontend_name_transport”: “%ft”,“backend_name”: “%b”,“server_name”: “%s”,“time_waited_for_client”: %Tq,“time_spent_in_queues”: %Tw,“time_waited_for_stablish_connection”: %Tc,“time_spent_waiting_for_full_http”: %Tr,“time_between_accept_and_close”: %Tt,“status_code”: %ST,“bytes_read”: %B,“captured_request_cookie”: “%CC”,“captured_respose_cookie”: “%CS”,“termination_state_with_cookie”: “%tsc”,“actconn”: “%ac”,“feconn”: “%fc”,“beconn”: “%bc”,“srv_conn”: “%sc”,“retries”: %rc,“srv_queue”: “%sq”,“backend_queue”: “%bq”,“http_method”: “%HM”,“http_path”: “%HP”,“ssl_ciphers”: “%sslc”,“ssl_version”: “%sslv”,“tls-warn”:“%[var(txn.tlswarn)]”,“bytes_uploaded”: %U}’

frontend http
mode http
bind *:80
redirect scheme https code 301 if !{ ssl_fc }

http-response set-header Strict-Transport-Security max-age=63072000

frontend https
mode http
bind *:443 ssl crt /etc/haproxy/pem/cert.pem alpn h2,http/1.1 ssl-min-ver TLSv1.0
acl bck_mgmt path_beg /mgmt
use_backend reject_bck_mgmt if bck_mgmt
default_backend bck_nodes

http-response set-header Strict-Transport-Security max-age=63072000

backend reject_bck_mgmt
mode http
errorfile 503 /etc/haproxy/errors/404.http

backend bck_nodes
mode http
option forwardfor
acl is_ssl_version_deprecated ssl_fc_protocol SSLv3 TLSv1 TLSv1.1
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header tls-version %sslv
balance roundrobin

##ROUTES BEGIN
server backend-server1:8081-28c6a60e backend-server1:8081 weight 1 check inter 2000 rise 2 fall 2 ssl verify none
server backend-server2:8081-28c6a60e backend-server2:8081 weight 1 check inter 2000 rise 2 fall 2 ssl verify none
server backend-server3:8081-28c6a60e backend-server3:8081 weight 1 check inter 2000 rise 2 fall 2 ssl verify none

##ROUTES END

listen admin
bind *:22002
mode http
stats enable
stats uri /
stats refresh 5s

2

We are also facing this issue with upgrading the HA Proxy to this version with similar configuraiton. It is working with 1.8.8.
Looking for guidance from experts here.