<NOSRV><SC> issue on haproxy server

Help!
1

Hi All,
We are observing intermittent error code “NOSRV” “SC” on haproxy server and which in turns throws error “ssh_exchange_identification” to Jenkins build Business user (build server connecting to Gerrit Master server via HAproxy server)

Looks like, at some moment, consolidated gerrit master server is not able to accept more connections from client and starts refusing it with “NOSRV”“SC” code. This error is happening almost on daily basis affecting builds.

Jun 25 13:26:19 localhost haproxy[44775]: xx:xx.xx.xx:40662 [25/Jun/2020:xx:xx:xx.xx] ssh-in gerrit-ssh-servers/ -1/-1/0 0 SC 1728/382/385/0/0 0/0 -:-
Jun 25 13:26:19 localhost haproxy[44775]: xx:xx.xx.xx:40664 [25/Jun/2020:xx:xx:xx.xx] ssh-in gerrit-ssh-servers/ -1/-1/0 0 SC 1729/381/384/0/0 0/0 -:-
Jun 25 13:26:19 localhost haproxy[44775]: xx:xx.xx.xx:40666 [25/Jun/2020:xx:xx:xx.xx] ssh-in gerrit-ssh-servers/ -1/-1/0 0 SC 1729/380/383/0/0 0/0 -:-

global
log 127.0.0.1 local2 debug
pidfile /var/run/haproxy.pid
maxconn 64000
user gerrit

defaults
#mode tcp
retries 3
timeout connect 60000ms
timeout client 600000ms
timeout server 600000ms

frontend ssh-in
log 127.0.0.1 local2 debug
mode tcp
bind *:29418
default_backend gerrit-ssh-servers
option tcplog
#maxconn 2000
log-format “%ci:%cp [%t] %ft %b/%s %Tw/%Tc/%Tt %B %ts %ac/%fc/%bc/%sc/%rc %sq/%bq %bi:%bp”

backend gerrit-ssh-servers
log 127.0.0.1 local2 debug
option redispatch
server gerritM1 xx.xx.xx.xx:29418 check

listen status
bind 0.0.0.0:1080
mode http
log global
stats refresh 30s
stats uri /haproxy?stats
stats realm Private lands
stats auth admin:password
stats hide-version

Regards,
Madhurendra

2

Haproxy cannot reach the server:

 SC   The server or an equipment between it and haproxy explicitly refused
          the TCP connection (the proxy received a TCP RST or an ICMP message
          in return). Under some circumstances, it can also be the network
          stack telling the proxy that the server is unreachable (e.g. no route,
          or no ARP response on local network). When this happens in HTTP mode,
          the status code is likely a 502 or 503 here.