Rapid SSH login/logout sequence cause SSH Frontend service unavailable for few seconds

Help!
1

I have 2 backend server where SSH being proxy through haproxy.
When having rapid SSH login/logout sequence, haproxy will throw connection reset error.
It normally happen after the 19th successful SSH attempt

kex_exchange_identification: Connection closed by remote host

It looks like some protection mechanism. But, I could not find any relevant default setting from the guide.

The configuration is fairly simple. shown below:

global
  log 127.0.0.1:514 local0
  stats socket /var/lib/haproxy/haproxy-stats level admin
  chroot /var/lib/haproxy
  maxconn 20000
  user haproxy
  group haproxy
  daemon
  # POODLE Hardening
  ssl-default-bind-options  no-sslv3 no-tls-tickets
  ssl-default-bind-ciphers  ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS

peers LB
  peer haproxy01 192.168.0.1:8443
  peer haproxy02 192.168.0.2:8443

defaults
  log global
  mode  http
  option httplog
  option logasap
  option  dontlognull
        timeout connect 5000
        timeout client 50000
        timeout server 50000

frontend P22-fe
    bind 192.168.100.1:22
    mode tcp
    option tcplog
    default_backend P22-be

backend P22-be
    mode tcp
    balance roundrobin
    server 192.168.1.3 192.168.1.3:22 check
    server 192.168.1.4 192.168.1.4:22 check

Appreciate if someone can point me to the right direction.