Rapid SSH login/logout sequence cause SSH Frontend service unavailable for few seconds

I have 2 backend server where SSH being proxy through haproxy.
When having rapid SSH login/logout sequence, haproxy will throw connection reset error.
It normally happen after the 19th successful SSH attempt

kex_exchange_identification: Connection closed by remote host

It looks like some protection mechanism. But, I could not find any relevant default setting from the guide.

The configuration is fairly simple. shown below:

  log local0
  stats socket /var/lib/haproxy/haproxy-stats level admin
  chroot /var/lib/haproxy
  maxconn 20000
  user haproxy
  group haproxy
  # POODLE Hardening
  ssl-default-bind-options  no-sslv3 no-tls-tickets

peers LB
  peer haproxy01
  peer haproxy02

  log global
  mode  http
  option httplog
  option logasap
  option  dontlognull
        timeout connect 5000
        timeout client 50000
        timeout server 50000

frontend P22-fe
    mode tcp
    option tcplog
    default_backend P22-be

backend P22-be
    mode tcp
    balance roundrobin
    server check
    server check

Appreciate if someone can point me to the right direction.