Frontend TCP connection torn down when one server backed is restarted (ungracefully)

We use HAProxy with Keepalived to loadbalance traffic and kubernetes master nodes.
Upon a restart of one backend server (one master) TCP connection is closed to fronted on all remaining nodes (other masters & worker nodes).
While I could expect it for a subset of nodes i.e. part of frontend TCP connections are closed because connectivity to one backend server is down (well technically it probably times out because reboot is ungraceful), I wouldn’t expect it for all nodes. It just looks that ALL frontend TCP connections are closed while just SINGLE backend server goes down. Is that correct behaviour or is there some misconfiguration ?

Config is as below:

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000

frontend frontend-api
    bind <backend_ip>:7443
    bind <frontend_ip>:6443
    bind 127.0.0.1:7443
    mode tcp
    option tcplog
    timeout client 300000
    default_backend backend-api

backend backend-api
    mode tcp
    option tcplog
    option tcp-check
    timeout server 300000
    balance roundrobin
    default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100

    server master1 <backend_api_ip_srv1>:6443 check
    server master2 <backend_api_ip_srv2>:6443 check
    server master3 <backend_api_ip_srv3>:6443 check

This is not expected, and I’d be surprised if that is what actually happens here.

Share the logs in tcplog format for review.

Need to check if these are available. But do you mean that for the 1/3rd of connections it is expected but not for remaining 2/3rd ?

I don’t have tcplog available… But I have confirmed that Frontend TCP connection is closed (FIN, ACK) when backend connection is closed. So only 1/3rd of connections shall be affected but not all.

Expected behavior in TCP mode is that a frontend connection is closed when the server closes the corresponding connection, because there is no other way to handle this scenario.

That doesn’t mean connections going to different server are affected. Likely what you are seeing is that your slaves somehow also close the connection, when the master dies.

That’s why you’d have to look at the behavior haproxy/slaves, not the behavior of the haproxy/master.

Tcpdumps, tcplogs, etc can help here.