I am looking for suggestions on how to understand an issue we’re observing in our environment where use have a couple of GRPC service replicas (active/backup) fronted by HAProxy (version 1.6.11). The haproxy configuration looks something like this:
listen stats maxconn 2000 balance leastconn bind 127.0.0.1:8888 mode http stats enable stats uri / stats hide-version stats refresh 5s frontend myservers maxconn 2000 bind 0.0.0.0:12000 mode tcp default_backend myservers backend myservers mode tcp balance roundrobin server server1 server1:12000 check port 12002 inter 1s server server2 server2:12000 check port 12002 inter 1s backup
The gRPC service is littered with traces like:
"transport: http2Server.HandleStreams failed to read frame: read tcp 10.64.37.24:12000-\u003e10.64.37.213:59012: read: connection reset by peer
I was able to pinpoint the presence of the traces to the fact that haproxy connections are not being closed with a FIN flag. Connections closed with a FIN flag don’t seem to trigger the trace. It seems also that some connections are closed with the RST packet, others with the FIN flag.
While I can appreciate that closing with RST in legitimate, I wonder what is the source of the flapping and whether there’ is a way to stop it. Has anyone else experienced something of this nature?
Thanks in advance!!