armax
March 16, 2018, 4:26pm
1
Hi folks,
I am looking for suggestions on how to understand an issue we’re observing in our environment where use have a couple of GRPC service replicas (active/backup) fronted by HAProxy (version 1.6.11). The haproxy configuration looks something like this:
listen stats
maxconn 2000
balance leastconn
bind 127.0.0.1:8888
mode http
stats enable
stats uri /
stats hide-version
stats refresh 5s
frontend myservers
maxconn 2000
bind 0.0.0.0:12000
mode tcp
default_backend myservers
backend myservers
mode tcp
balance roundrobin
server server1 server1:12000 check port 12002 inter 1s
server server2 server2:12000 check port 12002 inter 1s backup
The gRPC service is littered with traces like:
"transport: http2Server.HandleStreams failed to read frame: read tcp 10.64.37.24:12000-\u003e10.64.37.213:59012: read: connection reset by peer
I was able to pinpoint the presence of the traces to the fact that haproxy connections are not being closed with a FIN flag. Connections closed with a FIN flag don’t seem to trigger the trace. It seems also that some connections are closed with the RST packet, others with the FIN flag.
While I can appreciate that closing with RST in legitimate, I wonder what is the source of the flapping and whether there’ is a way to stop it. Has anyone else experienced something of this nature?
Thanks in advance!!
armax
March 26, 2018, 10:03pm
2
Continuing the discussion from gRPC + HAProxy and connection reset by peer :
Have I perhaps posted to the wrong forum? I’d love to get some feedback on the matter. Thanks!
Provide haproxy logs please.
armax
March 28, 2018, 12:09am
4
I tried to tweak logs to get more verbose with debug/tcplog mode on. A log snippet below. Please let me know if there’s something else I could tweak to get more info. I appreciate the help! Thanks in advance.
2018-03-27T23:57:07+00:00 localhost haproxy[16437]: 10.38.10.2:43496 [27/Mar/2018:23:57:07.105] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0