gRPC + HAProxy and connection reset by peer


#1

Hi folks,

I am looking for suggestions on how to understand an issue we’re observing in our environment where use have a couple of GRPC service replicas (active/backup) fronted by HAProxy (version 1.6.11). The haproxy configuration looks something like this:

listen stats
  maxconn 2000
  balance leastconn
  bind 127.0.0.1:8888
  mode http
  stats enable
  stats uri /
  stats hide-version
  stats refresh 5s

frontend myservers
  maxconn 2000
  bind 0.0.0.0:12000
  mode tcp
  default_backend myservers

backend myservers
  mode tcp
  balance roundrobin
  server server1 server1:12000 check port 12002 inter 1s
  server server2 server2:12000 check port 12002 inter 1s backup

The gRPC service is littered with traces like:

"transport: http2Server.HandleStreams failed to read frame: read tcp 10.64.37.24:12000-\u003e10.64.37.213:59012: read: connection reset by peer

I was able to pinpoint the presence of the traces to the fact that haproxy connections are not being closed with a FIN flag. Connections closed with a FIN flag don’t seem to trigger the trace. It seems also that some connections are closed with the RST packet, others with the FIN flag.

While I can appreciate that closing with RST in legitimate, I wonder what is the source of the flapping and whether there’ is a way to stop it. Has anyone else experienced something of this nature?

Thanks in advance!!
Armando


#2

Continuing the discussion from gRPC + HAProxy and connection reset by peer:

Have I perhaps posted to the wrong forum? I’d love to get some feedback on the matter. Thanks!


#3

Provide haproxy logs please.


#4

I tried to tweak logs to get more verbose with debug/tcplog mode on. A log snippet below. Please let me know if there’s something else I could tweak to get more info. I appreciate the help! Thanks in advance.

2018-03-27T23:57:07+00:00 localhost haproxy[16437]: 10.38.10.2:43496 [27/Mar/2018:23:57:07.105] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:08+00:00 localhost haproxy[16437]: 10.38.10.2:43500 [27/Mar/2018:23:57:08.144] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:09+00:00 localhost haproxy[16437]: 10.38.10.2:43504 [27/Mar/2018:23:57:09.182] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:10+00:00 localhost haproxy[16437]: 10.38.10.2:43508 [27/Mar/2018:23:57:10.221] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:11+00:00 localhost haproxy[16437]: 10.38.10.2:43512 [27/Mar/2018:23:57:11.259] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:12+00:00 localhost haproxy[16437]: 10.38.10.2:43516 [27/Mar/2018:23:57:12.297] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:13+00:00 localhost haproxy[16437]: 10.38.10.2:43520 [27/Mar/2018:23:57:13.336] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:14+00:00 localhost haproxy[16437]: 10.38.10.2:43524 [27/Mar/2018:23:57:14.374] myservers myservers/stage-myservers01 1/0/24 2780 CD 0/0/0/0/0 0/0
2018-03-27T23:57:15+00:00 localhost haproxy[16437]: 10.38.10.2:43529 [27/Mar/2018:23:57:15.412] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:16+00:00 localhost haproxy[16437]: 10.38.10.2:43533 [27/Mar/2018:23:57:16.450] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:17+00:00 localhost haproxy[16437]: 10.37.0.161:35520 [27/Mar/2018:23:57:17.318] myservers myservers/stage-myservers01 1/0/0 0 – 0/0/0/0/0 0/0
2018-03-27T23:57:17+00:00 localhost haproxy[16437]: 10.38.10.2:43538 [27/Mar/2018:23:57:17.489] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:18+00:00 localhost haproxy[16437]: 10.38.10.2:43542 [27/Mar/2018:23:57:18.527] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:19+00:00 localhost haproxy[16437]: 10.38.10.2:43546 [27/Mar/2018:23:57:19.566] myservers myservers/stage-myservers01 1/0/24 2795 CD 0/0/0/0/0 0/0
2018-03-27T23:57:20+00:00 localhost haproxy[16437]: 10.38.10.2:43551 [27/Mar/2018:23:57:20.604] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:21+00:00 localhost haproxy[16437]: 10.38.10.2:43555 [27/Mar/2018:23:57:21.642] myservers myservers/stage-myservers01 1/0/24 2780 CD 1/1/1/1/0 0/0
2018-03-27T23:57:22+00:00 localhost haproxy[16437]: 10.38.10.2:43559 [27/Mar/2018:23:57:22.680] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:23+00:00 localhost haproxy[16437]: 10.38.10.2:43563 [27/Mar/2018:23:57:23.719] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:24+00:00 localhost haproxy[16437]: 10.38.10.2:43567 [27/Mar/2018:23:57:24.757] myservers myservers/stage-myservers01 1/0/24 2826 CD 1/1/1/1/0 0/0
2018-03-27T23:57:25+00:00 localhost haproxy[16437]: 10.38.10.2:43571 [27/Mar/2018:23:57:25.796] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:26+00:00 localhost haproxy[16437]: 10.38.10.2:43575 [27/Mar/2018:23:57:26.834] myservers myservers/stage-myservers01 1/0/24 2826 CD 1/1/1/1/0 0/0
2018-03-27T23:57:27+00:00 localhost haproxy[16437]: 10.38.10.2:43579 [27/Mar/2018:23:57:27.872] myservers myservers/stage-myservers01 1/0/24 2826 CD 1/1/1/1/0 0/0
2018-03-27T23:57:28+00:00 localhost haproxy[16437]: 10.38.10.2:43584 [27/Mar/2018:23:57:28.911] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:29+00:00 localhost haproxy[16437]: 10.38.10.2:43588 [27/Mar/2018:23:57:29.949] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:31+00:00 localhost haproxy[16437]: 10.38.10.2:43592 [27/Mar/2018:23:57:30.987] myservers myservers/stage-myservers01 1/0/26 2826 CD 1/1/1/1/0 0/0
2018-03-27T23:57:32+00:00 localhost haproxy[16437]: 10.38.10.2:43596 [27/Mar/2018:23:57:32.027] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:33+00:00 localhost haproxy[16437]: 10.38.10.2:43600 [27/Mar/2018:23:57:33.065] myservers myservers/stage-myservers01 1/0/24 2826 CD 1/1/1/1/0 0/0
2018-03-27T23:57:34+00:00 localhost haproxy[16437]: 10.38.10.2:43605 [27/Mar/2018:23:57:34.104] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:35+00:00 localhost haproxy[16437]: 10.38.10.2:43609 [27/Mar/2018:23:57:35.141] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:36+00:00 localhost haproxy[16437]: 10.38.10.2:43613 [27/Mar/2018:23:57:36.180] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:37+00:00 localhost haproxy[16437]: 10.38.10.2:43617 [27/Mar/2018:23:57:37.218] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:38+00:00 localhost haproxy[16437]: 10.38.10.2:43621 [27/Mar/2018:23:57:38.256] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:39+00:00 localhost haproxy[16437]: 10.38.10.2:43625 [27/Mar/2018:23:57:39.294] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:40+00:00 localhost haproxy[16437]: 10.38.10.2:43629 [27/Mar/2018:23:57:40.332] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:41+00:00 localhost haproxy[16437]: 10.38.10.2:43633 [27/Mar/2018:23:57:41.370] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:42+00:00 localhost haproxy[16437]: 10.38.10.2:43637 [27/Mar/2018:23:57:42.408] myservers myservers/stage-myservers01 1/0/24 2795 CD 1/1/1/1/0 0/0
2018-03-27T23:57:43+00:00 localhost haproxy[16437]: 10.38.10.2:43642 [27/Mar/2018:23:57:43.446] myservers myservers/stage-myservers01 1/0/24 2826 CD 1/1/1/1/0 0/0
2018-03-27T23:57:44+00:00 localhost haproxy[16437]: 10.38.10.2:43646 [27/Mar/2018:23:57:44.484] myservers myservers/stage-myservers01 1/0/24 2780 CD 1/1/1/1/0 0/0
2018-03-27T23:57:45+00:00 localhost haproxy[16437]: 10.38.10.2:43651 [27/Mar/2018:23:57:45.522] myservers myservers/stage-myservers01 1/0/24 2795 CD 2/2/2/2/0 0/0
2018-03-27T23:57:46+00:00 localhost haproxy[16437]: 10.38.10.2:43655 [27/Mar/2018:23:57:46.560] myservers myservers/stage-myservers01 1/0/24 2795 CD 2/2/2/2/0 0/0
2018-03-27T23:57:47+00:00 localhost haproxy[16437]: 10.38.10.2:43659 [27/Mar/2018:23:57:47.598] myservers myservers/stage-myservers01 1/0/24 2795 CD 2/2/2/2/0 0/0
2018-03-27T23:57:48+00:00 localhost haproxy[16437]: 10.38.10.2:43663 [27/Mar/2018:23:57:48.637] myservers myservers/stage-myservers01 1/0/24 2826 CD 2/2/2/2/0 0/0
2018-03-27T23:57:49+00:00 localhost haproxy[16437]: 10.38.10.2:43668 [27/Mar/2018:23:57:49.675] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:57:50+00:00 localhost haproxy[16437]: 10.38.10.2:43672 [27/Mar/2018:23:57:50.713] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:57:51+00:00 localhost haproxy[16437]: 10.38.10.2:43676 [27/Mar/2018:23:57:51.751] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:57:52+00:00 localhost haproxy[16437]: 10.38.10.2:43680 [27/Mar/2018:23:57:52.790] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:57:53+00:00 localhost haproxy[16437]: 10.38.10.2:43684 [27/Mar/2018:23:57:53.828] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:57:54+00:00 localhost haproxy[16437]: 10.38.10.2:43688 [27/Mar/2018:23:57:54.866] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:57:55+00:00 localhost haproxy[16437]: 10.38.10.2:43692 [27/Mar/2018:23:57:55.904] myservers myservers/stage-myservers01 1/0/24 2780 – 3/3/3/3/0 0/0
2018-03-27T23:57:56+00:00 localhost haproxy[16437]: 10.36.130.12:43604 [27/Mar/2018:23:57:55.976] myservers myservers/stage-myservers01 1/0/73 2826 – 4/4/4/4/0 0/0
2018-03-27T23:57:56+00:00 localhost haproxy[16437]: 10.36.130.11:46710 [27/Mar/2018:23:57:55.980] myservers myservers/stage-myservers01 1/0/119 2826 – 3/3/3/3/0 0/0
2018-03-27T23:57:56+00:00 localhost haproxy[16437]: 10.38.10.2:43699 [27/Mar/2018:23:57:56.942] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:57:57+00:00 localhost haproxy[16437]: 10.36.130.11:46738 [27/Mar/2018:23:57:57.197] myservers myservers/stage-myservers01 1/0/89 2826 – 3/3/3/3/0 0/0
2018-03-27T23:57:57+00:00 localhost haproxy[16437]: 10.36.130.12:43634 [27/Mar/2018:23:57:57.320] myservers myservers/stage-myservers01 1/0/78 2826 – 3/3/3/3/0 0/0
2018-03-27T23:57:57+00:00 localhost haproxy[16437]: 10.36.130.11:46744 [27/Mar/2018:23:57:57.643] myservers myservers/stage-myservers01 1/0/82 2826 – 3/3/3/3/0 0/0
2018-03-27T23:57:58+00:00 localhost haproxy[16437]: 10.38.10.2:43706 [27/Mar/2018:23:57:57.981] myservers myservers/stage-myservers01 1/0/25 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:57:59+00:00 localhost haproxy[16437]: 10.38.10.2:43710 [27/Mar/2018:23:57:59.019] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:00+00:00 localhost haproxy[16437]: 10.38.10.2:43714 [27/Mar/2018:23:58:00.057] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:01+00:00 localhost haproxy[16437]: 10.38.10.2:43718 [27/Mar/2018:23:58:01.095] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:02+00:00 localhost haproxy[16437]: 10.38.10.2:43722 [27/Mar/2018:23:58:02.133] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:03+00:00 localhost haproxy[16437]: 10.38.10.2:43726 [27/Mar/2018:23:58:03.171] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:04+00:00 localhost haproxy[16437]: 10.38.10.2:43731 [27/Mar/2018:23:58:04.210] myservers myservers/stage-myservers01 1/0/25 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:05+00:00 localhost haproxy[16437]: 10.38.10.2:43735 [27/Mar/2018:23:58:05.249] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:06+00:00 localhost haproxy[16437]: 10.38.10.2:43739 [27/Mar/2018:23:58:06.288] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:07+00:00 localhost haproxy[16437]: 10.38.10.2:43743 [27/Mar/2018:23:58:07.326] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:08+00:00 localhost haproxy[16437]: 10.38.10.2:43747 [27/Mar/2018:23:58:08.364] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:09+00:00 localhost haproxy[16437]: 10.38.10.2:43751 [27/Mar/2018:23:58:09.402] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:10+00:00 localhost haproxy[16437]: 10.38.10.2:43755 [27/Mar/2018:23:58:10.440] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:11+00:00 localhost haproxy[16437]: 10.38.10.2:43760 [27/Mar/2018:23:58:11.479] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:12+00:00 localhost haproxy[16437]: 10.38.10.2:43764 [27/Mar/2018:23:58:12.517] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:13+00:00 localhost haproxy[16437]: 10.38.10.2:43768 [27/Mar/2018:23:58:13.555] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:14+00:00 localhost haproxy[16437]: 10.38.10.2:43772 [27/Mar/2018:23:58:14.593] myservers myservers/stage-myservers01 1/0/24 2780 CD 3/3/3/3/0 0/0
2018-03-27T23:58:15+00:00 localhost haproxy[16437]: 10.38.10.2:43776 [27/Mar/2018:23:58:15.632] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:16+00:00 localhost haproxy[16437]: 10.38.10.2:43780 [27/Mar/2018:23:58:16.670] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:17+00:00 localhost haproxy[16437]: 10.38.10.2:43784 [27/Mar/2018:23:58:17.708] myservers myservers/stage-myservers01 1/0/24 2780 CD 3/3/3/3/0 0/0
2018-03-27T23:58:18+00:00 localhost haproxy[16437]: 10.38.10.2:43788 [27/Mar/2018:23:58:18.746] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:19+00:00 localhost haproxy[16437]: 10.37.0.77:35104 [27/Mar/2018:23:57:19.615] myservers myservers/stage-myservers01 1/0/60009 2622 cD 2/2/2/2/0 0/0
2018-03-27T23:58:19+00:00 localhost haproxy[16437]: 10.38.10.2:43793 [27/Mar/2018:23:58:19.784] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:20+00:00 localhost haproxy[16437]: 10.38.10.2:43797 [27/Mar/2018:23:58:20.822] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:21+00:00 localhost haproxy[16437]: 10.38.10.2:43801 [27/Mar/2018:23:58:21.861] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:22+00:00 localhost haproxy[16437]: 10.38.10.2:43805 [27/Mar/2018:23:58:22.899] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:23+00:00 localhost haproxy[16437]: 10.38.10.2:43809 [27/Mar/2018:23:58:23.937] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:24+00:00 localhost haproxy[16437]: 10.38.10.2:43813 [27/Mar/2018:23:58:24.975] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:26+00:00 localhost haproxy[16437]: 10.38.10.2:43818 [27/Mar/2018:23:58:26.013] myservers myservers/stage-myservers01 1/0/24 2780 CD 3/3/3/3/0 0/0
2018-03-27T23:58:27+00:00 localhost haproxy[16437]: 10.38.10.2:43822 [27/Mar/2018:23:58:27.051] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:28+00:00 localhost haproxy[16437]: 10.38.10.2:43826 [27/Mar/2018:23:58:28.091] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:29+00:00 localhost haproxy[16437]: 10.38.10.2:43830 [27/Mar/2018:23:58:29.129] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:30+00:00 localhost haproxy[16437]: 10.38.10.2:43834 [27/Mar/2018:23:58:30.167] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:31+00:00 localhost haproxy[16437]: 10.38.10.2:43838 [27/Mar/2018:23:58:31.206] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:32+00:00 localhost haproxy[16437]: 10.38.10.2:43842 [27/Mar/2018:23:58:32.244] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:32+00:00 localhost haproxy[16437]: 10.36.130.11:46816 [27/Mar/2018:23:58:32.275] myservers myservers/stage-myservers01 1/0/86 2826 – 3/3/3/3/0 0/0
2018-03-27T23:58:33+00:00 localhost haproxy[16437]: 10.38.10.2:43847 [27/Mar/2018:23:58:33.282] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:34+00:00 localhost haproxy[16437]: 10.38.10.2:43852 [27/Mar/2018:23:58:34.320] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:35+00:00 localhost haproxy[16437]: 10.38.10.2:43856 [27/Mar/2018:23:58:35.358] myservers myservers/stage-myservers01 1/0/24 2780 CD 3/3/3/3/0 0/0
2018-03-27T23:58:36+00:00 localhost haproxy[16437]: 10.38.10.2:43860 [27/Mar/2018:23:58:36.396] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:37+00:00 localhost haproxy[16437]: 10.38.10.2:43864 [27/Mar/2018:23:58:37.434] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:38+00:00 localhost haproxy[16437]: 10.38.10.2:43868 [27/Mar/2018:23:58:38.472] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:39+00:00 localhost haproxy[16437]: 10.38.10.2:43873 [27/Mar/2018:23:58:39.511] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:40+00:00 localhost haproxy[16437]: 10.38.10.2:43877 [27/Mar/2018:23:58:40.549] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:41+00:00 localhost haproxy[16437]: 10.38.10.2:43881 [27/Mar/2018:23:58:41.588] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:42+00:00 localhost haproxy[16437]: 10.38.10.2:43885 [27/Mar/2018:23:58:42.626] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:43+00:00 localhost haproxy[16437]: 10.38.10.2:43889 [27/Mar/2018:23:58:43.664] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:44+00:00 localhost haproxy[16437]: 10.37.0.77:60110 [27/Mar/2018:23:57:44.548] myservers myservers/stage-myservers01 1/0/60007 2622 cD 2/2/2/2/0 0/0
2018-03-27T23:58:44+00:00 localhost haproxy[16437]: 10.38.10.2:43894 [27/Mar/2018:23:58:44.703] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:45+00:00 localhost haproxy[16437]: 10.38.10.2:43898 [27/Mar/2018:23:58:45.741] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:46+00:00 localhost haproxy[16437]: 10.38.10.2:43902 [27/Mar/2018:23:58:46.779] myservers myservers/stage-myservers01 1/0/24 2795 CD 3/3/3/3/0 0/0
2018-03-27T23:58:47+00:00 localhost haproxy[16437]: 10.38.10.2:43906 [27/Mar/2018:23:58:47.817] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:48+00:00 localhost haproxy[16437]: 10.38.10.2:43910 [27/Mar/2018:23:58:48.855] myservers myservers/stage-myservers01 1/0/24 2826 CD 3/3/3/3/0 0/0
2018-03-27T23:58:49+00:00 localhost haproxy[16437]: 10.37.0.158:55535 [27/Mar/2018:23:57:49.097] myservers myservers/stage-myservers01 1/0/60007 2622 cD 2/2/2/2/0 0/0
2018-03-27T23:58:49+00:00 localhost haproxy[16437]: 10.38.10.2:43914 [27/Mar/2018:23:58:49.893] myservers myservers/stage-myservers01 1/0/24 2795 CD 2/2/2/2/0 0/0
2018-03-27T23:58:50+00:00 localhost haproxy[16437]: 10.38.10.2:43918 [27/Mar/2018:23:58:50.931] myservers myservers/stage-myservers01 1/0/24 2795 CD 2/2/2/2/0 0/0
2018-03-27T23:58:51+00:00 localhost haproxy[16437]: 10.38.10.2:43922 [27/Mar/2018:23:58:51.970] myservers myservers/stage-myservers01 1/0/24 2795 CD 2/2/2/2/0 0/0
2018-03-27T23:58:53+00:00 localhost haproxy[16437]: 10.38.10.2:43926 [27/Mar/2018:23:58:53.008] myservers myservers/stage-myservers01 1/0/24 2795 CD 2/2/2/2/0 0/0