HAProxy community

Doubling tcp resets

Hi,

While researching an issue, I’ve found out doubling tcp resets:

# tcpdump -nn "host 172.30.2.194 and port 3443" | grep "[R]"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
16:53:27.454475 IP 172.30.2.194.3443 > 172.30.1.30.10434: Flags [R], seq 4150895860, win 0, length 0
16:53:27.454628 IP 172.30.2.194.3443 > 172.30.1.30.10434: Flags [R], seq 4150895860, win 0, length 0
16:53:28.540711 IP 172.30.2.194.3443 > 172.30.1.30.10578: Flags [R], seq 4211144735, win 0, length 0
16:53:28.540730 IP 172.30.2.194.3443 > 172.30.1.30.10578: Flags [R], seq 4211144735, win 0, length 0
16:53:30.741290 IP 172.30.2.194.3443 > 172.30.2.55.34986: Flags [R], seq 4066360692, win 0, length 0
16:53:30.741376 IP 172.30.2.194.3443 > 172.30.2.55.34986: Flags [R], seq 4066360692, win 0, length 0
16:53:36.051007 IP 172.30.2.194.3443 > 172.30.1.30.8370: Flags [R], seq 3608015152, win 0, length 0
16:53:36.051020 IP 172.30.2.194.3443 > 172.30.1.30.8370: Flags [R], seq 3608015152, win 0, length 0
16:53:45.898741 IP 172.30.2.194.3443 > 172.30.2.55.35228: Flags [R], seq 3409875850, win 0, length 0
16:53:45.898780 IP 172.30.2.194.3443 > 172.30.2.55.35228: Flags [R], seq 3409875850, win 0, length 0
16:53:46.603130 IP 172.30.2.194.3443 > 172.30.2.55.34398: Flags [R], seq 1417708259, win 0, length 0
16:53:46.603242 IP 172.30.2.194.3443 > 172.30.2.55.34398: Flags [R], seq 1417708259, win 0, length 0

Can anyone explain them?

PS HAProxy v.1.9.5 running under Amazon Linux on AWS EC2 instance
PPS There’re two EC2 instances and both of them send tcp reset twice.
PPS HAProxy serves on 172.30.2.194:3443

Alright… Here is another test:

frontend lb-useast
  mode http
  bind *:4080 name lb-useast_frontend_http_new
  bind *:4443 name lb-useast_frontend_new ssl crt ....
  monitor-uri /haproxy

Curling monitoring URI using https:
curl -v http://proxy:4443/haproxy

16:44:36.777010 IP 172.30.2.194.55212 > 172.30.1.195.4443: Flags [S], seq 3893092203, win 26883, options [mss 8961,sackOK,TS val 3016125645 ecr 0,nop,wscale 9], length 0
16:44:36.777030 IP 172.30.1.195.4443 > 172.30.2.194.55212: Flags [S.], seq 785191015, ack 3893092204, win 26847, options [mss 8961,sackOK,TS val 3988408654 ecr 3016125645,nop,wscale 9], length 0
16:44:36.777536 IP 172.30.2.194.55212 > 172.30.1.195.4443: Flags [.], ack 1, win 53, options [nop,nop,TS val 3016125645 ecr 3988408654], length 0
16:44:36.786258 IP 172.30.2.194.55212 > 172.30.1.195.4443: Flags [P.], seq 1:518, ack 1, win 53, options [nop,nop,TS val 3016125654 ecr 3988408654], length 517
16:44:36.787649 IP 172.30.1.195.4443 > 172.30.2.194.55212: Flags [P.], seq 1:4114, ack 518, win 55, options [nop,nop,TS val 3988408665 ecr 3016125654], length 4113
16:44:36.788232 IP 172.30.2.194.55212 > 172.30.1.195.4443: Flags [.], ack 4114, win 69, options [nop,nop,TS val 3016125656 ecr 3988408665], length 0
16:44:36.789932 IP 172.30.2.194.55212 > 172.30.1.195.4443: Flags [P.], seq 518:644, ack 4114, win 69, options [nop,nop,TS val 3016125658 ecr 3988408665], length 126
16:44:36.790206 IP 172.30.1.195.4443 > 172.30.2.194.55212: Flags [P.], seq 4114:4165, ack 644, win 55, options [nop,nop,TS val 3988408667 ecr 3016125658], length 51
16:44:36.791471 IP 172.30.2.194.55212 > 172.30.1.195.4443: Flags [P.], seq 644:785, ack 4165, win 69, options [nop,nop,TS val 3016125659 ecr 3988408667], length 141
16:44:36.791872 IP 172.30.1.195.4443 > 172.30.2.194.55212: Flags [P.], seq 4165:4340, ack 785, win 57, options [nop,nop,TS val 3988408669 ecr 3016125659], length 175
16:44:36.791960 IP 172.30.1.195.4443 > 172.30.2.194.55212: Flags [P.], seq 4340:4371, ack 785, win 57, options [nop,nop,TS val 3988408669 ecr 3016125659], length 31
16:44:36.792059 IP 172.30.1.195.4443 > 172.30.2.194.55212: Flags [F.], seq 4371, ack 785, win 57, options [nop,nop,TS val 3988408669 ecr 3016125659], length 0
16:44:36.792912 IP 172.30.2.194.55212 > 172.30.1.195.4443: Flags [.], ack 4372, win 85, options [nop,nop,TS val 3016125661 ecr 3988408669], length 0
16:44:36.793124 IP 172.30.2.194.55212 > 172.30.1.195.4443: Flags [P.], seq 785:816, ack 4372, win 85, options [nop,nop,TS val 3016125661 ecr 3988408669], length 31
16:44:36.793131 IP 172.30.1.195.4443 > 172.30.2.194.55212: Flags [R], seq 785195387, win 0, length 0

Notice the last TCP RST.

Now, curling same but via http:
curl -v http://proxy:4080/haproxy

16:46:43.407168 IP 172.30.2.194.44032 > 172.30.1.195.4080: Flags [S], seq 2523157193, win 26883, options [mss 8961,sackOK,TS val 3016252276 ecr 0,nop,wscale 9], length 0
16:46:43.407187 IP 172.30.1.195.4080 > 172.30.2.194.44032: Flags [S.], seq 1529397891, ack 2523157194, win 26847, options [mss 8961,sackOK,TS val 3988535286 ecr 3016252276,nop,wscale 9], length 0
16:46:43.407669 IP 172.30.2.194.44032 > 172.30.1.195.4080: Flags [.], ack 1, win 53, options [nop,nop,TS val 3016252276 ecr 3988535286], length 0
16:46:43.408830 IP 172.30.2.194.44032 > 172.30.1.195.4080: Flags [P.], seq 1:113, ack 1, win 53, options [nop,nop,TS val 3016252278 ecr 3988535286], length 112
16:46:43.408941 IP 172.30.1.195.4080 > 172.30.2.194.44032: Flags [F.], seq 1:147, ack 113, win 53, options [nop,nop,TS val 3988535288 ecr 3016252278], length 146
16:46:43.410026 IP 172.30.2.194.44032 > 172.30.1.195.4080: Flags [F.], seq 113, ack 148, win 55, options [nop,nop,TS val 3016252279 ecr 3988535288], length 0
16:46:43.410033 IP 172.30.1.195.4080 > 172.30.2.194.44032: Flags [.], ack 114, win 53, options [nop,nop,TS val 3988535289 ecr 3016252279], length 0

No RST at the end. Why tcp session termination differ? Is that OK?

Teardown of HTTPS is completely different due to TLS shutdown (sending close_notify), etc.

Is there a real, actual problem here?

Well… There is an AWS NLB which distributes traffic between two targets. There is a metric of NLB which is called “Target reset count” which counts tcp resets received from a target. I was going to use this metric to check health of the NLB: when something goes wrong targets might refuse requests and I’ll get an alert.

So, NLB’s TCP listener port 80 forwards traffic to one target (port 4080) and then to haproxy:4080, and port 443 forwards to another target (port 4443) and then to haproxy:4443.
The healthchecker of the 1st target queries HTTP port 4080 and retrieves /haproxy URI - this one has no issues.

The other healthchecker was set up to query HTTPS port 4443 and here the problem comes up: target reset count becomes non-zero. I’m sure it’s the healthchecker issue, b/c no customers’ traffic is routed to the NLB yet and tcpdump shows multiple tcp resets to NLB queries.

It’s worth to mention, that if I use HTTP healthchecker instead of HTTPS, the issue goes away and tcp reset count becomes zero.

Haproxy will always do what is most efficient, in many cases that means closing the TCP connection with a reset, as opposed to FIN close-down.

Monitoring for TCP resets and considering that a problem is wrong, in my opinion.

1 Like