HAProxy community

Tcp-check leaves lots of TIME_WAITs

Typical tcp-check in backend section:

backend backend_redis_write
  mode tcp
  option tcp-check
  tcp-check connect
  tcp-check send AUTH\ password\r\n
  tcp-check expect string +OK
  tcp-check send info\ replication\r\n
  tcp-check expect string role:master
  tcp-check send QUIT\r\n
  tcp-check expect string +OK
  server server1 server1:6379 check inter 1s on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions
  server server2 server2:6379 check inter 1s on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions
  server server3 server3:6379 check inter 1s on-marked-down shutdown-sessions on-marked-up shutdown-backup-sessions

With the config above Haproxy host has lots of sockets in TIME_WAIT state. I wonder if it’s possible to use persistent connections for this and not open new connection on next tcp-check sequence run?

Regards,

It seems that Haproxy sends FIN first, not waiting until server does this. This explains why we have TIME_WAITs.

TIME_WAIT’s are not a problem per-se, unless you run out of source-ports.

Make sure you enable net.ipv4.tcp_tw_reuse, but don’t enable net.ipv4.tcp_tw_recycle.

That’s not supported for HTTP and pretty much impossible for TCP health checks, as we would have to implement a syntax that is able to describe proprietary TCP level transactions.

But like I said a high number of TIME_WAITs are a non-problem. Running out of source-ports is a problem, but I assume that’s not happening here.

I’m aware of tw_reuse (it’s turned ON). tw_recycle is evil (and removed in modern Linux kernels).

Just to clarify the problem: if I set up external script that does almost the same check with help of netcat - I get perfect scenario when _Redis server sends FIN first, not Haproxy, so TIME_WAIT states are almost eliminated.

The problem is that Haproxy (as client) sends FIN packet first when check phase is finished. Not waiting for FIN from backend (Redis etc.).

Is there any way to ‘sleep’ or ‘delay’ sending FIN packet from Haproxy so remote side will be able to send FIN packet first?

P.S. Having lots of TIME_WAIT at server’s side - not a problem. But having them at client’s side - get ready for errors like: “Cannot bind to source address before connect() for backend xxx. Aborting.”

That’s what I said: unless you run out of source ports, it’s not a problem.

You health check every second, so I’d assume you have about 60 sockets in TIME_WAIT per server. I don’t see how you’d run out of source ports realistically.

I’m not aware of an option to delay the health checks.

Unfortunately, there’re lots of checks, much more than 60. For each backend it counts up to 7-8K of TIME_WAITs as I have several ports on backends to check. Anyway, thank you for clarification with current state of heathchecks.

What I meant was 60 sockets in TIME_WAIT state per haproxy declared server (which is per destination IP + per port).

Source port exhaustion happens based on the 5 tuple (minus the source port obviously), therefor you will never run out of source ports in this case. You are using 60 source ports out of 65000 for each destination IP/port combination.

Whether you have a total of 8k or 2 million of sockets in TIME_WAIT does not really matter for source port exhaustion.