Haproxy reloads and total-to-server counter

Help!
1

we have an haproxy config two nodes and the balance alogrithm as “leastconn”. If we load 10 connections then each backend server end up getting 5 connections. Now we add another node to the backend server list and do “systemctl reload haproxy”. The existing 10 connections remain. But if we make another 5 new front end connections then they are supposed to go to 3rd backend server only. However we see that the first and second backend servers too get connections along with the newly added third backend server.
If we see haproxy logs we see that “total-to-server” counter has become 0 on the earlier existing nodes. Seems across reloads this stat is not persisted. Explored the option https://cbonte.github.io/haproxy-dconv/1.7/configuration.html#4.2-load-server-state-from-file. Even after doing
socat /tmp/socket - <<< “show servers state” > /tmp/server_state
before reloading the haproxy service when new connections arrive the “total-to-server” still seems to start all over from 0 for existing servers. I also do not see this stat stored in the “/tmp/server_state” file.

Is there something we are missing here or is it working as expected? Please let know

2

A reload means that the old process keeps serving existing sessions until they are closed (your 10 connections).

The new process only sees the 5 new connections and correctly balances those between the 3 servers.

What you can do is:

  • in HTTP, with long-running keep-alive sessions, reduce timeout http-keep-alive, something like 2 seconds should be perfectly fine for Internet facing web services
  • in TCP (or HTTP when you want to kill long running responses/requests, like slow downloads), force haproxy to kill the remaining sessions after a specific time with hard-stop-after
  • you can also just restart instead of reload, killing old sessions immediately
3

Thank you so much for the response. In our case these are long standing TCP connections from various endpoints (devices, laptops etc.,) and it will be in thousands. We do not want to disrupt these connections when a node is added by admin. Hence the reload is called after editing the configuration file. Is there anything we can do to transfer the number of existing connection info per backend server across the process?
If not is it something that can go as enhancement? Please advise

4

No, running TCP connections cannot be transferred from one process to the other.

You may want to take a look at the different possibilities to avoid the reload for new servers (DNS discovery, admin socket configuration).

Take a look at this thread: