Client reconnects to old server during blue-green deployment

Help!
1

A. Overview of the config:

  1. We use HAProxy to proxy a WebSocket client connection to one of the servers.
  2. We balance based on a query parameter and stick on sticky table.
  3. We use Data Plane API for config changes. When config changes then new child process is created and old one remains until all clients disconnect. Hence we use peers configuration to persist sticky table.
  4. When sticky table space is filled up, old entries are removed. However, that never happens because we allocate 0.5GB for sticky table and we do not have so many active connections.
  5. Server exposes GET /health endpoint that is configured using http-check

B. There are 3 phases during our blue-green deployment:

  1. ATTACH: Creating and attaching new servers one-by-one.
  • We use POST /services/haproxy/configuration/servers with {"weight": 10, "maxconn": 100000, "on-marked-down": "shutdown-sessions", "check": "enabled", "address": "$address", "name": "$name"}
  • We only attach new server after making sure that it is healthy by directly checking GET /health
  1. DRAIN: Draining old servers one-by-one
  • We use PUT /services/haproxy/configuration/servers/${name} with {"weight": 0, "maxconn": 100000, "on-marked-down": "shutdown-sessions", "check": "enabled", "address": "$address", "name": "$name"} Only weight changes.
  1. DETACH: Detaching old servers one-by-one
  • We use DELETE /services/haproxy/configuration/servers/${name}

C. Problem

We expect that during DETACH, old client always reconnects to new server (with weight: 10).

However, HAProxy chooses old server (with weight: 0) that is not detached yet.
When the chosen server is detached, HAProxy sometimes chooses another old server.
At the final state, after all old servers are detached, old client reconnects to new server.

Any idea why HAProxy chooses old server (with weight: 0) over new server (with weight: 10) when client reconnects?