we are still seeing some bad behavior of HAProxy in our environment.
To provide some context:
- we are currently running version 1.8.17 and 1.9.9 (in our canary to test it out)
- we are running HAProxy in master-worker mode (option -Ws)
- HAProxy is running as a sidecar in our Kubernetes pods
- Our environment is high-throughput (more than 1000RPS per HAProxy instance) and highly dynamic, i.e. many server changes requiring on average about 5 reloads per hour!
- some HAProxy instances have up to 50 backends with some backends having up to 1000 servers
- We use HAProxy for SSL termination, meaning that most backend server are using SSL
To better understand some of the behavior and to potentially tune HAProxy, I would like to understand the exact reload sequence of HAProxy:
- when a reload command gets issued to the master, does the old process stop accepting new connections exactly once the new process could start listening to the socket(s)?
- when does the new process do the SSL handshake with backend servers (we have “check” enabled on our backend server): during initialization or when the first request is being made to a backend server?
- does the new process accept incoming requests even if not all backend servers have been initialized yet?
I was going through the “Stopping and Starting” section in the management guide multiple times, but not all questions are answered there.