Hi there!
we are still seeing some bad behavior of HAProxy in our environment.
To provide some context:
- we are currently running version 1.8.17 and 1.9.9 (in our canary to test it out)
- we are running HAProxy in master-worker mode (option -Ws)
- HAProxy is running as a sidecar in our Kubernetes pods
- Our environment is high-throughput (more than 1000RPS per HAProxy instance) and highly dynamic, i.e. many server changes requiring on average about 5 reloads per hour!
- some HAProxy instances have up to 50 backends with some backends having up to 1000 servers
- We use HAProxy for SSL termination, meaning that most backend server are using SSL
To better understand some of the behavior and to potentially tune HAProxy, I would like to understand the exact reload sequence of HAProxy:
- when a reload command gets issued to the master, does the old process stop accepting new connections exactly once the new process could start listening to the socket(s)?
- when does the new process do the SSL handshake with backend servers (we have “check” enabled on our backend server): during initialization or when the first request is being made to a backend server?
- does the new process accept incoming requests even if not all backend servers have been initialized yet?
I was going through the “Stopping and Starting” section in the management guide multiple times, but not all questions are answered there.
Thank you.
No, the new process tries to bind the sockets BEFORE the old process is stopped, this will work when your kernel support SO_REUSEPORT (Linux 3.9 and newer, BSD’s).
On Linux there is still a race condition even with SO_REUSEPORT, which can be fixed by the seamless reload feature.
Technical background about all of this can be found in the following blog post:
When a request on the frontend requires a connection to that backend server.
A worker immediately accepts requests. Whether or not your backend is immediately ready depends on your configuration.
Thanks @lukastribus. Your response is very much appreciated.
We are preparing to test the seamless reload in our environment as I am writing this. Thank you for pointing this out and reminding me again (I was reading about it before but then somehow forgot).
With regards to SSL handshake I ran a simple test and found that HAProxy is actually doing the handshake already immediately after initialization and before a frontend requested a connection to any backend. I suspect this is because we have a “check” on each backend server and in order to perform this, HAProxy will have to do the SSL handshake to establish a TLS socket. This would also explain the huge spike in CPU utilization for almost 30 seconds on HAProxy instances with thousands of backend server.
I noticed another interesting behavior: during reload, it appears that the stats port is not responding to requests for an extended period of time. We have a metrics collector that will do a HTTP GET on the stats endpoint which will parse the CSV and send the metrics back to Wavefront. During reloads, the HTTP GET quite often times out causing the metrics collector to abort and getting restarted. With our current setting - the timeout on the GET is set to 5 seconds and we have 6 retries - we imply that the stats endpoint can be unresponsive for more than 30 seconds.
So, my follow-up question would be: at what time after a reload is the stats endpoint of the new process ready to accept requests or ready to respond to a request?
I don’t know your configuration, but yes, I’d assume that you health check via TLS connections, and if you have thousands or tens of thousands of backend servers, startup would be extremely expensive due all TLS handshakes.
I’m not sure the stats endpoint is not available, I guess it is just so slow that it times out while the CPU is starved by TLS handshakes.
What can help here is to dump the server state to a file and use it after a reload, instead of health checking everything at startup.