Is there a size limit for the haproxy.cfg file? If yes, what would that limit be?
Also, what is the time it should take to parse a really large cfg file until HAProxy is ready to accept connections?
Background: we have backends with a LOT of servers! In one example, we have a cfg file with almost 3000 lines. Two backends have a little over 1200 servers each. It appears that sometimes HAProxy takes more than 30 second to parse it and accept connections. This causes our health-check to time out resulting in a restart of pods (our services run in Kubernetes).
There is no limit.
How long it takes to be ready depends on the configuration. The amount of lines is not as important, what is important is what you are asking haproxy to do. For example, reading tens of thousands of certificates from disk is quite expensive.
I don’t know anything about Kubernetes; but you cannot expect that haproxy is immediately available after starting it.
Haproxy can be reloaded to reduce downtime and since 1.8 even without dropping a single connection (“seamless reloads”). How Kuberenetes could leverage this, I don’t know, like I said I don’t know anything about it.
Thanks for the clarification @lukastribus!
I was looking deeper into it and found that it is most likely a combination of multiple issues.
To give you some high-level background: Kubernetes is running our application as a so-called “pod” and HAProxy is running as a sidecar in that pod, i.e. it is running as a separate container but in the pod of our service.
Kubernetes now allows you to define a resource quota for each container in a pod, e.g. the max CPU and RAM. If that quota is being reached, Kubernetes will first “throttle” the process in the container allowing it to temporarily “spike”. If the resource usage is above the limit for too long, Kubernetes will then just kill the container.
What we are observing here are two issues happening concurrently:
- HAproxy requires a larger amount of RAM during startup and also the CPU usage will go up while parsing the config file
- We are seeing that at least two HAProxy processes “hang around” after a seamless reload and when another reload is happening, we suddenly have up to three HAProxy processes.
This situation will push the CPU and memory usage of the container running HAProxy to its upper limit causing Kubernetes to throttle the processes resulting in a significantly increased load time of the new HAProxy process.
We temporarily solved #1 above by increasing the CPU and memory quota of our HAProxy container. However, we do not have a solution for the “hanging” HAProxy processes.
NOTE: we are currently running on the latest version 1.8.x in production and are validating/testing 1.9 hoping that the “hanging” processes will be solved in that version.
Old processes hang around to finish in flight transactions or other pending transmissions.
You should check your timeout settings and consider setting a hard limit with hard-stop-after.
You will see about twice the memory usage, until the old process is closed. Of course, CPU usage will also be higher will parsing.