Thanks for the clarification @lukastribus!
I was looking deeper into it and found that it is most likely a combination of multiple issues.
To give you some high-level background: Kubernetes is running our application as a so-called “pod” and HAProxy is running as a sidecar in that pod, i.e. it is running as a separate container but in the pod of our service.
Kubernetes now allows you to define a resource quota for each container in a pod, e.g. the max CPU and RAM. If that quota is being reached, Kubernetes will first “throttle” the process in the container allowing it to temporarily “spike”. If the resource usage is above the limit for too long, Kubernetes will then just kill the container.
What we are observing here are two issues happening concurrently:
- HAproxy requires a larger amount of RAM during startup and also the CPU usage will go up while parsing the config file
- We are seeing that at least two HAProxy processes “hang around” after a seamless reload and when another reload is happening, we suddenly have up to three HAProxy processes.
This situation will push the CPU and memory usage of the container running HAProxy to its upper limit causing Kubernetes to throttle the processes resulting in a significantly increased load time of the new HAProxy process.
We temporarily solved #1 above by increasing the CPU and memory quota of our HAProxy container. However, we do not have a solution for the “hanging” HAProxy processes.
NOTE: we are currently running on the latest version 1.8.x in production and are validating/testing 1.9 hoping that the “hanging” processes will be solved in that version.