Hello,
I found this discussion on some older HAProxy forum. I have very similar “issue” to solve.
We have quite complex set-up so I don’t want to dive into details. To describe it simply - we are running HAProxy in multiple containers / pods in Kubernetes. There are multiple clients and one “type” of them is another HAProxy in the different Kuberentes cluster 
Between these two HAProxy deployments are also different L4 network components but I will omit them for simplicity as I don’t think they are important for this case
Additional notes:
- Between HAProxy deployments is higher latency. They are running in different geographical locations.
- Traffic is encrypted with https.
- They are using keepalive connection with timeout set to 1 min.
- We have lower amount of active connections 10-30 but high rate of reused connections 1000 - 3500 on HAProxy which initiate connections.
- If I mention HAProxy it actually means multiple / separate containers / pods.
The issue: I think that when HAProxy pod is terminating (receiving side) there are left keepalive connections (high reuse rate on client side as mentioned above) which are later killed forcefully generating 5xx. I’m trying to find a way how to terminate the long lived connections gracefully e.g. to say other said close connection once request is finished and create new one for the next request. Is there anything like that ?
Thx
I was considering to set connection close header in frontend when HAProxy receive graceful stop (SIGUSR1).
Something like:
http-response set-header connection close if { stopping }
I was testing HAProxy behaviour when it receives SIGUSR1 signal.
It seems that HAProxy adds connection: close header by default so no need to add above snipped into config. Here is output from my testing:
└──╼$ while :; do curl -s -D - http://127.0.0.1:9001/evil -o /dev/null;done
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 29 Nov 2022 13:30:53 GMT
content-length: 2
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 29 Nov 2022 13:30:56 GMT
content-length: 2
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 29 Nov 2022 13:30:59 GMT
content-length: 2
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 29 Nov 2022 13:31:02 GMT
content-length: 2
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 29 Nov 2022 13:31:05 GMT
content-length: 2
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 29 Nov 2022 13:31:08 GMT
content-length: 2
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 29 Nov 2022 13:31:11 GMT
content-length: 2
HTTP/1.1 200 OK
content-type: application/json
date: Tue, 29 Nov 2022 13:31:14 GMT
content-length: 2
connection: close
See the last response ^.
However I kinda not 100% sure about it because header addition isn’t documented at least I didn’t found it here. For that reason I’m not sure if that is expected. I tried to find it in the code but no luck so far (I’m not C programmer). Anybody has any experience with it ?
I found this magic option called close-spread-time where is hidden info that header connection: close is added when HAProxy receives SIGUSR1 (soft-stop).
Define a time window during which idle connections and active connections
closing is spread in case of soft-stop. After a SIGUSR1 is received and the
grace period is over (if any), the idle connections will all be closed at
once if this option is not set, and active HTTP or HTTP2 connections will be
ended after the next request is received, either by appending a "Connection:
close" line to the HTTP response, or by sending a GOAWAY frame in case of
HTTP2. When this option is set, connection closing will be spread over this
set <time>.
If the close-spread-time is set to "infinite", active connection closing
during a soft-stop will be disabled. The "Connection: close" header will not
be added to HTTP responses (or GOAWAY for HTTP2) anymore and idle connections
will only be closed once their timeout is reached (based on the various
timeouts set in the configuration).