Intermittent "SD" termination state

Just a theory but not really validated.
We had the option nolinger enabled on most HAProxys which will send a RST after HAProxy responded with the last byte. We are running with a redundant network and all our machines are multi-homed (multiple NICs). Our theory is that the RST got sent over a different network path and intermittently was received before the last byte or even before any byte in case the payload was smaller than our MTU. In this case, the remaining data in transit may have gotten “lost” as described here https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_termination

In any case, once we removed the nolinger setting we no longer saw those SDs. We didn’t need the nolinger anyways since it was some remaining “tech debt” from an earlier “experiment” that a previous employee who since then left forgot to remove.

By the way, we are moving away from HAProxy and are in the process of migrating to Envoy. We had so many ongoing issues with HAProxy in our highly dynamic Kubernetes clusters that we got tired of chasing them and “plugging one hole after another”. We have already migrated a fair number of services to use Envoy and it is “smooth sailing” since then.