Just a theory but not really validated.
We had the option nolinger
enabled on most HAProxys which will send a RST
after HAProxy responded with the last byte. We are running with a redundant network and all our machines are multi-homed (multiple NICs). Our theory is that the RST
got sent over a different network path and intermittently was received before the last byte or even before any byte in case the payload was smaller than our MTU. In this case, the remaining data in transit may have gotten “lost” as described here https://en.wikipedia.org/wiki/Transmission_Control_Protocol#Connection_termination
In any case, once we removed the nolinger
setting we no longer saw those SD
s. We didn’t need the nolinger
anyways since it was some remaining “tech debt” from an earlier “experiment” that a previous employee who since then left forgot to remove.
By the way, we are moving away from HAProxy and are in the process of migrating to Envoy. We had so many ongoing issues with HAProxy in our highly dynamic Kubernetes clusters that we got tired of chasing them and “plugging one hole after another”. We have already migrated a fair number of services to use Envoy and it is “smooth sailing” since then.