Hey, you’re welcome.
It very much looks like you have an underlying network issue, that or a very unstable backend host (or application). Assuming the latter isn’t true as you’ve suggested you don’t have issues when connecting ‘on-host’ you need to take a look at everything along the network path between the two, starting with the network interface on the host running HAProxy. Moving onto routing and whatever network devices are sat between the two. It might also be worth checking there is nothing rate limiting or similar (iptables, in the app wherever).
It’s hard to be more precise with no view of what your infrastructure looks like.