Slow downloads through Haproxy when under load

Help!
1

We running an Haproxy server on version 2.0.29 on Ubuntu 20.04.

It has a 10Gb/s NIC for connection to our LAN and the backend servers and is connected to a firewall capable of running at 1Gb/s.

It is running various frontend and backends, some of which are using HTTP mode and a few are using TCP. The HTTP frontends are providing SSL and one HTTP frontend in particular gets most of the traffic. It has ACL rules based on which subdomain is being used and some based on which folder path.

When under normal load, it has around 7000 connections according to the stats page and is using around 700-900Mb/s. So close to the maximum of the internet connection but never using 100% and usually around 70%.

If we download from one of our backend web server servers directly, via a NAT rule pointing at one of the servers, we get speeds matching the available remaining bandwidth or the maximum of the client’s internet connection for example 30MB/s, but if we download from those same backends through the Haproxy we get a maximum of 2Mb/s from that same client.

If I run an iperf on the Haproxy server while under load from another server on the same LAN, it can max out the other servers 1Gb/s network card, so it appears the 10Gb/s NIC isn’t a limiting factor and is working correctly.

Here is a log entry from an attempted download while under load, I’m hoping this might help with diagnosing the issue (I’ve renamed things for privacy):
Feb 1 10:46:30 localhost haproxy[2357254]: ipofclient:35331 [01/Feb/2023:10:45:36.192] 443frontend~ 443backend/webserver 0/0/2/11/54136 200 30703724 - - CD-- 7142/4987/500/88/0 0/0 “GET /packages/test.exe HTTP/1.1”

We have a secondary server with the same configuration and if we use that while it has no other load it all seems to function correctly, so we don’t think it’s specifically a config issue but we wonder if there’s something we need to optimise to help it cope with the number of SSL connections.

The server is 4 core 8 threads CPU with 8GB RAM and Haproxy is running with 8 threads using around 20% CPU on each of the 8 “cores” and around 2.5GB RAM utilised total.
On the Haproxy stats page it says it’s running at 80-90% idle.

We’re not sure what we can tweak or change to improve the download performance as nothing we’ve found has shown any over utilisation.

2

These are only my best guesses. At a glance, I wonder if it’s one of two things:

  1. The log entry says it responded with a 200, but the CD-- part means that “The client unexpectedly aborted during data transfer.” (See Termination States) I wonder if that might be your version of HAProxy showing it’s age a bit. 2.0 is only getting critical bug fixes. An easy try at a fix might upgrading to 2.2 or 2.4.
  2. Using that same CD-- makes me think that maybe there’s a hardware failure or something in between HAProxy and its clients, perhaps a device that doesn’t handle the load as well as the server does? That might explain why another server with similar configs doesn’t exibit similar symptoms, especially since it’s also not under load.

Admittedly, I barely use the 2GB of RAM, so the only tuning parameter I actively use is nbthread which tells HAProxy how many threads to run on. My understanding is it defaults to 1 on some systems and the number of available threads on others. You can check by running haproxy -vv. Mine displays about 15 or 20 lines down:
Built with multi-threading support (MAX_THREADS=64, default=2).