We running an Haproxy server on version 2.0.29 on Ubuntu 20.04.
It has a 10Gb/s NIC for connection to our LAN and the backend servers and is connected to a firewall capable of running at 1Gb/s.
It is running various frontend and backends, some of which are using HTTP mode and a few are using TCP. The HTTP frontends are providing SSL and one HTTP frontend in particular gets most of the traffic. It has ACL rules based on which subdomain is being used and some based on which folder path.
When under normal load, it has around 7000 connections according to the stats page and is using around 700-900Mb/s. So close to the maximum of the internet connection but never using 100% and usually around 70%.
If we download from one of our backend web server servers directly, via a NAT rule pointing at one of the servers, we get speeds matching the available remaining bandwidth or the maximum of the client’s internet connection for example 30MB/s, but if we download from those same backends through the Haproxy we get a maximum of 2Mb/s from that same client.
If I run an iperf on the Haproxy server while under load from another server on the same LAN, it can max out the other servers 1Gb/s network card, so it appears the 10Gb/s NIC isn’t a limiting factor and is working correctly.
Here is a log entry from an attempted download while under load, I’m hoping this might help with diagnosing the issue (I’ve renamed things for privacy):
Feb 1 10:46:30 localhost haproxy: ipofclient:35331 [01/Feb/2023:10:45:36.192] 443frontend~ 443backend/webserver 0/0/2/11/54136 200 30703724 - - CD-- 7142/4987/500/88/0 0/0 “GET /packages/test.exe HTTP/1.1”
We have a secondary server with the same configuration and if we use that while it has no other load it all seems to function correctly, so we don’t think it’s specifically a config issue but we wonder if there’s something we need to optimise to help it cope with the number of SSL connections.
The server is 4 core 8 threads CPU with 8GB RAM and Haproxy is running with 8 threads using around 20% CPU on each of the 8 “cores” and around 2.5GB RAM utilised total.
On the Haproxy stats page it says it’s running at 80-90% idle.
We’re not sure what we can tweak or change to improve the download performance as nothing we’ve found has shown any over utilisation.