Connections stopping after 2,000 connections

Help!
1

Afternoon all, I have read and tested so many things my eyes are bleeding, but basically I had a similar issue with MQTT and mosquitto and have that resolved, but when I test through my HAProxy server, it’s stopping after 2,000 connections (yet direct I can get well over 10k).

I have ulimit -n showing 64k connections for both root and haproxy. When I check the processes I have 3.
/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds . (2 identical ones).

If I check the 2 that are the same (/proc/ID/limits) I see the following;
Max open files 8000035 8000035 files
but the 1st shows;
Max open files 1024 4096 files

I don’t know if that is the issue, but my haproxy.cfg shows the following;
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 4000000

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private

# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL). This list is from:
#  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3

defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

listen mqtt
bind *:1883
mode tcp
option tcplog
balance leastconn
server broker_1 192.168.1.231:1883 check

listen mqtt-swd1
bind *:1884
mode tcp
option tcplog
balance leastconn
server broker_1 192.168.1.231:1883 check

So, that single server has a public IP which I mentioned, if I go direct with a test app can make over 10k connections, but when I say go through HAProxy, it stops at exactly 2k connections.

This is running on an amazon AWS linux EC2 server (I can really use any distro) but this should be a non issue.

Any help or .02 is appreciated.

2

Hi,

Your Global maxconn is way too big and I guess HAProxy falls back to the default 2K value.
Try with a global maxconn at 10K for example and see if you can beat the 2K.
You must have a warning message printed by HAProxy at start up. You should read it.

Note that there is also a default hidden maxconn on the frontend set to 2K as well. You might want to add a maxconn parameter in your frontend/listen sections with a higher value.
https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-maxconn