Quick question : the facts
We are running haproxy 1.8 for the moment and are thinking of upgrading to Haproxy 2.0.7 but something is bugging me regarding the default maxconn.
haproxy -vv for 1.8 does display defaults : maxconn = 2000
however haproxy -vv for 2.0.7 there is no default for maxconn.
We are migrating from 1.8 with nbproc 3 with a global maxconn at 1024 and no other explicit maxconn setting in haproxy.cfg to 2.0.7 with no nbproc setting and a maxconn at 4000 to compensate for the loss of 2 processes.
My question is the following : assuming I don’t want to limit any of my frontend/backend maxconn, is setting maxconn globally at 4000 enough for frontend and backends to inherit this setting or is the default maxconn = 2000 still present even though it is no longer reported by haproxy -vv ?
If this is still the case, that means I need to explicitly set maxconn to 4000 for every backend/frontend and it would make my configuration file quite redundant.
As of haproxy 2.0 there no longer is a default maxconn for frontends. Unless specified, frontend maxconn is inherited from global maxconn. global maxconn is automatically computed if you don’t set one in your global section.
May I ask how the global maxconn is actually computed if unspecified ?
How reliable is this auto setting for large machines (4 core 8 GB RAM) ?
We added LimitNOFILE=65536 to the systemd service (RHEL7) unit because we were hard capped at 4096 file descriptors which would have prevented going over 2000 maxconn.
Are there any other system tweaks I should perform to ensure HaProxy can fully use the server it’s running on?
Thanks again for your help, it is very much appreciated.
If this value is not set, it will automatically be
calculated based on the current file descriptors limit reported by the
"ulimit -n" command, possibly reduced to a lower value if a memory limit
is enforced, based on the buffer size, memory allocated to compression, SSL
cache size, and use or not of SSL and the associated maxsslconn (which can
also be automatic).
you can check the value set with the show info command on the stats socket.
as for system tweaks it depends on the type of workload you expect.
Our traffic is 99% SSL with security restrictions such as ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets
a restricted cipher list and tune.ssl.default-dh-param 2048
Most rules are TCP based and we only have one http-request set-path %[path,regsub(/service/,/)] to replace an old reqrep.
From my (admittedly limited) sysadmin experience, this is not the case, at least for RHEL 7 (or the particular flavor used at my company… I cannot guarantee we use a stock image of RHEL7)
Maybe I missed something in the configuration guide somewhere, and believe me, I would LOVE to not touch the service unit, but I don’t see how I can avoid this.
Also, I used to have a weird behavior where Haproxy used the Hard limit instead of soft (still the case on my production server with Ha 1.8) but that’s totally off-topic.
In my first example, as I said, LimitNOFILE was not set at all.
I’m starting to realize that maxconn has no effect at all on our RPM version of HaProxy. I did not package it and I’m starting to suspect that something might be wrong with the build (it used to be built for v 1.5 and not everything might have been updated correctly in the makefiles).
Testing this again with proof :
[root@myserver ~]# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network.target
[Service]
EnvironmentFile=-/etc/default/haproxy
EnvironmentFile=-/etc/sysconfig/haproxy
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid" "EXTRAOPTS=-S /run/haproxy-master.sock"
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS
ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify
# The following lines leverage SystemD's sandboxing options to provide
# defense in depth protection at the expense of restricting some flexibility
# in your setup (e.g. placement of your configuration files) or possibly
# reduced performance. See systemd.service(5) and systemd.exec(5) for further
# information.
# NoNewPrivileges=true
# ProtectHome=true
# If you want to use 'ProtectSystem=strict' you should whitelist the PIDFILE,
# any state files and any other files written using 'ReadWritePaths' or
# 'RuntimeDirectory'.
# ProtectSystem=true
# ProtectKernelTunables=true
# ProtectKernelModules=true
# ProtectControlGroups=true
# If your SystemD version supports them, you can add: @reboot, @swap, @sync
# SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io
[Install]
WantedBy=multi-user.target
[root@myserver ~]# systemctl daemon-reload
[root@myserver ~]# systemctl stop haproxy
[root@myserver ~]# systemctl start haproxy
[root@myserver ~]# cat /etc/haproxy/haproxy.cfg | grep maxconn
maxconn 120000
[root@myserver ~]# echo "show info" | socat - /var/run/haproxy.sock | grep -E -i "maxconn|ulimit|maxsock"
Ulimit-n: 1024
Maxsock: 1024
Maxconn: 476
Hard_maxconn: 476
MaxConnRate: 2
My build of HaProxy seems to completely ignore the maxconn parameter, that really does not seem right and I suspect it might be caused by the way it’s packaged…
Edit: Went back to 1.8 in production and checked : maxconn is set at 1024 but show info reports maxconn = 2000. tried another machine that still has 1.5 and maxconn is also ignored…
Interesting. Do you see any warning in the unit logs (journalctl -xe haproxy)? I suggest to make a secondary configuration file, not listening to any production ports, just to set a specfic maxconn value and run it manually from the shell with haproxy -f /path/to/secondary-test-config.cfg -d, to see if any warnings or errors show up.
I think it’s very unlikely that this is build related, especially since you just confirmed the problem on different builds with 1.8 and 1.5 releases.
There are lot of things to check first.
Isolating the issue with a small configuration without production impact is the first step, then I’d suggest to start reproduce the issue through strace -tt so we see what actually happens.
But we are gonna need the full picture here. Output of:
uname -a
haproxy -vv
full configuration (the configuration needed to actually reproduce the issue)
Hi Lukas, first of all, I want to thank you for taking the time to troubleshoot this issue with us.
All tests below are done on dev server using HaProxy 2.0.7
Here’s a small config file I used to perform the test you asked for in debug mode
[root@myserver haproxy]# cat haproxy-test-minimal-config.cfg
# Simple configuration for an HTTP proxy listening on port 80 on all
# interfaces and forwarding requests to a single backend "servers" with a
# single server "server1" listening on 127.0.0.1:8000
global
master-worker
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:8080
default_backend servers
backend servers
server server1 127.0.0.1:8000 maxconn 32
[root@myserver haproxy]# haproxy -f haproxy-test-minimal-config.cfg -d
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Using epoll() as the polling mechanism.
[NOTICE] 297/102545 (4923) : New worker #1 (4924) forked
I changed the default frontend port from 80 to 8080 to avoid any impact but otherwise it is the default conf provided in the documentation
here’s the output of uname -a and haproxy -vv
[root@myserver haproxy]# uname -a
Linux myserver.mycompany.tld 3.10.0-693.2.2.el7.x86_64 #1 SMP Sat Sep 9 03:55:24 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@myserver haproxy]# haproxy -vv
HA-Proxy version 2.0.7 2019/09/27 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -m64 -march=x86-64 -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_PCRE=1 USE_THREAD=1 USE_REGPARM=1 USE_LINUX_TPROXY=1 USE_CRYPT_H=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_ZLIB=1 USE_SYSTEMD=1
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
<default> : mode=HTX side=FE|BE mux=H1
<default> : mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
I will post the full configuration in a bit, I need to anonymize qui a few things inside first…
Here is the general configuration part of our config file
# reference: http://www.haproxy.org/download/1.5/doc/configuration.txt
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
log 127.0.0.1 local3 err
# logging notes
# 1) configuration above sends haproxy events to local syslog as local2
#
# 2) configuration for rsyslog is in /etc/rsyslog.d/haproxy.conf
#
# 3) optionally disable local2 events going to /var/log/messages
# by editing /etc/rsyslog.conf
#
# # Don't log local2/haproxy events in messages
# *.info;mail.none;authpriv.none;cron.none;local2.none /var/log/messages
#
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
user haproxy
group haproxy
master-worker
# turn on stats unix socket
stats socket /var/run/haproxy.sock level admin
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets
# cipher configuration comes from https://wiki.mozilla.org/Security/Server_Side_TLS
# intermediate compatibility
# updated <redacted date>
ssl-default-bind-ciphers <redacted list of ciphers>
ssl-default-server-ciphers <redacted list of ciphers>
tune.ssl.default-dh-param 2048
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode tcp
option tcplog
log global
option dontlognull
option dontlog-normal
option redispatch
retries 3
timeout queue 1m
timeout connect 5s
timeout client 5m
timeout server 5m
timeout http-keep-alive 10s
timeout check 10s
maxconn 120000
Maxconn has obviously been changed for testing purposes, it used to be 1024, but as mentioned in this thread, all values have 0 effect on the info reporter by the linux socket.
If you need frontend and backend info I will need to check If I am allowed to discose this info.
Here is proof there is no other global / defaults / maxconn setting in our config file :
[root@myserver ~]# grep -E -i "global|maxconn|defaults" /etc/haproxy/haproxy.cfg
# Global settings
global
# common defaults that all the 'listen' and 'backend' sections will
defaults
log global
maxconn 120000
You have maxconn in the defaults section. You need to put it into the global section instead, or move the entire defaults section before the global section, otherwise the defaults “maxconn” configuration will be applied only to frontends.
