# Understanding maxconn and maxonnrate and delays

**URL:** <https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230>\
**Category:** Help!\
**Created:** [November 15, 2018, 5:03pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230 "2018-11-15T17:03:47Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![ndakotaeq](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@ndakotaeq](https://discourse.haproxy.org/u/ndakotaeq)\
**Post date:** [November 15, 2018, 5:03pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/1 "2018-11-15T17:03:47Z")

</div>

Hello, I am troubleshooting an issue most likely related with high load per second.

While investigating I am looking on network capture and I see the delay ~50 sec between client`s SYN sent to haproxy, haproxy sent ACK back to the client but from the haproxy to the server there is delay ~50sec. For me it seems like haproxy process get request ~50sec after it is arrived to VM.

CPU is 100% and swap memory is being utilized which is not good I believe. There a lot about 100K CLOSE\_WAIT connections. Also there a lot from FIN\_WAIT2 connections

My theory is once many clients hit haproxy at once it starts to put connections to backlog and while processing connections new arrives, those which are processing already has delay so later they doesn’t make sens for the client and client retries with new connections resulting in more connections in backlog…

Is there any way to protect haproxy or linux from being overloaded by dropping connections, is it a good practice?

> **haproxy.cfg**
>
> global  
> log /dev/log local0  
> log /dev/log local1 debug  
> maxconnrate 280  
> maxsessrate 280  
> maxconn 100000  
> daemon  
> user haproxy  
> group haproxy  
> stats socket /var/run/haproxy.sock level admin  
> defaults  
> mode tcp  
> log global  
> option tcplog  
> option dontlognull  
> timeout connect 5s  
> timeout client 24h  
> timeout server 60m  
> maxconn 100000
> 
> frontend service\_name  
> bind 50.1.1.3:1234  
> acl p1234 dst\_port 1234  
> use\_backend service\_name\_1234 if p1234
> 
> backend service\_name\_1234  
> balance leastconn  
> option independant-streams  
> server server\_vir1 x1:1234 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 172.1.2.3  
> server server\_vir2 x2:1234 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 172.1.2.4  
> server server\_vir3 x3:1234 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 172.1.2.5

> **sysctl config**
>
> abi.vsyscall32 = 1  
> crypto.fips\_enabled = 1  
> debug.exception-trace = 1  
> debug.kprobes-optimization = 1  
> debug.panic\_on\_rcu\_stall = 0  
> dev.hpet.max-user-freq = 64  
> fs.aio-max-nr = 65536  
> fs.aio-nr = 0  
> fs.nr\_open = 1048576  
> fs.overflowgid = 65534  
> fs.overflowuid = 65534  
> fs.pipe-max-size = 1048576  
> fs.pipe-user-pages-hard = 0  
> fs.pipe-user-pages-soft = 16384  
> fs.protected\_hardlinks = 1  
> fs.protected\_symlinks = 1  
> fs.quota.allocated\_dquots = 0  
> fs.quota.cache\_hits = 0  
> fs.quota.drops = 0  
> fs.quota.free\_dquots = 0  
> fs.quota.lookups = 0  
> fs.quota.reads = 0  
> fs.quota.syncs = 4  
> fs.quota.warnings = 1  
> fs.quota.writes = 0  
> fs.suid\_dumpable = 2  
> kernel.random.entropy\_avail = 3472  
> kernel.random.poolsize = 4096  
> kernel.random.read\_wakeup\_threshold = 64  
> kernel.random.urandom\_min\_reseed\_secs = 60  
> kernel.random.write\_wakeup\_threshold = 896  
> kernel.randomize\_va\_space = 2  
> kernel.real-root-dev = 0  
> kernel.sched\_autogroup\_enabled = 0  
> kernel.sched\_cfs\_bandwidth\_slice\_us = 5000  
> kernel.sched\_child\_runs\_first = 0  
> kernel.sched\_domain.cpu0.domain0.busy\_factor = 32  
> kernel.sched\_domain.cpu0.domain0.busy\_idx = 2  
> kernel.sched\_domain.cpu0.domain0.cache\_nice\_tries = 1  
> kernel.sched\_domain.cpu0.domain0.flags = 4143  
> kernel.sched\_domain.cpu0.domain0.forkexec\_idx = 0  
> kernel.sched\_domain.cpu0.domain0.idle\_idx = 1  
> kernel.sched\_domain.cpu0.domain0.imbalance\_pct = 125  
> kernel.sched\_domain.cpu0.domain0.max\_interval = 4  
> kernel.sched\_domain.cpu0.domain0.max\_newidle\_lb\_cost = 17558  
> kernel.sched\_domain.cpu0.domain0.min\_interval = 2  
> kernel.sched\_domain.cpu0.domain0.name = DIE  
> kernel.sched\_domain.cpu0.domain0.newidle\_idx = 0  
> kernel.sched\_domain.cpu0.domain0.wake\_idx = 0  
> kernel.sched\_domain.cpu1.domain0.busy\_factor = 32  
> kernel.sched\_domain.cpu1.domain0.busy\_idx = 2  
> kernel.sched\_domain.cpu1.domain0.cache\_nice\_tries = 1  
> kernel.sched\_domain.cpu1.domain0.flags = 4143  
> kernel.sched\_domain.cpu1.domain0.forkexec\_idx = 0  
> kernel.sched\_domain.cpu1.domain0.idle\_idx = 1  
> kernel.sched\_domain.cpu1.domain0.imbalance\_pct = 125  
> kernel.sched\_domain.cpu1.domain0.max\_interval = 4  
> kernel.sched\_domain.cpu1.domain0.max\_newidle\_lb\_cost = 9445  
> kernel.sched\_domain.cpu1.domain0.min\_interval = 2  
> kernel.sched\_domain.cpu1.domain0.name = DIE  
> kernel.sched\_domain.cpu1.domain0.newidle\_idx = 0  
> kernel.sched\_domain.cpu1.domain0.wake\_idx = 0  
> kernel.sched\_latency\_ns = 12000000  
> kernel.sched\_migration\_cost\_ns = 500000  
> kernel.sched\_min\_granularity\_ns = 1500000  
> kernel.sched\_nr\_migrate = 32  
> kernel.sched\_rr\_timeslice\_ms = 100  
> kernel.sched\_rt\_period\_us = 1000000  
> kernel.sched\_rt\_runtime\_us = 950000  
> kernel.sched\_schedstats = 0  
> kernel.sched\_shares\_window\_ns = 10000000  
> kernel.sched\_time\_avg\_ms = 1000  
> kernel.sched\_tunable\_scaling = 1  
> kernel.sched\_wakeup\_granularity\_ns = 2000000  
> kernel.sem = 250 32000 32 128  
> kernel.sem\_next\_id = -1  
> kernel.shm\_next\_id = -1  
> kernel.shm\_rmid\_forced = 0  
> kernel.shmall = 18446744073692774399  
> kernel.shmmax = 18446744073692774399  
> kernel.shmmni = 4096  
> kernel.softlockup\_all\_cpu\_backtrace = 0  
> kernel.softlockup\_panic = 0  
> kernel.stack\_tracer\_enabled = 0  
> kernel.sysrq = 16  
> kernel.tainted = 0  
> kernel.threads-max = 62405  
> kernel.timer\_migration = 1  
> kernel.traceoff\_on\_warning = 0  
> kernel.unknown\_nmi\_panic = 1  
> kernel.usermodehelper.bset = 4294967295 31  
> kernel.usermodehelper.inheritable = 4294967295 31  
> kernel.version = #1 SMP Fri Oct 13 10:46:25 EDT 2017  
> kernel.watchdog = 1  
> kernel.watchdog\_cpumask = 0-1  
> kernel.watchdog\_thresh = 10  
> kernel.yama.ptrace\_scope = 0  
> net.core.bpf\_jit\_enable = 0  
> net.core.busy\_poll = 0  
> net.core.busy\_read = 0  
> net.core.default\_qdisc = pfifo\_fast  
> net.core.dev\_weight = 64  
> net.core.message\_burst = 10  
> net.core.message\_cost = 5  
> net.core.netdev\_budget = 300  
> net.core.netdev\_max\_backlog = 1000  
> net.core.netdev\_rss\_key = 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00  
> net.core.netdev\_tstamp\_prequeue = 1  
> net.core.optmem\_max = 20480  
> net.core.rmem\_default = 212992  
> net.core.rmem\_max = 212992  
> net.core.rps\_sock\_flow\_entries = 0  
> net.core.somaxconn = 1024  
> net.core.warnings = 1  
> net.core.wmem\_default = 212992  
> net.core.wmem\_max = 212992  
> net.core.xfrm\_acq\_expires = 30  
> net.core.xfrm\_aevent\_etime = 10  
> net.core.xfrm\_aevent\_rseqth = 2  
> net.core.xfrm\_larval\_drop = 1  
> net.ipv4.cipso\_cache\_bucket\_size = 10  
> net.ipv4.cipso\_cache\_enable = 1  
> net.ipv4.cipso\_rbm\_optfmt = 0  
> net.ipv4.cipso\_rbm\_strictvalid = 1  
> net.ipv4.conf.all.accept\_local = 0  
> net.ipv4.conf.all.accept\_redirects = 0  
> net.ipv4.conf.all.accept\_source\_route = 0  
> net.ipv4.conf.all.arp\_accept = 0  
> net.ipv4.conf.all.arp\_announce = 0  
> net.ipv4.conf.all.arp\_filter = 0  
> net.ipv4.conf.all.arp\_ignore = 0  
> net.ipv4.conf.all.arp\_notify = 0  
> net.ipv4.conf.all.bootp\_relay = 0  
> net.ipv4.conf.all.disable\_policy = 0  
> net.ipv4.conf.all.disable\_xfrm = 0  
> net.ipv4.conf.all.force\_igmp\_version = 2  
> net.ipv4.conf.all.forwarding = 1  
> net.ipv4.conf.all.log\_martians = 1  
> net.ipv4.conf.all.mc\_forwarding = 0  
> net.ipv4.conf.all.medium\_id = 0  
> net.ipv4.conf.all.promote\_secondaries = 1  
> net.ipv4.conf.all.proxy\_arp = 0  
> net.ipv4.conf.all.proxy\_arp\_pvlan = 0  
> net.ipv4.conf.all.route\_localnet = 0  
> net.ipv4.conf.all.rp\_filter = 1  
> net.ipv4.conf.all.secure\_redirects = 0  
> net.ipv4.conf.all.send\_redirects = 0  
> net.ipv4.conf.all.shared\_media = 1  
> net.ipv4.conf.all.src\_valid\_mark = 0  
> net.ipv4.conf.all.tag = 0  
> net.ipv4.conf.default.accept\_local = 0  
> net.ipv4.conf.default.accept\_redirects = 0  
> net.ipv4.conf.default.accept\_source\_route = 0  
> net.ipv4.conf.default.arp\_accept = 0  
> net.ipv4.conf.default.arp\_announce = 0  
> net.ipv4.conf.default.arp\_filter = 0  
> net.ipv4.conf.default.arp\_ignore = 0  
> net.ipv4.conf.default.arp\_notify = 0  
> net.ipv4.conf.default.bootp\_relay = 0  
> net.ipv4.conf.default.disable\_policy = 0  
> net.ipv4.conf.default.disable\_xfrm = 0  
> net.ipv4.conf.default.force\_igmp\_version = 2  
> net.ipv4.conf.default.forwarding = 1  
> net.ipv4.conf.default.log\_martians = 0  
> net.ipv4.conf.default.mc\_forwarding = 0  
> net.ipv4.conf.default.medium\_id = 0  
> net.ipv4.conf.default.promote\_secondaries = 1  
> net.ipv4.conf.default.proxy\_arp = 0  
> net.ipv4.conf.default.proxy\_arp\_pvlan = 0  
> net.ipv4.conf.default.route\_localnet = 0  
> net.ipv4.conf.default.rp\_filter = 1  
> net.ipv4.conf.default.secure\_redirects = 0  
> net.ipv4.conf.default.send\_redirects = 0  
> net.ipv4.conf.default.shared\_media = 1  
> net.ipv4.conf.default.src\_valid\_mark = 0  
> net.ipv4.conf.default.tag = 0  
> net.ipv4.conf.eth0.accept\_local = 0  
> net.ipv4.conf.eth0.accept\_redirects = 0  
> net.ipv4.conf.eth0.accept\_source\_route = 0  
> net.ipv4.conf.eth0.arp\_accept = 0  
> net.ipv4.conf.eth0.arp\_announce = 0  
> net.ipv4.conf.eth0.arp\_filter = 0  
> net.ipv4.conf.eth0.arp\_ignore = 0  
> net.ipv4.conf.eth0.arp\_notify = 0  
> net.ipv4.conf.eth0.bootp\_relay = 0  
> net.ipv4.conf.eth0.disable\_policy = 0  
> net.ipv4.conf.eth0.disable\_xfrm = 0  
> net.ipv4.conf.eth0.force\_igmp\_version = 0  
> net.ipv4.conf.eth0.forwarding = 1  
> net.ipv4.conf.eth0.log\_martians = 0  
> net.ipv4.conf.eth0.mc\_forwarding = 0  
> net.ipv4.conf.eth0.medium\_id = 0  
> net.ipv4.conf.eth0.promote\_secondaries = 1  
> net.ipv4.conf.eth0.proxy\_arp = 0  
> net.ipv4.conf.eth0.proxy\_arp\_pvlan = 0  
> net.ipv4.conf.eth0.route\_localnet = 0  
> net.ipv4.conf.eth0.rp\_filter = 1  
> net.ipv4.conf.eth0.secure\_redirects = 0  
> net.ipv4.conf.eth0.send\_redirects = 0  
> net.ipv4.conf.eth0.shared\_media = 1  
> net.ipv4.conf.eth0.src\_valid\_mark = 0  
> net.ipv4.conf.eth0.tag = 0  
> net.ipv4.conf.eth2.accept\_local = 0  
> net.ipv4.conf.eth2.accept\_redirects = 0  
> net.ipv4.conf.eth2.accept\_source\_route = 0  
> net.ipv4.conf.eth2.arp\_accept = 0  
> net.ipv4.conf.eth2.arp\_announce = 0  
> net.ipv4.conf.eth2.arp\_filter = 0  
> net.ipv4.conf.eth2.arp\_ignore = 0  
> net.ipv4.conf.eth2.arp\_notify = 0  
> net.ipv4.conf.eth2.bootp\_relay = 0  
> net.ipv4.conf.eth2.disable\_policy = 0  
> net.ipv4.conf.eth2.disable\_xfrm = 0  
> net.ipv4.conf.eth2.force\_igmp\_version = 0  
> net.ipv4.conf.eth2.forwarding = 1  
> net.ipv4.conf.eth2.log\_martians = 0  
> net.ipv4.conf.eth2.mc\_forwarding = 0  
> net.ipv4.conf.eth2.medium\_id = 0  
> net.ipv4.conf.eth2.promote\_secondaries = 1  
> net.ipv4.conf.eth2.proxy\_arp = 0  
> net.ipv4.conf.eth2.proxy\_arp\_pvlan = 0  
> net.ipv4.conf.eth2.route\_localnet = 0  
> net.ipv4.conf.eth2.rp\_filter = 1  
> net.ipv4.conf.eth2.secure\_redirects = 0  
> net.ipv4.conf.eth2.send\_redirects = 0  
> net.ipv4.conf.eth2.shared\_media = 1  
> net.ipv4.conf.eth2.src\_valid\_mark = 0  
> net.ipv4.conf.eth2.tag = 0  
> net.ipv4.conf.lo.accept\_local = 0  
> net.ipv4.conf.lo.accept\_redirects = 1  
> net.ipv4.conf.lo.accept\_source\_route = 1  
> net.ipv4.conf.lo.arp\_accept = 0  
> net.ipv4.conf.lo.arp\_announce = 0  
> net.ipv4.conf.lo.arp\_filter = 0  
> net.ipv4.conf.lo.arp\_ignore = 0  
> net.ipv4.conf.lo.arp\_notify = 0  
> net.ipv4.conf.lo.bootp\_relay = 0  
> net.ipv4.conf.lo.disable\_policy = 1  
> net.ipv4.conf.lo.disable\_xfrm = 1  
> net.ipv4.conf.lo.force\_igmp\_version = 0  
> net.ipv4.conf.lo.forwarding = 1  
> net.ipv4.conf.lo.log\_martians = 0  
> net.ipv4.conf.lo.mc\_forwarding = 0  
> net.ipv4.conf.lo.medium\_id = 0  
> net.ipv4.conf.lo.promote\_secondaries = 0  
> net.ipv4.conf.lo.proxy\_arp = 0  
> net.ipv4.conf.lo.proxy\_arp\_pvlan = 0  
> net.ipv4.conf.lo.route\_localnet = 0  
> net.ipv4.conf.lo.rp\_filter = 0  
> net.ipv4.conf.lo.secure\_redirects = 1  
> net.ipv4.conf.lo.send\_redirects = 1  
> net.ipv4.conf.lo.shared\_media = 1  
> net.ipv4.conf.lo.src\_valid\_mark = 0  
> net.ipv4.conf.lo.tag = 0  
> net.ipv4.icmp\_echo\_ignore\_all = 0  
> net.ipv4.icmp\_echo\_ignore\_broadcasts = 1  
> net.ipv4.icmp\_errors\_use\_inbound\_ifaddr = 0  
> net.ipv4.icmp\_ignore\_bogus\_error\_responses = 1  
> net.ipv4.icmp\_msgs\_burst = 50  
> net.ipv4.icmp\_msgs\_per\_sec = 1000  
> net.ipv4.icmp\_ratelimit = 1000  
> net.ipv4.icmp\_ratemask = 6168  
> net.ipv4.igmp\_max\_memberships = 20  
> net.ipv4.igmp\_max\_msf = 10  
> net.ipv4.igmp\_qrv = 2  
> net.ipv4.inet\_peer\_maxttl = 600  
> net.ipv4.inet\_peer\_minttl = 120  
> net.ipv4.inet\_peer\_threshold = 65664  
> net.ipv4.ip\_default\_ttl = 64  
> net.ipv4.ip\_dynaddr = 0  
> net.ipv4.ip\_early\_demux = 1  
> net.ipv4.ip\_forward = 1  
> net.ipv4.ip\_forward\_use\_pmtu = 0  
> net.ipv4.ip\_local\_port\_range = 1024 65535  
> net.ipv4.ip\_local\_reserved\_ports =  
> net.ipv4.ip\_no\_pmtu\_disc = 0  
> net.ipv4.ip\_nonlocal\_bind = 1  
> net.ipv4.ipfrag\_high\_thresh = 4194304  
> net.ipv4.ipfrag\_low\_thresh = 3145728  
> net.ipv4.ipfrag\_max\_dist = 64  
> net.ipv4.ipfrag\_secret\_interval = 600  
> net.ipv4.ipfrag\_time = 30  
> net.ipv4.neigh.default.anycast\_delay = 100  
> net.ipv4.neigh.default.app\_solicit = 0  
> net.ipv4.neigh.default.base\_reachable\_time\_ms = 30000  
> net.ipv4.neigh.default.delay\_first\_probe\_time = 5  
> net.ipv4.neigh.default.gc\_interval = 30  
> net.ipv4.neigh.default.gc\_stale\_time = 60  
> net.ipv4.neigh.default.gc\_thresh1 = 128  
> net.ipv4.neigh.default.gc\_thresh2 = 512  
> net.ipv4.neigh.default.gc\_thresh3 = 1024  
> net.ipv4.neigh.default.locktime = 100  
> net.ipv4.neigh.default.mcast\_solicit = 3  
> net.ipv4.neigh.default.proxy\_delay = 80  
> net.ipv4.neigh.default.proxy\_qlen = 64  
> net.ipv4.neigh.default.retrans\_time\_ms = 1000  
> net.ipv4.neigh.default.ucast\_solicit = 3  
> net.ipv4.neigh.default.unres\_qlen = 31  
> net.ipv4.neigh.default.unres\_qlen\_bytes = 65536  
> net.ipv4.neigh.eth0.anycast\_delay = 100  
> net.ipv4.neigh.eth0.app\_solicit = 0  
> net.ipv4.neigh.eth0.base\_reachable\_time\_ms = 30000  
> net.ipv4.neigh.eth0.delay\_first\_probe\_time = 5  
> net.ipv4.neigh.eth0.gc\_stale\_time = 60  
> net.ipv4.neigh.eth0.locktime = 100  
> net.ipv4.neigh.eth0.mcast\_solicit = 3  
> net.ipv4.neigh.eth0.proxy\_delay = 80  
> net.ipv4.neigh.eth0.proxy\_qlen = 64  
> net.ipv4.neigh.eth0.retrans\_time\_ms = 1000  
> net.ipv4.neigh.eth0.ucast\_solicit = 3  
> net.ipv4.neigh.eth0.unres\_qlen = 31  
> net.ipv4.neigh.eth0.unres\_qlen\_bytes = 65536  
> net.ipv4.neigh.eth1.anycast\_delay = 100  
> net.ipv4.neigh.eth1.app\_solicit = 0  
> net.ipv4.neigh.eth1.base\_reachable\_time\_ms = 30000  
> net.ipv4.neigh.eth1.delay\_first\_probe\_time = 5  
> net.ipv4.neigh.eth1.gc\_stale\_time = 60  
> net.ipv4.neigh.eth1.locktime = 100  
> net.ipv4.neigh.eth1.mcast\_solicit = 3  
> net.ipv4.neigh.eth1.proxy\_delay = 80  
> net.ipv4.neigh.eth1.proxy\_qlen = 64  
> net.ipv4.neigh.eth1.retrans\_time\_ms = 1000  
> net.ipv4.neigh.eth1.ucast\_solicit = 3  
> net.ipv4.neigh.eth1.unres\_qlen = 31  
> net.ipv4.neigh.eth1.unres\_qlen\_bytes = 65536  
> net.ipv4.neigh.eth2.anycast\_delay = 100  
> net.ipv4.neigh.eth2.app\_solicit = 0  
> net.ipv4.neigh.eth2.base\_reachable\_time\_ms = 30000  
> net.ipv4.neigh.eth2.delay\_first\_probe\_time = 5  
> net.ipv4.neigh.eth2.gc\_stale\_time = 60  
> net.ipv4.neigh.eth2.locktime = 100  
> net.ipv4.neigh.eth2.mcast\_solicit = 3  
> net.ipv4.neigh.eth2.proxy\_delay = 80  
> net.ipv4.neigh.eth2.proxy\_qlen = 64  
> net.ipv4.neigh.eth2.retrans\_time\_ms = 1000  
> net.ipv4.neigh.eth2.ucast\_solicit = 3  
> net.ipv4.neigh.eth2.unres\_qlen = 31  
> net.ipv4.neigh.eth2.unres\_qlen\_bytes = 65536  
> net.ipv4.neigh.lo.anycast\_delay = 100  
> net.ipv4.neigh.lo.app\_solicit = 0  
> net.ipv4.neigh.lo.base\_reachable\_time\_ms = 30000  
> net.ipv4.neigh.lo.delay\_first\_probe\_time = 5  
> net.ipv4.neigh.lo.gc\_stale\_time = 60  
> net.ipv4.neigh.lo.locktime = 100  
> net.ipv4.neigh.lo.mcast\_solicit = 3  
> net.ipv4.neigh.lo.proxy\_delay = 80  
> net.ipv4.neigh.lo.proxy\_qlen = 64  
> net.ipv4.neigh.lo.retrans\_time\_ms = 1000  
> net.ipv4.neigh.lo.ucast\_solicit = 3  
> net.ipv4.neigh.lo.unres\_qlen = 31  
> net.ipv4.neigh.lo.unres\_qlen\_bytes = 65536  
> net.ipv4.ping\_group\_range = 1 0  
> net.ipv4.route.error\_burst = 5000  
> net.ipv4.route.error\_cost = 1000  
> net.ipv4.route.gc\_elasticity = 8  
> net.ipv4.route.gc\_interval = 60  
> net.ipv4.route.gc\_min\_interval = 0  
> net.ipv4.route.gc\_min\_interval\_ms = 500  
> net.ipv4.route.gc\_thresh = -1  
> net.ipv4.route.gc\_timeout = 300  
> net.ipv4.route.max\_size = 2147483647  
> net.ipv4.route.min\_adv\_mss = 256  
> net.ipv4.route.min\_pmtu = 552  
> net.ipv4.route.mtu\_expires = 600  
> net.ipv4.route.redirect\_load = 20  
> net.ipv4.route.redirect\_number = 9  
> net.ipv4.route.redirect\_silence = 20480  
> net.ipv4.tcp\_abort\_on\_overflow = 0  
> net.ipv4.tcp\_adv\_win\_scale = 1  
> net.ipv4.tcp\_allowed\_congestion\_control = cubic reno  
> net.ipv4.tcp\_app\_win = 31  
> net.ipv4.tcp\_autocorking = 1  
> net.ipv4.tcp\_available\_congestion\_control = cubic reno  
> net.ipv4.tcp\_base\_mss = 1024  
> net.ipv4.tcp\_challenge\_ack\_limit = 2147483647  
> net.ipv4.tcp\_congestion\_control = cubic  
> net.ipv4.tcp\_dsack = 1  
> net.ipv4.tcp\_early\_retrans = 3  
> net.ipv4.tcp\_ecn = 2  
> net.ipv4.tcp\_fack = 1  
> net.ipv4.tcp\_fastopen = 0  
> net.ipv4.tcp\_fastopen\_key = 00000000-00000000-00000000-00000000  
> net.ipv4.tcp\_fin\_timeout = 60  
> net.ipv4.tcp\_frto = 2  
> net.ipv4.tcp\_invalid\_ratelimit = 500  
> net.ipv4.tcp\_keepalive\_intvl = 75  
> net.ipv4.tcp\_keepalive\_probes = 9  
> net.ipv4.tcp\_keepalive\_time = 7200  
> net.ipv4.tcp\_limit\_output\_bytes = 262144  
> net.ipv4.tcp\_low\_latency = 0  
> net.ipv4.tcp\_max\_orphans = 32768  
> net.ipv4.tcp\_max\_ssthresh = 0  
> net.ipv4.tcp\_max\_syn\_backlog = 1280  
> net.ipv4.tcp\_max\_tw\_buckets = 32768  
> net.ipv4.tcp\_mem = 185361 247148 370722  
> net.ipv4.tcp\_min\_tso\_segs = 2  
> net.ipv4.tcp\_moderate\_rcvbuf = 1  
> net.ipv4.tcp\_mtu\_probing = 2  
> net.ipv4.tcp\_no\_metrics\_save = 0  
> net.ipv4.tcp\_notsent\_lowat = -1  
> net.ipv4.tcp\_orphan\_retries = 0  
> net.ipv4.tcp\_reordering = 3  
> net.ipv4.tcp\_retrans\_collapse = 1  
> net.ipv4.tcp\_retries1 = 3  
> net.ipv4.tcp\_retries2 = 15  
> net.ipv4.tcp\_rfc1337 = 0  
> net.ipv4.tcp\_rmem = 4096 87380 6291456  
> net.ipv4.tcp\_sack = 1  
> net.ipv4.tcp\_slow\_start\_after\_idle = 0  
> net.ipv4.tcp\_stdurg = 0  
> net.ipv4.tcp\_syn\_retries = 6  
> net.ipv4.tcp\_synack\_retries = 5  
> net.ipv4.tcp\_syncookies = 1  
> net.ipv4.tcp\_thin\_dupack = 0  
> net.ipv4.tcp\_thin\_linear\_timeouts = 0  
> net.ipv4.tcp\_timestamps = 1  
> net.ipv4.tcp\_tso\_win\_divisor = 3  
> net.ipv4.tcp\_tw\_recycle = 0  
> net.ipv4.tcp\_tw\_reuse = 1  
> net.ipv4.tcp\_window\_scaling = 1  
> net.ipv4.tcp\_wmem = 4096 16384 4194304  
> net.ipv4.tcp\_workaround\_signed\_windows = 0  
> net.ipv4.udp\_mem = 187218 249624 374436  
> net.ipv4.udp\_rmem\_min = 4096  
> net.ipv4.udp\_wmem\_min = 4096  
> net.ipv4.vs.am\_droprate = 10  
> net.ipv4.vs.amemthresh = 1024  
> net.ipv4.vs.backup\_only = 0  
> net.ipv4.vs.cache\_bypass = 0  
> net.ipv4.vs.conn\_reuse\_mode = 1  
> net.ipv4.vs.conntrack = 0  
> net.ipv4.vs.drop\_entry = 0  
> net.ipv4.vs.drop\_packet = 0  
> net.ipv4.vs.expire\_nodest\_conn = 0  
> net.ipv4.vs.expire\_quiescent\_template = 0  
> net.ipv4.vs.nat\_icmp\_send = 0  
> net.ipv4.vs.pmtu\_disc = 1  
> net.ipv4.vs.secure\_tcp = 0  
> net.ipv4.vs.snat\_reroute = 1  
> net.ipv4.vs.sync\_ports = 1  
> net.ipv4.vs.sync\_qlen\_max = 61800  
> net.ipv4.vs.sync\_refresh\_period = 0  
> net.ipv4.vs.sync\_retries = 0  
> net.ipv4.vs.sync\_sock\_size = 0  
> net.ipv4.vs.sync\_threshold = 3 50  
> net.ipv4.vs.sync\_version = 1  
> net.ipv4.xfrm4\_gc\_thresh = 32768  
> net.netfilter.nf\_conntrack\_acct = 0  
> net.netfilter.nf\_conntrack\_buckets = 65536  
> net.netfilter.nf\_conntrack\_checksum = 1  
> net.netfilter.nf\_conntrack\_count = 7575  
> net.netfilter.nf\_conntrack\_dccp\_loose = 1  
> net.netfilter.nf\_conntrack\_dccp\_timeout\_closereq = 64  
> net.netfilter.nf\_conntrack\_dccp\_timeout\_closing = 64  
> net.netfilter.nf\_conntrack\_dccp\_timeout\_open = 43200  
> net.netfilter.nf\_conntrack\_dccp\_timeout\_partopen = 480  
> net.netfilter.nf\_conntrack\_dccp\_timeout\_request = 240  
> net.netfilter.nf\_conntrack\_dccp\_timeout\_respond = 480  
> net.netfilter.nf\_conntrack\_dccp\_timeout\_timewait = 240  
> net.netfilter.nf\_conntrack\_events = 1  
> net.netfilter.nf\_conntrack\_events\_retry\_timeout = 15  
> net.netfilter.nf\_conntrack\_expect\_max = 1024  
> net.netfilter.nf\_conntrack\_generic\_timeout = 600  
> net.netfilter.nf\_conntrack\_helper = 1  
> net.netfilter.nf\_conntrack\_icmp\_timeout = 30  
> net.netfilter.nf\_conntrack\_log\_invalid = 0  
> net.netfilter.nf\_conntrack\_max = 1048576  
> net.netfilter.nf\_conntrack\_sctp\_timeout\_closed = 10  
> net.netfilter.nf\_conntrack\_sctp\_timeout\_cookie\_echoed = 3  
> net.netfilter.nf\_conntrack\_sctp\_timeout\_cookie\_wait = 3  
> net.netfilter.nf\_conntrack\_sctp\_timeout\_established = 432000  
> net.netfilter.nf\_conntrack\_sctp\_timeout\_heartbeat\_acked = 210  
> net.netfilter.nf\_conntrack\_sctp\_timeout\_heartbeat\_sent = 30  
> net.netfilter.nf\_conntrack\_sctp\_timeout\_shutdown\_ack\_sent = 3  
> net.netfilter.nf\_conntrack\_sctp\_timeout\_shutdown\_recd = 0  
> net.netfilter.nf\_conntrack\_sctp\_timeout\_shutdown\_sent = 0  
> net.netfilter.nf\_conntrack\_tcp\_be\_liberal = 0  
> net.netfilter.nf\_conntrack\_tcp\_loose = 0  
> net.netfilter.nf\_conntrack\_tcp\_max\_retrans = 3  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_close = 10  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_close\_wait = 60  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_established = 3600  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_fin\_wait = 120  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_last\_ack = 30  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_max\_retrans = 300  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_syn\_recv = 60  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_syn\_sent = 120  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_time\_wait = 120  
> net.netfilter.nf\_conntrack\_tcp\_timeout\_unacknowledged = 300  
> net.netfilter.nf\_conntrack\_timestamp = 0  
> net.netfilter.nf\_conntrack\_udp\_timeout = 30  
> net.netfilter.nf\_conntrack\_udp\_timeout\_stream = 180  
> net.netfilter.nf\_log.0 = NONE  
> net.netfilter.nf\_log.1 = NONE  
> net.netfilter.nf\_log.2 = nfnetlink\_log  
> net.netfilter.nf\_log.3 = NONE  
> net.netfilter.nf\_log.4 = NONE  
> net.netfilter.nf\_log.5 = NONE  
> net.netfilter.nf\_log.6 = NONE  
> net.netfilter.nf\_log.7 = NONE  
> net.netfilter.nf\_log.8 = NONE  
> net.netfilter.nf\_log.9 = NONE  
> net.nf\_conntrack\_max = 1048576  
> net.unix.max\_dgram\_qlen = 512  
> sunrpc.max\_resvport = 1023  
> sunrpc.min\_resvport = 665  
> sunrpc.nfs\_debug = 0x0000  
> sunrpc.nfsd\_debug = 0x0000  
> sunrpc.nlm\_debug = 0x0000  
> sunrpc.rpc\_debug = 0x0000  
> sunrpc.tcp\_fin\_timeout = 15  
> sunrpc.tcp\_max\_slot\_table\_entries = 65536  
> sunrpc.tcp\_slot\_table\_entries = 2  
> sunrpc.transports = tcp 1048576  
> sunrpc.transports = udp 32768  
> sunrpc.transports = tcp-bc 1048576  
> sunrpc.udp\_slot\_table\_entries = 16  
> user.max\_ipc\_namespaces = 31202  
> user.max\_mnt\_namespaces = 31202  
> user.max\_net\_namespaces = 31202  
> user.max\_pid\_namespaces = 31202  
> user.max\_user\_namespaces = 0  
> user.max\_uts\_namespaces = 31202  
> vm.admin\_reserve\_kbytes = 8192  
> vm.block\_dump = 0  
> vm.dirty\_background\_bytes = 0  
> vm.dirty\_background\_ratio = 10  
> vm.dirty\_bytes = 0  
> vm.dirty\_expire\_centisecs = 3000  
> vm.dirty\_ratio = 20  
> vm.dirty\_writeback\_centisecs = 500  
> vm.drop\_caches = 0  
> vm.extfrag\_threshold = 500  
> vm.hugepages\_treat\_as\_movable = 0  
> vm.hugetlb\_shm\_group = 0  
> vm.laptop\_mode = 0  
> vm.legacy\_va\_layout = 0  
> vm.lowmem\_reserve\_ratio = 256 256 32  
> vm.max\_map\_count = 65530  
> vm.memory\_failure\_early\_kill = 0  
> vm.memory\_failure\_recovery = 1  
> vm.min\_free\_kbytes = 67584  
> vm.min\_slab\_ratio = 5  
> vm.min\_unmapped\_ratio = 1  
> vm.mmap\_min\_addr = 4096  
> vm.mmap\_rnd\_bits = 28  
> vm.mmap\_rnd\_compat\_bits = 8  
> vm.nr\_hugepages = 0  
> vm.nr\_hugepages\_mempolicy = 0  
> vm.nr\_overcommit\_hugepages = 0  
> vm.nr\_pdflush\_threads = 0  
> vm.numa\_zonelist\_order = default  
> vm.oom\_dump\_tasks = 1  
> vm.oom\_kill\_allocating\_task = 0  
> vm.overcommit\_kbytes = 0  
> vm.overcommit\_memory = 0  
> vm.overcommit\_ratio = 50  
> vm.page-cluster = 3  
> vm.panic\_on\_oom = 0  
> vm.percpu\_pagelist\_fraction = 0  
> vm.stat\_interval = 1  
> vm.swappiness = 60  
> vm.user\_reserve\_kbytes = 131072  
> vm.vfs\_cache\_pressure = 100  
> vm.zone\_reclaim\_mode = 0

> **hostnamectl**
>
> Icon name: computer-vm  
> Chassis: vm  
> Virtualization: vmware  
> Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)  
> CPE OS Name: cpe:/o:redhat:enterprise\_linux:7.5:GA:server  
> Kernel: Linux 3.10.0-862.11.6.el7.x86\_64  
> Architecture: x86-64

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [November 16, 2018, 1:30pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/2 "2018-11-16T13:30:08Z")

</div>

Is the CPU spend in haproxy, another application or the kernel? Why is it swapping? How much RAM do you have, and how much is used in haproxy?

Please provide the output of `haproxy -vv`.

I’m not sure why you’d configure maxconnrate and maxsessrate at all? That will slow down the amount of new connections that can be established. And it seems that is one of the problems you are facing.

Also, you have huge timeouts (24h). That is a very bad idea, because connections will take that long to timeout.

---

<div class="post-metadata">

**Author:** ![ndakotaeq](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@ndakotaeq](https://discourse.haproxy.org/u/ndakotaeq)\
**Post date:** [November 16, 2018, 2:18pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/3 "2018-11-16T14:18:42Z")

</div>

VM has 8Gb RAM int total.

I am thinking to:

1. do not allow to use swap.
2. reduce client timeout 24h
3. reduce net.ipv4.tcp\_keepalive\_time

What I have fond is that FIN\_WAIT2 are dominating and latest file in /proc//fd is about ~2h

maxconnrate and maxsessrate was configured by intend to protect the servers. Seems like new connections are being added to backlog… The rate limit actually never get hit,

> **Summary**
>
> # haproxy -vv
> 
> HA-Proxy version 1.5.18 2016/05/10  
> Copyright 2000-2016 Willy Tarreau [willy@haproxy.org](mailto:willy@haproxy.org)
> 
> Build options :  
> TARGET = linux2628  
> CPU = generic  
> CC = gcc  
> CFLAGS = -O2 -g -fno-strict-aliasing -DTCP\_USER\_TIMEOUT=18  
> OPTIONS = USE\_LINUX\_TPROXY=1 USE\_ZLIB=1 USE\_REGPARM=1 USE\_OPENSSL=1 USE\_PCRE=1
> 
> Default settings :  
> maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
> 
> Encrypted password support via crypt(3): yes  
> Built with zlib version : 1.2.7  
> Compression algorithms supported : identity, deflate, gzip  
> Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017  
> Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017  
> OpenSSL library supports TLS extensions : yes  
> OpenSSL library supports SNI : yes  
> OpenSSL library supports prefer-server-ciphers : yes  
> Built with PCRE version : 8.32 2012-11-30  
> PCRE library supports JIT : no (USE\_PCRE\_JIT not set)  
> Built with transparent proxy support using: IP\_TRANSPARENT IPV6\_TRANSPARENT IP\_FREEBIND
> 
> Available polling systems :  
> epoll : pref=300, test result OK  
> poll : pref=200, test result OK  
> select : pref=150, test result OK  
> Total: 3 (3 usable), will use epoll.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [November 16, 2018, 4:31pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/4 "2018-11-16T16:31:16Z")

</div>

I suggest to lower both `timeout client` and `timeout server`. What protocol are you forwarding? I assume it’s not HTTP?

From the admin socket, can you provide the output of:

- `show info`
- `show pools`
- `show stat`

---

<div class="post-metadata">

**Author:** ![ndakotaeq](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@ndakotaeq](https://discourse.haproxy.org/u/ndakotaeq)\
**Post date:** [November 16, 2018, 5:17pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/5 "2018-11-16T17:17:12Z")

</div>

There mix of protocos, everything works in tcp mode. some are web sockets.

> **show info**
>
> # echo “show info” | socat unix-connect:/var/run/haproxy.sock stdio
> 
> Name: HAProxy  
> Version: 1.5.18  
> Release\_date: 2016/05/10  
> Nbproc: 1  
> Process\_num: 1  
> Pid: 26690  
> Uptime: 2d 0h29m13s  
> Uptime\_sec: 174553  
> Memmax\_MB: 0  
> Ulimit-n: 520810  
> Maxsock: 520810  
> Maxconn: 260000  
> Hard\_maxconn: 260000  
> CurrConns: 936  
> CumConns: 415383  
> CumReq: 355230  
> MaxSslConns: 0  
> CurrSslConns: 0  
> CumSslConns: 139553  
> Maxpipes: 0  
> PipesUsed: 0  
> PipesFree: 0  
> ConnRate: 0  
> ConnRateLimit: 280  
> MaxConnRate: 280  
> SessRate: 0  
> SessRateLimit: 280  
> MaxSessRate: 280  
> SslRate: 0  
> SslRateLimit: 0  
> MaxSslRate: 8  
> SslFrontendKeyRate: 0  
> SslFrontendMaxKeyRate: 4  
> SslFrontendSessionReuse\_pct: 0  
> SslBackendKeyRate: 0  
> SslBackendMaxKeyRate: 0  
> SslCacheLookups: 139553  
> SslCacheMisses: 9  
> CompressBpsIn: 0  
> CompressBpsOut: 0  
> CompressBpsRateLim: 0  
> ZlibMemUsage: 0  
> MaxZlibMemUsage: 0  
> Tasks: 2462  
> Run\_queue: 1  
> Idle\_pct: 100  
> node: lb1  
> description:

> **show pools**
>
> # echo “show pools” | socat unix-connect:/var/run/haproxy.sock stdio
> 
> Dumping pools usage. Use SIGQUIT to flush them.
> 
> - Pool pipe (32 bytes) : 8 allocated (256 bytes), 8 used, 3 users [SHARED]
> - Pool capture (64 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
> - Pool channel (80 bytes) : 37302 allocated (2984160 bytes), 1878 used, 1 users [SHARED]
> - Pool task (112 bytes) : 20174 allocated (2259488 bytes), 2462 used, 1 users [SHARED]
> - Pool uniqueid (128 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
> - Pool sticktables (144 bytes) : 2 allocated (288 bytes), 0 used, 3 users [SHARED]
> - Pool connection (320 bytes) : 37302 allocated (11936640 bytes), 1878 used, 1 users [SHARED]
> - Pool hdr\_idx (416 bytes) : 2 allocated (832 bytes), 0 used, 1 users [SHARED]
> - Pool session (864 bytes) : 18651 allocated (16114464 bytes), 939 used, 1 users [SHARED]
> - Pool requri (1024 bytes) : 0 allocated (0 bytes), 0 used, 1 users [SHARED]
> - Pool buffer (16416 bytes) : 37302 allocated (612349632 bytes), 1878 used, 1 users [SHARED]  
> Total: 11 pools, 645645760 bytes allocated, 32667744 used.

> **show stat**
>

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [November 17, 2018, 12:58pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/6 "2018-11-17T12:58:15Z")

</div>

The configuration you posed does not match what you are currently running. Please provide the actual running configuration.

First of all, you only have 8 GB of RAM, you must no raise your global maxconn much over 150000, otherwise you will run out of RAM, which seem to happen in your configuration (give that you really use 280000 maxconn, as opposed to the 100000 that your configuration mentions.

I suggest:

- provide the actual configuration you run with
- either limit global maxconn to 150000 or increase your RAM - its about 20000 per GB of RAM
- remove `maxconnrate` and `maxsessrate`, have haproxy and the kernel handle this with the maxconn configuration
- use timeouts that actual reflect your needs, also specify much smaller timeouts for [timeout client-fin](https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4.2-timeout%20client-fin) and [timeout server-fin](https://cbonte.github.io/haproxy-dconv/1.5/configuration.html#4.2-timeout%20server-fin)

---

<div class="post-metadata">

**Author:** ![ndakotaeq](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@ndakotaeq](https://discourse.haproxy.org/u/ndakotaeq)\
**Post date:** [November 17, 2018, 1:33pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/7 "2018-11-17T13:33:10Z")

</div>

Thanks, I am going to reduce timeout client ( 24h) and timeout server (60m) to 5m both ( expecting that long live connections sends ping-pong it should not be a problem to reduce it to 5m) .  
If I don’t specify timeout client-fin, timeout server-fin will it use setting from the kernel (nf\_conntrack\_tcp\_timeout\_fin\_wait ) ? do you recommend to specify the client-fin timeout explicitly to let say 30s ?  
The configuration is actual the only difference is that it is 280000 maxconn instead of 100000  
Why are you suggesting to remove maxconnrate? it was introduced to to protect the services.

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [November 17, 2018, 2:46pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/8 "2018-11-17T14:46:42Z")

</div>

> [@ndakotaeq](#):
>
> If I don’t specify timeout client-fin, timeout server-fin will it use setting from the kernel (nf\_conntrack\_tcp\_timeout\_fin\_wait ) ?

No, that’s not about FIN\_WAIT. It’s the timeout for half-closed connections. `client-fin` will default to `timeout client` and `server-fin` will default to `timeout tunnel`, if not configured.

> [@ndakotaeq](#):
>
> do you recommend to specify the client-fin timeout explicitly to let say 30s ?

Yes, that’s what I’m recommending.

> [@ndakotaeq](#):
>
> Why are you suggesting to remove maxconnrate? it was introduced to to protect the services.

You first of all have to make sure your maxconn configuration matches the actually available RAM, this will avoid swapping and probably reduce CPU usage. Second of all fixing timeouts will make sure you don’t have tens of thousands of connections idling around, occupying lots of RAM.

`maxconnrate` and `maxsessrate` will only slow down new requests, which is the wrong thing to do, because a) you want to fix your 50 second delay problem, right? and b) because when you don’t accept connections the just will pile up in the queue and either get dropped or get delayed.

Also, I still don’t know how maxconn is actually configured across the front and backends. Please provide the actual configuration, as I cannot continue to guess what you configured. For example, share the actual configuration of your busiest frontend and backend please.

---

<div class="post-metadata">

**Author:** ![ndakotaeq](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@ndakotaeq](https://discourse.haproxy.org/u/ndakotaeq)\
**Post date:** [November 17, 2018, 9:52pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/9 "2018-11-17T21:52:03Z")

</div>

Adding haproxy.cfg, I selected only one frontend and a backend the rest of them are identical but and almost all of them has relation - 1 frontend with 1 backend

> **haproxy.cfg**
>
> global  
> log /dev/log local0  
> log /dev/log local1 notice
> 
> ```
> daemon
> user haproxy
> group haproxy
> maxconnrate 280
> maxsessrate 280
> maxconn 260000
> 
> stats socket /var/run/haproxy.sock level admin
> 
> ```
> 
> defaults  
> mode tcp  
> log global  
> option dontlognull  
> timeout connect 5s  
> timeout client 24h  
> timeout server 60m  
> maxconn 260000
> 
> peers lbs  
> peer lb1a 1.9.100.1:5555  
> peer lb2a 1.9.100.2:5555  
> peer lb3a 1.9.100.3:5555  
> peer lb1b 1.8.100.1:5555  
> peer lb2b 1.8.100.2:5555  
> peer lb3b 1.8.100.3:5555
> 
> frontend servers\_1234  
> bind 100.100.100.1:1234  
> acl p1234 dst\_port 1234  
> use\_backend servers\_1234\_1234 if p1234
> 
> backend servers\_1234\_1234  
> balance leastconn  
> option independant-streams  
> server serverB\_vir1 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.67  
> server serverB\_vir2 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.90  
> server serverB\_vir3 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.99  
> server serverB\_vir4 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.100  
> server serverB\_vir5 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.101  
> server serverB\_vir6 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.102  
> server serverB\_vir7 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.103  
> server serverB\_vir8 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.104  
> server serverB\_vir9 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.91  
> server serverB\_vir10 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.92  
> server serverB\_vir11 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.93  
> server serverB\_vir12 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.94  
> server serverB\_vir13 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.95  
> server serverB\_vir14 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.96  
> server serverB\_vir15 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.97  
> server serverB\_vir16 server-b:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.98  
> server serverC\_vir1 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.67  
> server serverC\_vir2 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.90  
> server serverC\_vir3 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.99  
> server serverC\_vir4 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.100  
> server serverC\_vir5 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.101  
> server serverC\_vir6 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.102  
> server serverC\_vir7 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.103  
> server serverC\_vir8 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.104  
> server serverC\_vir9 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.91  
> server serverC\_vir10 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.92  
> server serverC\_vir11 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.93  
> server serverC\_vir12 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.94  
> server serverC\_vir13 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.95  
> server serverC\_vir14 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.96  
> server serverC\_vir15 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.97  
> server serverC\_vir16 server-c:1234 maxconn 100000 on-marked-down shutdown-sessions check fall 3 rise 2 inter 10s slowstart 200s source 1.1.1.98

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [November 17, 2018, 10:17pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/10 "2018-11-17T22:17:11Z")

</div>

> I am looking for a way to protect my proxy from being overloaded by configuring linux to drop the connections after limit is reached so client will fail fast and retry on another load balancer ( we have 3 in a region ) For this I am thinking to reduce backlog.

Just configure `maxconn` correctly as per your RAM availability and then configure `timeout queue` to the value that you want.

---

<div class="post-metadata">

**Author:** ![ndakotaeq](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@ndakotaeq](https://discourse.haproxy.org/u/ndakotaeq)\
**Post date:** [November 18, 2018, 8:05pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/11 "2018-11-18T20:05:32Z")

</div>

I end up with setting just to better understand tcp stack  
timeout client 24h  
timeout server 60m  
timeout client-fin 2m  
timeout server-fin 2m

the problem is that: after server close the connection then client respond just after 100 seconds with its FIN and RST, and that was a problem which was solved by increasing close\_wait timer on netfilter to 120 sec so linux doesn’t filter out this connection. but I still have problem with this connection. The problem is that the connection stays in memory even after server closed the connection and timed out on server’s side ( so server doesn’t have TIME\_WAIT2 on his side ) and client sent RST after 100 second, but the connection to the client is still in TIME\_WAIT2 state. So I end up having 2 connections, 1 TIME\_WAIT2 to the client and one to the server with CLOSE\_WAIT ( however server doesn’t have TIME\_WAIT2 for this connection ) these two connections will be cleaned up in 24h as it is in timeout client, but I don’t understand why is like that, why connection not closed after client sent RST ( even it is 100 sec after LB’s FIN ) ( can it be filtered out by kernel ? ) and why client-fin didn’t kick in.

The solution would be for me to just set timeout client 5m. But I would like to understand why is that the connection that is seems to be in half-open state to client closed only by expiration of “timeout client”  
so, why : session not closed after RST sent by the client, second not closed after client-fin 2m expires.

LB - load balancer  
1.  
Client ( established ) - LB ( established )  
LB ( established ) - server ( established )  
2. Exchanging data  
3. 200 sec idle ( client doesn’t respond )  
4. Server sends “FIN”  
Server ( TIME\_WAIT2 ) - LB ( CLOSE\_WAIT )  
LB sends FIN to Client  
LB ( TIME\_WAIT2 ) - Client ( unknown )  
5. 100 sec of waiting for response from client to FIN  
Client sends FIN followed by RST  
Client ( unknown ) - LB ( TIME\_WAIT2 )  
LB sends FIN to the server  
Server ( closed ) - LB ( CLOSE\_WAIT)  
6.  
from that moment I see session in “show sess” that will expire in 24h  
and in netstats two connections

1. lb client FIN\_WAIT2
2. lb server CLOSE\_WAIT

---

<div class="post-metadata">

**Author:** ![lukastribus](https://avatars.discourse-cdn.com/v4/letter/l/7ea924/32.png) [@lukastribus](https://discourse.haproxy.org/u/lukastribus)\
**Post date:** [November 19, 2018, 5:48pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/12 "2018-11-19T17:48:16Z")

</div>

I think conntrack is causing this mess, and I can’t help you troubleshooting conntrack timers. Try confirming this is related to conntrack by disabling (or bypassing it for certain IPs), and if confirmed that conntrack is causing this, get appropriate help.

I also doubt that conntrack is sufficiently tuned to handle 260000 connections, so you probably want to take a look at the scale issues here also.

---

<div class="post-metadata">

**Author:** ![danielrangelsa](https://avatars.discourse-cdn.com/v4/letter/d/58956e/32.png) [@danielrangelsa](https://discourse.haproxy.org/u/danielrangelsa)\
**Post date:** [February 4, 2019, 9:15pm UTC](https://discourse.haproxy.org/t/understanding-maxconn-and-maxonnrate-and-delays/3230/13 "2019-02-04T21:15:54Z")

</div>

Check multi thread config.  
exemple:  
global  
nbproc 8  
cpu-map 1 0  
cpu-map 2 1  
cpu-map 3 2  
cpu-map 4 3  
cpu-map 5 4  
cpu-map 6 5  
cpu-map 7 6  
cpu-map 8 7  
stats bind-process 8