I’ve been facing a pretty difficult challenge over the past couple of months with large repetitive attacks which peak at 500k+ req/sec.
For many years, the rate limiting directives that I have in place have worked great (even as far back as HAProxy 1.5), however, after attackers switched to using HTTP/2, something changed and I’ve started facing issues (specifically, HAProxy gets overwhelmed, memory usage spikes to 20 / 30+ GB, and load also increases (due to the swap I/O and CPU usage).
I’ve upgraded to HAProxy 1.8 and I’m currently on 2.6.19, tested various potential improvements, but so far the LB is still overwhelmed during these large attacks.
First of all, here is my current HAProxy config (with some less important parts removed, since I’m confident that they are not relevant to this issue):
global
log 127.0.0.1 local2 notice
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 300000
user haproxy
group haproxy
daemon
nbthread 12
cpu-map auto:1/1-12 0-11
tune.ssl.default-dh-param 2048
tune.ssl.cachesize 50000
tune.ssl.lifetime 300
tune.http.cookielen 512
# turn on stats unix socket
stats socket /var/lib/haproxy/stats mode 755 user nrpe level admin
# utilize system-wide crypto-policies
ssl-default-bind-ciphers PROFILE=SYSTEM
ssl-default-server-ciphers PROFILE=SYSTEM
defaults
log global
option httplog
option dontlognull
option http-server-close
option log-health-checks
option tcp-smart-accept
option tcp-smart-connect
option logasap
## Keeping retries disabled as well, because it's very likely that it causes a domino effect during attacks
retries 0
timeout connect 3s
timeout client 5s ## it was 30 in the past, now testing 5 to see how it goes
timeout server 5s ## it was 30 in the past, now testing 5 to see how it goes
timeout client-fin 1s
timeout server-fin 1s
timeout queue 2s
timeout http-request 3s
timeout http-keep-alive 5s
timeout check 10s
maxconn 50000
listen lb_statistics
bind xx.xx.xx.xx:8080 ssl crt /etc/haproxy/ssl/mycrt.pem alpn h2
mode http
balance leastconn
server lb1 127.0.0.1:80
stats uri /
stats realm "HAProxy Stats"
stats auth <user>:<password>
frontend prod_website
bind xx.xx.xx.xx:80
bind xx.xx.xx.xx:443 ssl crt /etc/haproxy/ssl/mycrt.pem alpn h2,http/1.1
mode http
maxconn 100000
## Disabled for now, since it seems to cause even worse behavior during attacks
# option http-buffer-request
log 127.0.0.1 local3
capture request header User-Agent len 256
capture request header Referer len 128
capture cookie my.custom.Token len 512
tcp-request inspect-delay 1s
acl website_prod_is_http dst_port 80
acl website_prod_is_ssl dst_port 443
## ACL for missing User agent header
acl missing_ua hdr_cnt(user-agent) eq 0
acl all_whitelisted_ips src -f /etc/haproxy/all_whitelisted_ips.lst
## Track connection counters
tcp-request session track-sc2 src table website_connection_abuse if !all_whitelisted_ips
## tcp-request connection track-sc2 src table website_connection_abuse if !all_whitelisted_ips
# Limiting the connection rate per client. No more than 200 connections over 3 seconds.
tcp-request content reject if { src_conn_rate(website_connection_abuse) ge 200 } !all_whitelisted_ips
## tcp-request connection reject if { src_conn_rate(website_connection_abuse) ge 200 } !all_whitelisted_ips
# Reject if more than 200 connections from client.
tcp-request content reject if { src_conn_cur(website_connection_abuse) ge 200 } !all_whitelisted_ips
## tcp-request connection reject if { src_conn_cur(website_connection_abuse) ge 200 } !all_whitelisted_ips
acl website_too_many_requests_1 sc0_gpc0_rate(website_dummy_sticktable1) ge 300
acl website_mark_seen_1 sc0_inc_gpc0(website_dummy_sticktable1) gt 0
## tcp-request content track-sc0 src table website_dummy_sticktable1 if !all_whitelisted_ips
http-request track-sc0 src table website_dummy_sticktable1 if !all_whitelisted_ips
## Requested by Benjamin on December 5th, 2023
acl website_too_many_requests_2 sc1_gpc0_rate(website_dummy_sticktable2) ge 100
acl website_mark_seen_2 sc1_inc_gpc0(website_dummy_sticktable2) gt 0
## tcp-request content track-sc1 src,ipmask(24) table website_dummy_sticktable2 if !all_whitelisted_ips
http-request track-sc1 src,ipmask(24) table website_dummy_sticktable2 if !all_whitelisted_ips
# Deny requests with no user-agent header
http-request deny if missing_ua
http-request set-header X-Forwarded-Proto https if website_prod_is_ssl
redirect scheme https code 301 if website_prod_is_http !is_asmx_path !is_ads_url
use_backend website_rate_limiting if !all_whitelisted_ips !blacklisted-ips website_mark_seen_1 website_too_many_requests_1
use_backend website_rate_limiting2 if !all_whitelisted_ips !blacklisted-ips website_mark_seen_2 website_too_many_requests_2
default_backend website_prod_nodes
backend website_prod_nodes
mode http
balance leastconn
http-request del-header ^X-Forwarded-For:.*
option forwardfor except 127.0.0.0/8
option http-server-close
timeout queue 1s
http-check send meth GET uri /status ver HTTP/1.1 hdr Host www.mywebsite.com hdr User-Agent HAProxyLB
http-check expect string OK
server prod-node01 192.168.1.11:80 weight 5 check inter 10s rise 2 fall 5 maxconn 300 maxqueue 100
server prod-node02 192.168.1.12:80 weight 5 check inter 10s rise 2 fall 5 maxconn 300 maxqueue 100
server prod-node03 192.168.1.13:80 weight 5 check inter 10s rise 2 fall 5 maxconn 300 maxqueue 100
server prod-node04 192.168.1.14:80 weight 5 check inter 10s rise 2 fall 5 maxconn 300 maxqueue 100
server prod-node05 192.168.1.15:80 weight 5 check inter 10s rise 2 fall 5 maxconn 300 maxqueue 100
server prod-node06 192.168.1.16:80 weight 5 check inter 10s rise 2 fall 5 maxconn 300 maxqueue 100
server prod-node07 192.168.1.17:80 weight 5 check inter 10s rise 2 fall 5 maxconn 300 maxqueue 100
server prod-node08 192.168.1.18:80 weight 5 check inter 10s rise 2 fall 5 maxconn 300 maxqueue 100
backend website_dummy_sticktable1
stick-table type ip size 500k expire 30m store gpc0_rate(5m)
backend website_dummy_sticktable2
stick-table type ip size 500k expire 30m store gpc0_rate(5m)
backend website_connection_abuse
stick-table type ip size 500k expire 30m store conn_rate(3s),conn_cur
backend website_rate_limiting
mode http
timeout tarpit 2s
errorfile 500 /etc/haproxy/errorfiles/429_http
http-request tarpit
## http-request silent-drop
backend website_rate_limiting2
mode http
timeout tarpit 2s
errorfile 500 /etc/haproxy/errorfiles/429_http
http-request tarpit
## http-request silent-drop
Also, here is a small log sample from the last attack:
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_1>:46199 [21/Oct/2024:10:16:59.018] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2988 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_2>:46931 [21/Oct/2024:10:17:00.599] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1407 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_3>:15801 [21/Oct/2024:10:16:59.656] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2351 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_4>:33986 [21/Oct/2024:10:17:01.831] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/174 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_5>:46370 [21/Oct/2024:10:17:01.965] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/42 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_6>:15915 [21/Oct/2024:10:17:00.902] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1099 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_7>:5555 [21/Oct/2024:10:16:59.684] prod_website~ website_rate_limiting/<NOSRV> -1/2328/-1/-1/2327 500 274 - - PT-- 9047/8373/76013/0/0 0/0 {Dalvik/
2.1.0 (Linux; U; Android 12; Pixel 6 Pro Build/SD1A.210817.036)|} "GET https://www.mywebsite.com HTTP/2.0"
Oct 21 10:17:01 localhost haproxy[2388065]: <malicious_ip_4>:34019 [21/Oct/2024:10:17:00.976] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1009 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:01 localhost haproxy[2388065]: <malicious_ip_1>:23959 [21/Oct/2024:10:17:01.462] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/532 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_8>:51268 [21/Oct/2024:10:17:00.742] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1266 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_9>:47490 [21/Oct/2024:10:17:01.173] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/828 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_1>:46199 [21/Oct/2024:10:16:59.018] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2988 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_2>:46931 [21/Oct/2024:10:17:00.599] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1407 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_3>:15801 [21/Oct/2024:10:16:59.656] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2351 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_6>:15915 [21/Oct/2024:10:17:00.902] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1099 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_10>:34619 [21/Oct/2024:10:17:01.629] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/382 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_5>:46370 [21/Oct/2024:10:17:01.965] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/42 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:01 localhost haproxy[2388065]: <malicious_ip_1>:23959 [21/Oct/2024:10:17:01.462] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/532 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:01 localhost haproxy[2388065]: <malicious_ip_4>:34019 [21/Oct/2024:10:17:00.976] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1009 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_4>:33986 [21/Oct/2024:10:17:01.831] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/174 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_9>:47490 [21/Oct/2024:10:17:01.173] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/828 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_3>:15801 [21/Oct/2024:10:16:59.656] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2351 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_2>:46931 [21/Oct/2024:10:17:00.599] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1407 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_7>:5555 [21/Oct/2024:10:16:59.684] prod_website~ website_rate_limiting/<NOSRV> -1/2328/-1/-1/2327 500 274 - - PT-- 9047/8373/76012/0/0 0/0 {Dalvik/
2.1.0 (Linux; U; Android 12; Pixel 6 Pro Build/SD1A.210817.036)|} "GET https://www.mywebsite.com HTTP/2.0"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_8>:51268 [21/Oct/2024:10:17:00.742] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1266 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_1>:46199 [21/Oct/2024:10:16:59.018] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2988 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_6>:15915 [21/Oct/2024:10:17:00.902] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1099 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_10>:34619 [21/Oct/2024:10:17:01.629] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/382 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_5>:46370 [21/Oct/2024:10:17:01.965] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/42 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:01 localhost haproxy[2388065]: <malicious_ip_4>:34019 [21/Oct/2024:10:17:00.976] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1009 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:01 localhost haproxy[2388065]: <malicious_ip_1>:23959 [21/Oct/2024:10:17:01.462] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/532 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_4>:33986 [21/Oct/2024:10:17:01.831] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/174 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_3>:15801 [21/Oct/2024:10:16:59.656] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2351 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_9>:47490 [21/Oct/2024:10:17:01.173] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/828 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_2>:46931 [21/Oct/2024:10:17:00.599] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1407 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_7>:5555 [21/Oct/2024:10:16:59.684] prod_website~ website_rate_limiting/<NOSRV> -1/2328/-1/-1/2327 500 274 - - PT-- 9047/8373/76011/0/0 0/0 {Dalvik/
2.1.0 (Linux; U; Android 12; Pixel 6 Pro Build/SD1A.210817.036)|} "GET https://www.mywebsite.com HTTP/2.0"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_6>:15915 [21/Oct/2024:10:17:00.902] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1099 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_8>:51268 [21/Oct/2024:10:17:00.742] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1266 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_10>:34619 [21/Oct/2024:10:17:01.629] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/382 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:01 localhost haproxy[2388065]: <malicious_ip_4>:34019 [21/Oct/2024:10:17:00.976] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1009 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_1>:46199 [21/Oct/2024:10:16:59.018] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2988 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_3>:15801 [21/Oct/2024:10:16:59.656] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2351 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:01 localhost haproxy[2388065]: <malicious_ip_1>:23959 [21/Oct/2024:10:17:01.462] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/532 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_4>:33986 [21/Oct/2024:10:17:01.831] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/174 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_2>:46931 [21/Oct/2024:10:17:00.599] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1407 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:01 localhost haproxy[2388065]: <malicious_ip_4>:34019 [21/Oct/2024:10:17:00.976] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1009 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_7>:5555 [21/Oct/2024:10:16:59.684] prod_website~ website_rate_limiting/<NOSRV> -1/2328/-1/-1/2327 500 274 - - PT-- 9047/8373/76010/0/0 0/0 {Dalvik/
2.1.0 (Dalvik/2.1.0 (Linux; U; Android 12; Pixel 6 Pro Build/SD1A.210817.036)|} "GET https://www.mywebsite.com HTTP/2.0"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_6>:15915 [21/Oct/2024:10:17:00.902] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1099 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_8>:51268 [21/Oct/2024:10:17:00.742] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/1266 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_3>:15801 [21/Oct/2024:10:16:59.656] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2351 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_10>:34619 [21/Oct/2024:10:17:01.629] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/382 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_1>:46199 [21/Oct/2024:10:16:59.018] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/2988 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_9>:47490 [21/Oct/2024:10:17:01.173] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/828 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
Oct 21 10:17:02 localhost haproxy[2388065]: <malicious_ip_5>:46370 [21/Oct/2024:10:17:01.965] prod_website~ prod_website/<NOSRV> -1/-1/-1/-1/42 0 0 - - PR-- 9047/8373/0/0/0 0/0 "<BADREQ>"
The entries that contain ‘BADREQ’ are generated by the connection rate limiting seen in the above config (the “tcp-request content reject” entries), while the others with “website_rate_limiting” fall under the http-request trackers/sticktables.
One thing that stands out to me is the fact that the connections don’t seem to actually be closed (I see the same source ports for the same IPs being used). In the documentation I see that they should be: " tcp-request content reject - Closes the connection without a response once a session has been created, but before the HTTP parser has been initialized. These requests still show in your logs."
I also tried tcp-request connection reject (I left the lines commented in the config), but they don’t seem to make much of a difference and, worst of all, they are not logged - which is a real problem (especially since I have further automation that parses the logs and blacklists abuser IPs at the layer 3 / layer 4 level - which so far has been the only thing that saved me from experiencing downtimes greater than a minute or so).
Another thing is the fact that the requests take way too long to complete (i.e.: -1/-1/-1/-1/2988). When I did my own tests, the connections were cut instantly with absolutely no delay, so what gives, how come the connections are so long-lived (with a great deal of variation)?
Last but not least, I see “website_rate_limiting” requests from the same IP, during the same second, which should be tarpitted and yet they don’t seem to be.
As of now, I’m yet to figure out if it’s an issue related to my particular configuration or maybe a yet undiscovered HAProxy bug. Any advice would be greatly appreciated. Thank you!
HAProxy is running on a pretty powerful, dedicated physical server with an Intel Xeon E-2388G CPU (8 cores / 16 threads), 32 GB RAM, and very fast NVMe storage.
Also adding my sysctl.conf for completeness:
# Reboot in 5 seconds if there's a kernel panic
kernel.panic = 5
# Discourage the kernel from swapping
vm.swappiness = 10
# Use more ports for incoming connections
net.ipv4.ip_local_port_range = 10000 65535
# Increase Linux autotuning TCP buffer limits
# # Set max to 16MB for 1GE and 32M (33554432) or 54M (56623104) for 10GE
# # Don't set tcp_mem itself! Let the kernel scale it based on RAM.
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.core.rmem_default = 16777216
net.core.wmem_default = 16777216
net.core.optmem_max = 40960
net.ipv4.tcp_rmem = 4096 16384 16777216
net.ipv4.tcp_wmem = 4096 8192 16777216
#net.ipv4.tcp_mem = 4096 16384 16777216
#net.ipv4.tcp_rmem = 4096 87380 16777216
#net.ipv4.tcp_wmem = 4096 65536 16777216
# Make room for more TIME_WAIT sockets due to more clients,
# # and allow them to be reused if we run out of sockets
# # Also increase the max packet backlog
net.core.netdev_max_backlog = 50000
net.ipv4.tcp_max_syn_backlog = 30000
net.ipv4.tcp_max_tw_buckets = 2000000
net.ipv4.tcp_tw_reuse = 1
# Decrease the time that sockets stay in the TIME_WAIT state
net.ipv4.tcp_fin_timeout = 5
# Disable TCP slow start on idle connections
net.ipv4.tcp_slow_start_after_idle = 0
# Disable source routing and redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
# Increase the max orphans and max connections
net.ipv4.tcp_max_orphans = 500000
net.ipv4.tcp_synack_retries = 2
net.core.somaxconn = 65535
# Protect Against TCP Time-Wait attack
net.ipv4.tcp_rfc1337 = 1
# Increase the maximum number of netfilter handled connections
#net.netfilter.nf_conntrack_max = 524288
net.netfilter.nf_conntrack_max = 1048576
net.netfilter.nf_conntrack_tcp_timeout_established = 54000
net.netfilter.nf_conntrack_generic_timeout = 120
# Prevent TCP SACK PANICK attack:
net.ipv4.tcp_sack = 0