Hi,
Yesterday, I installed HAproxy 1.8.6 on CentOS 7 VM and crashed today.
what could be the reason?
thank you.
It’s not a crash, haproxy is getting OOM killed (haproxy uses too much memory).
Most likely cause is maxconn and timeout misconfiguration. Share the output of haproxy -vv and the entire configuration please.
Hey Lukastribus,
[root@hp ~]# haproxy -vv
HA-Proxy version 1.8.6 2018/04/05
Copyright 2000-2018 Willy Tarreau willy@haproxy.orgBuild options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
OPTIONS =Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built without PCRE or PCRE2 support (using libc’s regex instead)
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity(“identity”)
Built with network namespace support.Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace[root@hp ~]#
Like I said, the entire configuration is needed.
Sorry, I’ve attached the configuration file.
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#log loghost local0 info
maxconn 500000
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
#debug
#quiet
stats socket /var/lib/haproxy/stats
tune.maxrewrite 536870912
tune.bufsize 1073741824
#---------------------------------------------------------------------
# DEFAULTS KONFIG
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 6
timeout http-request 500s
timeout queue 5m
timeout connect 500s
timeout client 5000s
timeout server 24h
timeout http-keep-alive 500s
timeout check 500s
errorfile 503 /etc/haproxy/errorfiles/503sorry.http
#---------------------------------------------------------------------
# HAPROXY DASHBOARD KONFIG
#---------------------------------------------------------------------
listen stats
bind *:8080
mode http
#option forwardfor
option httpclose
stats enable
stats show-legends
stats refresh 30s
stats uri /stats
stats auth xxxx:yyyy
#---------------------------------------------------------------------
# SSH & FTP BAGLANTILARI
#---------------------------------------------------------------------
listen yyy
bind *:49999
mode tcp
option tcplog
server ftp-xxx 192.168.1.211:22
listen zzz
bind *:7001
mode http
option httplog
server web-xx 192.168.1.91:7001
...
...
...
...
#---------------------------------------------------------------------
# POOL KONFIG
#---------------------------------------------------------------------
frontend MAIN
bind *:21
bind *:80
...
...
...
...
mode http
#---------------------------------------------------------------------
# ALAN ADLARI
#---------------------------------------------------------------------
acl XXXXX_URL hdr(host) -i www.xxxx.com xxxx.com
...
...
...
...
#---------------------------------------------------------------------
# ALAN ADIMANLARININ ARKAPLAN UCLARI
#---------------------------------------------------------------------
use_backend xxxx if XXXXX_URL
...
...
...
...
#---------------------------------------------------------------------
# BACKEND SUNUCULAR
#---------------------------------------------------------------------
backend xxxx
mode http
reqadd X-Forwarded-Proto:\ https
rspadd Strict-Transport-Security:\ max-age=15768000;\ includeSubDomains #enable HSTS header for this backend
balance leastconn
option httpclose
option forwardfor
cookie JSESSIONID prefix
server web-xxxx 192.168.1.91
...
...
...
...You configured tune.bufsize, which defaults to 16 KiB, to 1 GiB, thats 64000 times the default, and raising it is explicitly warned against. You also configured maxconn 500000.
Quickly calculating the memory usage for this configuration (not even accounting for overheads here):
RAM usage = tune.bufsize * 2 * maxconn
That would be at least 1 Petabyte of RAM. But you only have a 4 GB RAM VM here.
Please remove tune.maxrewrite and tune.bufsize, and reduce maxconn. For a 4 GB RAM VM, without bufsize or maxrewrite misconfiguration, you could configure somewhere around 100000 maxconn connections for the process. But not more and not with maxrewrite misconfiguration and also not with SSL.
timeout http-request 500s timeout queue 5m timeout connect 500s timeout client 5000s timeout server 24h timeout http-keep-alive 500s timeout check 500s
This is also pretty bad. check, connect, http-keep-alive, queue and http-request should be a few seconds only. If you must, then configure client and server to a few minutes but certainly not 24 hours. A few requests and an attacker has completely congested all connections.
Thank you for your help. I increased the amount of ram to 8 GB.
I don’t know how useful this upgrade can be.
I changed it this way on your advice.
Is there any significant performance degradation with these values?
global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
#log loghost local0 info
maxconn 150000
chroot /var/lib/haproxy
user haproxy
group haproxy
daemon
#debug
#quiet
stats socket /var/lib/haproxy/stats
#tune.maxrewrite 536870912
#tune.bufsize 1073741824
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 6
timeout http-request 30s
timeout queue 1m
timeout connect 30s
timeout client 30s
timeout server 10m
timeout http-keep-alive 100s
timeout check 30s
timeout tunnel 1h
Also, would it be better to upgrade from HAproxy 1.8.6 to 2.0?
Thank you.
This should be fine, yes.
No.
No. You can upgrade to a new major release if you need the features.
Thank you for help @lukastribus