HAProxy community

Haproxy 1.8.6 crashed

Hi,

Yesterday, I installed HAproxy 1.8.6 on CentOS 7 VM and crashed today.

what could be the reason?
thank you.

df

It’s not a crash, haproxy is getting OOM killed (haproxy uses too much memory).

Most likely cause is maxconn and timeout misconfiguration. Share the output of haproxy -vv and the entire configuration please.

Hey Lukastribus,

[root@hp ~]# haproxy -vv
HA-Proxy version 1.8.6 2018/04/05
Copyright 2000-2018 Willy Tarreau willy@haproxy.org

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
OPTIONS =

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built without PCRE or PCRE2 support (using libc’s regex instead)
Built without compression support (neither USE_ZLIB nor USE_SLZ are set).
Compression algorithms supported : identity(“identity”)
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

[root@hp ~]#

Like I said, the entire configuration is needed.

Sorry, I’ve attached the configuration file.

https://pastebin.com/9RHPyXUN

global
	log 127.0.0.1   local0
	log 127.0.0.1   local1 notice
	#log loghost    local0 info
	maxconn 500000
	chroot /var/lib/haproxy
	user haproxy
	group haproxy
	daemon
	#debug
	#quiet
	stats socket /var/lib/haproxy/stats
	tune.maxrewrite 536870912 
    tune.bufsize 1073741824
	
	
#---------------------------------------------------------------------
# DEFAULTS KONFIG
#---------------------------------------------------------------------
 
defaults
    mode                    http
    log                     global
    option                  httplog 
    option                  dontlognull
    option                  http-server-close
    option  forwardfor      except 127.0.0.0/8
    option                  redispatch 
    retries                 6
 
    timeout http-request    500s
    timeout queue           5m
    timeout connect         500s
    timeout client          5000s
    timeout server          24h
    timeout http-keep-alive 500s
    timeout check           500s 
	errorfile 503 /etc/haproxy/errorfiles/503sorry.http

#---------------------------------------------------------------------
# HAPROXY DASHBOARD KONFIG
#---------------------------------------------------------------------
 
listen stats
	bind *:8080
        mode http	
        #option forwardfor
        option httpclose
        stats enable
        stats show-legends
        stats refresh 30s
        stats uri /stats
        stats auth xxxx:yyyy
 
#---------------------------------------------------------------------
# SSH & FTP BAGLANTILARI 
#---------------------------------------------------------------------
		
listen yyy
	bind *:49999 
		mode tcp
		option tcplog
		server ftp-xxx	192.168.1.211:22
		
listen zzz
	bind *:7001
		mode http
		option httplog 
		server web-xx	192.168.1.91:7001 
		
			
	...
	...
	...
	...
	
 
#---------------------------------------------------------------------
# POOL KONFIG
#---------------------------------------------------------------------
	
frontend MAIN 
	bind *:21  
	bind *:80
	
	...
	...
	...
	...
	
	mode http 
    
#---------------------------------------------------------------------
# ALAN ADLARI
#---------------------------------------------------------------------
	
    acl XXXXX_URL			hdr(host) -i 	www.xxxx.com xxxx.com
	
		
	...
	...
	...
	...
	
	
#---------------------------------------------------------------------
# ALAN ADIMANLARININ ARKAPLAN UCLARI
#---------------------------------------------------------------------
    
    use_backend xxxx 			if XXXXX_URL
	
	
	...
	...
	...
	...
	
	
#---------------------------------------------------------------------
# BACKEND SUNUCULAR
#---------------------------------------------------------------------
 
backend xxxx
    mode		http
	reqadd		X-Forwarded-Proto:\ https
	rspadd		Strict-Transport-Security:\ max-age=15768000;\ includeSubDomains #enable HSTS header for this backend
	balance		leastconn
    option		httpclose
    option		forwardfor
    cookie		JSESSIONID prefix
	server 		web-xxxx	192.168.1.91
 
 
 	
	...
	...
	...
	...

You configured tune.bufsize, which defaults to 16 KiB, to 1 GiB, thats 64000 times the default, and raising it is explicitly warned against. You also configured maxconn 500000.

Quickly calculating the memory usage for this configuration (not even accounting for overheads here):

RAM usage = tune.bufsize * 2 * maxconn

That would be at least 1 Petabyte of RAM. But you only have a 4 GB RAM VM here.

Please remove tune.maxrewrite and tune.bufsize, and reduce maxconn. For a 4 GB RAM VM, without bufsize or maxrewrite misconfiguration, you could configure somewhere around 100000 maxconn connections for the process. But not more and not with maxrewrite misconfiguration and also not with SSL.

timeout http-request    500s
timeout queue           5m
timeout connect         500s
timeout client          5000s
timeout server          24h
timeout http-keep-alive 500s
timeout check           500s 

This is also pretty bad. check, connect, http-keep-alive, queue and http-request should be a few seconds only. If you must, then configure client and server to a few minutes but certainly not 24 hours. A few requests and an attacker has completely congested all connections.

Thank you for your help. I increased the amount of ram to 8 GB.
I don’t know how useful this upgrade can be.

I changed it this way on your advice.
Is there any significant performance degradation with these values?

global
	log 127.0.0.1   local0
	log 127.0.0.1   local1 notice
	#log loghost    local0 info
	maxconn 150000
	chroot /var/lib/haproxy
	user haproxy
	group haproxy
	daemon
	#debug
	#quiet
	stats socket /var/lib/haproxy/stats
	#tune.maxrewrite 536870912 
        #tune.bufsize 1073741824

defaults
    mode                    http
    log                     global
    option                  httplog 
    option                  dontlognull
    option                  http-server-close
    option  forwardfor      except 127.0.0.0/8
    option                  redispatch 
    retries                 6
 
    timeout http-request    30s
    timeout queue           1m
    timeout connect         30s
    timeout client          30s
    timeout server          10m
    timeout http-keep-alive 100s
    timeout check           30s 
   timeout tunnel  	1h

Also, would it be better to upgrade from HAproxy 1.8.6 to 2.0?
Thank you.

This should be fine, yes.

No.

No. You can upgrade to a new major release if you need the features.

Thank you for help @lukastribus