Hello,
I am running haproxy 1.7.9-1~bpo9+1 on debian 9.1. And after running a while with production data haproxy stops working wiith segmentation fault:
haproxy[26291]: segfault at 5562af80e000 ip 00007f5985e48149 sp 00007ffe1d613488 error 4 in libc-2.24
Can you please help or have any ideas?
Thanks a lot!!!
Here is haproxy -vv:
HA-Proxy version 1.7.9-1~bpo9+1 2017/08/24
Copyright 2000-2017 Willy Tarreau willy@haproxy.org
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fdebug-prefix-map=/build/haproxy-1.7.9=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_NS=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with OpenSSL version : OpenSSL 1.1.0f 25 May 2017
Running on OpenSSL version : OpenSSL 1.1.0f 25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with network namespace support
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[COMP] compression
[TRACE] trace
[SPOE] spoe
Hey,
thanks for youre reply. I have done this like described there: installed haproxy-dbgsym and running gdb haproxy > set pagination 0 > run -f /ete/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
After the error exited haproxy the problem ist that there is “no stack.” with bt or bt full?
Can you help?
Kind regards!
Marcus
Provide the complete gdb output including your inputs please? Don’t know what’s could be wrong from your description.
Hello,
thanks for your reply. I have done what you said… but there ist nore core file dumped.
running gdb console it only says “[Inferior 1 (process 719) exited normally]” when the segfault error occours and haproxy restarts. bt and bt full results in “No stack.”
What I have done is:
- don’t use chroot directive in /etc/haproxy/haproxy.cfg
- sysctl -w kernel.core_pattern=/tmp/core.%e.%p.%h.%t
- systemctl edit haproxy.service and put the following:
[Service]
LimitCORE=infinity
- systemctl daemon-reload
- systemctl restart haproxy
Check with “cat /proc/$(pidof haproxy | head -1)/limits” both limits for
"core file size" is unlimited.
- apt-get install haproxy-dbgsym libc6-dbg gdb
- gdb /usr/sbin/haproxy /tmp/core.haproxy.something
- bt full
And the other way with gdb console:
gdb haproxy
(gdb) set pagination 0
(gdb) run -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
[Inferior 1 (process 719) exited normally]
(gdb) bt full
No stack.
Can you help about this?
kind regards,
marcus
Lets do it oldschool.
Triple check that no haproxy process is running.
Disable chroot in haproxy configuration and start it like this:
ulimit -c unlimited
haproxy -f /etc/haproxy/haproxy.cfg -Ds
when it drops a core file, point gdb to it:
gdb /usr/sbin/haproxy /tmp/path/to/haproxy/corefile
Okay, I did this… and it takes longer for getting the error and stop haproxy to work. But the error occurs after 30min an there was no core file droped I can use?! 
What’s the exact output when haproxy crashes and what do you see in dmesg?
Are you sure there is no new file in your current working directory or /tmp/core-… ?
With:
ulimit -c unlimited
haproxy -f /etc/haproxy/haproxy.cfg -Ds
the line only stops in foreground and haproxy closes while dmesg shows:
haproxy[20300]: segfault at 5600213d6ff0 ip 00007fbdc4b221aa sp 00007ffeff99aa68 error 4 in libc-2.24.so[7fbdc4a9e000+195000]
…and there is really no core file either in /etc/haproxy, nor in /tmp/ nor in my home folder…
I can’t tell why no coredump is created.
Can you tell more about your setup and share your configuration? Specifically are you using SSL, LUA, regular expressions, compression? Does haproxy crashes only under heavy load? How fast does it crash when you restart it?
Yes, most often it crashes under heavy load… the the service must restart every second or so… therefore I had to slow down the Servide restart with 1s because haproxy refuses to restart if tried to often. (Service: RestartSex=1)
But normally it crashes at least ones after 30minutes. I also testet to disable ssl, with no change.
here ist my config:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 4000
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# Default ciphers to use on SSL-enabled listening sockets.
# For more information, see ciphers(1SSL).
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3 no-tlsv10
tune.ssl.default-dh-param 4096
defaults
log global
mode http
option httplog
option dontlognull
option forwardfor
option http-server-close
option redispatch
option abortonclose
timeout http-request 60s
timeout connect 60s
timeout client 60s
timeout server 300s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
listen stats
bind :2000
mode http
stats enable
stats hide-version
stats realm Haproxy\ Statistics
stats uri /
stats auth xxx:xxx
stats admin if TRUE
frontend http_XXX_frontend
bind X.X.X.X:80
bind X.X.X.X:443 ssl crt /etc/haproxy/cert/xxx.pem
mode http
option httpclose
option forwardfor
option http-server-close
### Logging
capture request header Host len 40
capture request header X-Forwarded-For len 50
capture request header Accept-Language len 50
capture request header Referer len 200
capture request header User-Agent len 200
capture response header Content-Type len 30
capture response header Content-Encoding len 10
capture response header Cache-Control len 200
capture response header Last-Modified len 200
###
### SSL
acl secure dst_port eq 443
rsprep ^Set-Cookie:\ (.) Set-Cookie:\ \1;\ Secure if secure
###
### acl filter and protection of ddos attacks
#Slowloris protection
timeout http-request 5s
# whitelist bots | blacklist attackers & accept others wiht time/concurrence restrictions
tcp-request connection accept if { src -f /etc/haproxy/whitelist.lst }
tcp-request connection reject if { src -f /etc/haproxy/blacklist.lst }
# Dont allow more than 30 concurrent tcp connections OR 10 connections in 1s
tcp-request connection reject if { src_conn_rate(Abuse) ge 30 }
tcp-request connection reject if { src_conn_cur(Abuse) ge 30 }
tcp-request connection track-sc1 src table Abuse
tcp-request content reject if { src_get_gpc0(Abuse) gt 0 }
# blacklist bad bots and crawler with eg wget,curl… but allow all agents to whitelist
tcp-request inspect-delay 3s
acl whitelistips_agents src -f /etc/haproxy/whitelistips_agents.lst
acl badbots hdr_reg(User-Agent) -i -f /etc/haproxy/badbots.lst
http-request deny if badbots !whitelistips_agents
#deny sleep arguments in url
acl sleeper url_sub,url_dec,lower -m sub sleep(
http-request deny if sleeper
# Dont allow more than 10 requests in 1 second
acl abuse src_http_req_rate(Abuse) ge 100
acl flag_abuser src_inc_gpc0(Abuse) ge 0
acl scanner src_http_err_rate(Abuse) ge 10
# Returns a 403 to the abuser and flags for tcp-reject next time
http-request deny if abuse flag_abuser
http-request deny if scanner flag_abuser
###
### lets encrypt
acl lets_encrypt path_beg /.well-known/acme-challenge/
use_backend lets_encrypt if lets_encrypt
###
default_backend http_xxx_backend
backend http_xxx_backend
mode http
### SSL
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
###
balance roundrobin
cookie SERVERID insert indirect nocache
option httpchk HEAD /
server x1 X.X.X.X:80 cookie x1 weight 2 check inter 10000 port 80 maxconn 32 on-marked-down shutdown-sessions
server x2 X.X.X.X:80 cookie x2 weight 1 check inter 10000 port 80 maxconn 16 on-marked-down shutdown-sessions
server x3 X.X.X.X:80 cookie x3 weight 3 check inter 10000 port 80 maxconn 64 on-marked-down shutdown-sessions
server x4 X.X.X.X:80 cookie x4 weight 4 check inter 10000 port 80 maxconn 128 on-marked-down shutdown-sessions
server x0 X.X.X.X:80 cookie x0 backup check inter 10000 port 80 maxconn 64 on-marked-down shutdown-sessions
backend Abuse
stick-table type ip size 1m expire 30s store conn_rate(3s),conn_cur,gpc0,http_req_rate(10s),http_err_rate(20s)
backend lets_encrypt
mode http
server local localhost:60001
For anyone reading this the analysis will continue on the mailing list:
https://www.mail-archive.com/haproxy@formilux.org/msg27367.html
No need for this discourse thread here.