Hello All,
We are not able to figure out why the haproxy is getting killed.
Jan 31 15:58:16 ip-172-16-21-136.ca-central-1.compute.internal haproxy[5135]: Starting haproxy: [ OK ]
Jan 31 15:58:16 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 00:14:25 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: haproxy.service: main process exited, code=killed, status=6/ABRT
Feb 01 00:14:25 ip-172-16-21-136.ca-central-1.compute.internal haproxy[28927]: Shutting down haproxy: [ OK ]
Feb 01 00:14:25 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Unit haproxy.service entered failed state.
Feb 01 00:14:25 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: haproxy.service failed.
Feb 01 01:05:32 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 01:05:32 ip-172-16-21-136.ca-central-1.compute.internal haproxy[31699]: Starting haproxy: [ OK ]
Feb 01 01:05:32 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Stopping SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal haproxy[12228]: Shutting down haproxy: [ OK ]
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal haproxy[12237]: Starting haproxy: [ OK ]
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 08:35:54 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: haproxy.service: main process exited, code=killed, status=6/ABRT
Feb 01 08:35:54 ip-172-16-21-136.ca-central-1.compute.internal haproxy[22275]: Shutting down haproxy: [ OK ]
Feb 01 08:35:54 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Unit haproxy.service entered failed state.
Feb 01 08:35:54 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: haproxy.service failed.
Feb 01 08:41:34 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
I am also posting my sample config snipper below
global
nbproc 1
nbthread 4
maxconn 2000
spread-checks 50
daemon
nbproc 4
cpu-map 1 0
log 127.0.0.1 local2
lua-load /etc/haproxy/lua/400.lua
lua-load /etc/haproxy/lua/cors.lua
lua-load /etc/haproxy/lua/callback_handler.lua
#tune.chksize 1028
#tune.maxrewrite 16384
#tune.bufsize 32768
ssl-server-verify none
############
Defaults
############
defaults
log global
option log-separate-errors
maxconn 2000
timeout connect 60s
timeout queue 2s
timeout check 60s
timeout client 60s
timeout server 60s
timeout http-keep-alive 1s
timeout http-request 60s
mode http
retries 3
balance static-rr
timeout tunnel 60s
option http-server-close
option forwardfor
option http-pretend-keepalive
Haproxy 1.9.1 is a 1 year old .1 release with 513 known bugs that have subsequently been fixed.
Please use stable versions with recent bugfixes releases instead.
Hello @lukastribus - I did move to HA-Proxy version 2.0.10 2019/11/25.
But still got the same problem.
Below is the core dump I am getting
(gdb) bt
#0 0x00007fbdb6b35760 in raise () from /lib64/libc.so.6
#1 0x00007fbdb6b36d41 in abort () from /lib64/libc.so.6
#2 0x00007fbdb6b78007 in __libc_message () from /lib64/libc.so.6
#3 0x00007fbdb6b7e30a in malloc_printerr () from /lib64/libc.so.6
#4 0x00007fbdb6b7fe4c in _int_free () from /lib64/libc.so.6
#5 0x0000000000422d01 in chunk_destroy (chk=0x16b2308) at include/common/chunk.h:230
#6 ssl_sock_sctl_free_func (parent=<optimized out>, ptr=0x16b2308, ad=<optimized out>, idx=<optimized out>, argl=<optimized out>, argp=<optimized out>) at src/ssl_sock.c:10023
#7 0x00007fbdb7908066 in int_free_ex_data () from /lib64/libcrypto.so.10
#8 0x00007fbdb7d39916 in SSL_CTX_free () from /lib64/libssl.so.10
#9 0x00007fbdb3ddf28f in meth_destroy () from /usr/local/lib/lua/5.3/ssl.so
#10 0x0000000000568728 in luaD_precall ()
#11 0x00000000005689b3 in luaD_call ()
#12 0x0000000000568a01 in luaD_callnoyield ()
#13 0x0000000000567f3c in luaD_rawrunprotected ()
#14 0x0000000000568cab in luaD_pcall ()
#15 0x0000000000569d62 in GCTM ()
#16 0x0000000000569e0d in runafewfinalizers ()
#17 0x0000000000573c46 in luaV_execute ()
#18 0x00000000005689bf in luaD_call ()
#19 0x00000000005665bf in lua_pcallk ()
#20 0x000000000057a170 in luaB_pcall ()
#21 0x0000000000568728 in luaD_precall ()
#22 0x0000000000572aad in luaV_execute ()
#23 0x0000000000567f3c in luaD_rawrunprotected ()
#24 0x0000000000568acc in lua_resume ()
#25 0x0000000000436ffb in hlua_ctx_resume (lua=lua@entry=0x144eb60, yield_allowed=yield_allowed@entry=1) at src/hlua.c:1113
#26 0x000000000043cdf5 in hlua_applet_htx_fct (ctx=<optimized out>) at src/hlua.c:7322
#27 hlua_applet_http_fct (ctx=0x1476470) at src/hlua.c:7442
#28 0x00000000005445a7 in task_run_applet (t=0x174dd80, context=0x1476470, state=<optimized out>) at src/applet.c:86
#29 0x00000000005412d6 in process_runnable_tasks () at src/task.c:414
#30 0x00000000004b6477 in run_poll_loop () at src/haproxy.c:2516
#31 run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:2637
#32 0x000000000041f6f1 in main (argc=<optimized out>, argv=0x7ffd6bc4f0e8) at src/haproxy.c:3314
(gdb) quit
[root@ip-172-16-50-136 /]# gdb haproxy core.8361 ^C
[root@ip-172-16-50-136 /]# haproxy --help
HA-Proxy version 2.0.10 2019/11/25 - https://haproxy.org/
Usage : haproxy [-f <cfgfile|cfgdir>]* [ -vdVD ] [ -n <maxconn> ] [ -N <maxpconn> ]
[ -p <pidfile> ] [ -m <max megs> ] [ -C <dir> ] [-- <cfgfile>*]
-v displays version ; -vv shows known build options.
-d enters debug mode ; -db only disables background mode.
and the update config file I am using is
##########
# Global #
##########
global
nbproc 1
nbthread 1
maxconn 2000
maxsslconn 2000
spread-checks 5
daemon
cpu-map 1/all 0-3
log 127.0.0.1 local2
set-dumpable
lua-load /etc/haproxy/lua/400.lua
lua-load /etc/haproxy/lua/cors.lua
lua-load /etc/haproxy/lua/callback_handler.lua
#tune.chksize 1028
#tune.maxrewrite 16384
#tune.bufsize 32768
ssl-server-verify none
listen stats
bind :1936
mode http
maxconn 10000
timeout queue 100s
stats enable
stats hide-version
stats refresh 30s
stats show-node
stats auth admin:password
stats uri /haproxy?stats
# Defaults #
############
defaults
log global
option httplog
maxconn 2000
timeout connect 10s
timeout queue 2s
timeout check 60s
timeout client 60s
timeout server 60s
timeout http-keep-alive 1s
timeout http-request 10s
mode http
retries 3
balance static-rr
timeout tunnel 60s
option http-server-close
option forwardfor
option http-pretend-keepalive
#####################
# Frontend: GATEWAY #
#####################
frontend gateway80
bind *:80
mode http
redirect scheme https if !{ ssl_fc }
frontend gateway
bind *:8080
mode http
option httplog
option logasap
option httpclose
log global
acl acl_scg_gateway_api_path hdr(host) -i IOTHost.com
acl acl_scg_frontend_api_path hdr(host) -i clientHost.com
acl acl_scg_api_path path_beg /api /webhooks
acl acl_scg_frontapi_path path_beg /
acl acl_scg_iot_path path_beg /apidata
acl acl_scg_callback_path path_beg /callback/initiate
http-request use-service lua.cors-response if METH_OPTIONS { req.hdr(origin) -m found }
use_backend callback_redirect if acl_scg_gateway_api_path acl_scg_callback_path
use_backend callback_frontredirect if acl_scg_frontend_api_path acl_scg_frontapi_path
use_backend iot-%[req.hdr(client_id)] if acl_scg_gateway_api_path acl_scg_iot_path
use_backend %[req.hdr(client_id)] if !acl_scg_iot_path !acl_scg_callback_path acl_scg_gateway_api_path acl_scg_api_path
default_backend bad-request-backend
#############################
# Backend: Middleware/Front #
#############################
backend callback_frontredirect
mode http
balance roundrobin
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header X-Client-IP %[src]
http-request set-header Host backend.com
server server1 server1IP:4200
server server2 server2IP:4200
backend callback_redirect
mode http
http-request use-service lua.callback_redirect
backend 1001
option forwardfor
option abortonclose
log global
balance roundrobin
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header X-Client-IP %[src]
http-request set-header Host backend.com
server server1 backend.com:443 check ssl
backend iot-1001
mode http
balance roundrobin
http-request set-header X-Forwarded-Host %[req.hdr(Host)]
http-request set-header X-Client-IP %[src]
http-request set-header Host `.com
server server1 IOTHost.com:443 check ssl
backend bad-request-backend
mode http
http-request use-service lua.unauthorized-response
Can you file a bug at Github please:
Make sure you fill out the template completely, including haproxy -vv and please post the full backtrace (bt full) from gdb. Also you probably want to explain what those LUA scripts do, as the crash seems to be related to that.