HAProxy community

Haproxy1.9.1 getting crashed

Hello All,

We are not able to figure out why the haproxy is getting killed.

Jan 31 15:58:16 ip-172-16-21-136.ca-central-1.compute.internal haproxy[5135]: Starting haproxy: [ OK ]
Jan 31 15:58:16 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 00:14:25 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: haproxy.service: main process exited, code=killed, status=6/ABRT
Feb 01 00:14:25 ip-172-16-21-136.ca-central-1.compute.internal haproxy[28927]: Shutting down haproxy: [ OK ]
Feb 01 00:14:25 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Unit haproxy.service entered failed state.
Feb 01 00:14:25 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: haproxy.service failed.
Feb 01 01:05:32 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 01:05:32 ip-172-16-21-136.ca-central-1.compute.internal haproxy[31699]: Starting haproxy: [ OK ]
Feb 01 01:05:32 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Stopping SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal haproxy[12228]: Shutting down haproxy: [ OK ]
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal haproxy[12237]: Starting haproxy: [ OK ]
Feb 01 05:10:41 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 01 08:35:54 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: haproxy.service: main process exited, code=killed, status=6/ABRT
Feb 01 08:35:54 ip-172-16-21-136.ca-central-1.compute.internal haproxy[22275]: Shutting down haproxy: [ OK ]
Feb 01 08:35:54 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Unit haproxy.service entered failed state.
Feb 01 08:35:54 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: haproxy.service failed.
Feb 01 08:41:34 ip-172-16-21-136.ca-central-1.compute.internal systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…

I am also posting my sample config snipper below

global
nbproc 1
nbthread 4
maxconn 2000
spread-checks 50
daemon
nbproc 4
cpu-map 1 0
log 127.0.0.1 local2
lua-load /etc/haproxy/lua/400.lua
lua-load /etc/haproxy/lua/cors.lua
lua-load /etc/haproxy/lua/callback_handler.lua
#tune.chksize 1028
#tune.maxrewrite 16384
#tune.bufsize 32768
ssl-server-verify none
############

Defaults

############
defaults
log global
option log-separate-errors
maxconn 2000
timeout connect 60s
timeout queue 2s
timeout check 60s
timeout client 60s
timeout server 60s
timeout http-keep-alive 1s
timeout http-request 60s
mode http
retries 3
balance static-rr
timeout tunnel 60s
option http-server-close
option forwardfor
option http-pretend-keepalive

Haproxy 1.9.1 is a 1 year old .1 release with 513 known bugs that have subsequently been fixed.

Please use stable versions with recent bugfixes releases instead.

Hello @lukastribus - I did move to HA-Proxy version 2.0.10 2019/11/25.
But still got the same problem.

Below is the core dump I am getting

(gdb) bt
#0  0x00007fbdb6b35760 in raise () from /lib64/libc.so.6
#1  0x00007fbdb6b36d41 in abort () from /lib64/libc.so.6
#2  0x00007fbdb6b78007 in __libc_message () from /lib64/libc.so.6
#3  0x00007fbdb6b7e30a in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fbdb6b7fe4c in _int_free () from /lib64/libc.so.6
#5  0x0000000000422d01 in chunk_destroy (chk=0x16b2308) at include/common/chunk.h:230
#6  ssl_sock_sctl_free_func (parent=<optimized out>, ptr=0x16b2308, ad=<optimized out>, idx=<optimized out>, argl=<optimized out>, argp=<optimized out>) at src/ssl_sock.c:10023
#7  0x00007fbdb7908066 in int_free_ex_data () from /lib64/libcrypto.so.10
#8  0x00007fbdb7d39916 in SSL_CTX_free () from /lib64/libssl.so.10
#9  0x00007fbdb3ddf28f in meth_destroy () from /usr/local/lib/lua/5.3/ssl.so
#10 0x0000000000568728 in luaD_precall ()
#11 0x00000000005689b3 in luaD_call ()
#12 0x0000000000568a01 in luaD_callnoyield ()
#13 0x0000000000567f3c in luaD_rawrunprotected ()
#14 0x0000000000568cab in luaD_pcall ()
#15 0x0000000000569d62 in GCTM ()
#16 0x0000000000569e0d in runafewfinalizers ()
#17 0x0000000000573c46 in luaV_execute ()
#18 0x00000000005689bf in luaD_call ()
#19 0x00000000005665bf in lua_pcallk ()
#20 0x000000000057a170 in luaB_pcall ()
#21 0x0000000000568728 in luaD_precall ()
#22 0x0000000000572aad in luaV_execute ()
#23 0x0000000000567f3c in luaD_rawrunprotected ()
#24 0x0000000000568acc in lua_resume ()
#25 0x0000000000436ffb in hlua_ctx_resume (lua=lua@entry=0x144eb60, yield_allowed=yield_allowed@entry=1) at src/hlua.c:1113
#26 0x000000000043cdf5 in hlua_applet_htx_fct (ctx=<optimized out>) at src/hlua.c:7322
#27 hlua_applet_http_fct (ctx=0x1476470) at src/hlua.c:7442
#28 0x00000000005445a7 in task_run_applet (t=0x174dd80, context=0x1476470, state=<optimized out>) at src/applet.c:86
#29 0x00000000005412d6 in process_runnable_tasks () at src/task.c:414
#30 0x00000000004b6477 in run_poll_loop () at src/haproxy.c:2516
#31 run_thread_poll_loop (data=data@entry=0x0) at src/haproxy.c:2637
#32 0x000000000041f6f1 in main (argc=<optimized out>, argv=0x7ffd6bc4f0e8) at src/haproxy.c:3314
(gdb) quit
[root@ip-172-16-50-136 /]# gdb haproxy  core.8361 ^C
[root@ip-172-16-50-136 /]# haproxy --help
HA-Proxy version 2.0.10 2019/11/25 - https://haproxy.org/
Usage : haproxy [-f <cfgfile|cfgdir>]* [ -vdVD ] [ -n <maxconn> ] [ -N <maxpconn> ]
        [ -p <pidfile> ] [ -m <max megs> ] [ -C <dir> ] [-- <cfgfile>*]
        -v displays version ; -vv shows known build options.
        -d enters debug mode ; -db only disables background mode.

and the update config file I am using is

   ##########
    # Global #
    ##########
    global
     nbproc 1
     nbthread 1
     maxconn 2000
     maxsslconn 2000
     spread-checks 5
     daemon
     cpu-map 1/all 0-3
     log 127.0.0.1 local2
     set-dumpable
     lua-load /etc/haproxy/lua/400.lua
     lua-load /etc/haproxy/lua/cors.lua
     lua-load /etc/haproxy/lua/callback_handler.lua
     #tune.chksize 1028
     #tune.maxrewrite  16384
     #tune.bufsize     32768
     ssl-server-verify none

     listen stats
        bind :1936
        mode http
        maxconn 10000
        timeout queue 100s

        stats enable
        stats hide-version
        stats refresh 30s
        stats show-node
        stats auth admin:password
        stats uri /haproxy?stats

    # Defaults #
    ############
    defaults
     log global
     option httplog
     maxconn 2000
     timeout connect 10s
     timeout queue 2s
     timeout check 60s
     timeout client 60s
     timeout server 60s
     timeout http-keep-alive 1s
     timeout http-request 10s
     mode http
     retries 3
     balance static-rr
     timeout tunnel 60s
     option http-server-close
     option forwardfor
     option http-pretend-keepalive
    #####################
    # Frontend: GATEWAY #
    #####################
    frontend gateway80
     bind *:80
     mode http
     redirect scheme https if !{ ssl_fc }

    frontend gateway
     bind *:8080
     mode http
     option httplog
     option logasap
     option httpclose
     log global
     acl acl_scg_gateway_api_path hdr(host) -i IOTHost.com
     acl acl_scg_frontend_api_path hdr(host) -i clientHost.com
     acl acl_scg_api_path path_beg /api /webhooks
     acl acl_scg_frontapi_path path_beg /
     acl acl_scg_iot_path path_beg /apidata
     acl acl_scg_callback_path path_beg /callback/initiate
     http-request use-service lua.cors-response if METH_OPTIONS { req.hdr(origin) -m found }
     use_backend callback_redirect if acl_scg_gateway_api_path acl_scg_callback_path
     use_backend callback_frontredirect if acl_scg_frontend_api_path acl_scg_frontapi_path
     use_backend iot-%[req.hdr(client_id)] if acl_scg_gateway_api_path acl_scg_iot_path
     use_backend %[req.hdr(client_id)] if !acl_scg_iot_path !acl_scg_callback_path acl_scg_gateway_api_path acl_scg_api_path
     default_backend bad-request-backend

    #############################
    # Backend: Middleware/Front #
    #############################

    backend callback_frontredirect
     mode http
     balance roundrobin
     http-request set-header X-Forwarded-Host %[req.hdr(Host)]
     http-request set-header X-Client-IP %[src]
     http-request set-header Host  backend.com
     server server1 server1IP:4200
     server server2 server2IP:4200


    backend callback_redirect
     mode http
     http-request use-service lua.callback_redirect

    backend 1001
     option forwardfor
     option abortonclose
     log global
     balance roundrobin
     http-request set-header X-Forwarded-Host %[req.hdr(Host)]
     http-request set-header X-Client-IP %[src]
     http-request set-header Host backend.com
     server server1 backend.com:443 check ssl


    backend iot-1001
     mode http
     balance roundrobin
     http-request set-header X-Forwarded-Host %[req.hdr(Host)]
     http-request set-header X-Client-IP %[src]
     http-request set-header Host `.com
     server server1 IOTHost.com:443 check ssl

    backend bad-request-backend
     mode http
     http-request use-service lua.unauthorized-response

Can you file a bug at Github please:

Make sure you fill out the template completely, including haproxy -vv and please post the full backtrace (bt full) from gdb. Also you probably want to explain what those LUA scripts do, as the crash seems to be related to that.