Haproxy kills external-check process before script ends

Help!
1

I have a docker container running haproxy:2.8-lts, which does a check to the backend, but my script gets killed before it can finish resulting in a failed check and accumulated alot of processes in the meantime. Looking at the logs i see the following:

+ not_used_1=NOT_USED
+ not_used_2=NOT_USED
+ host=18.138.80.239
+ port=443
++ /usr/local/bin/te-ping 18.138.80.239 -p 443 -c 50
++ /usr/bin/awk '/packet loss/ { print strtonum($6) }'
[WARNING]  (101890) : kill 103391
[WARNING]  (1) : Process 103392 exited with code 141 (Broken pipe)

taking a close look at the processes being spawned i see:

root           1  0.3  0.1  90656 10240 ?        Ss   08:47   0:00 haproxy -W -db -f /etc/haproxy/haproxy.cfg -S /tmp/socket
root           8  0.2  0.8 260208 72404 ?        S    08:47   0:00 haproxy -W -db -f /etc/haproxy/haproxy.cfg -S /tmp/socket
root          24  0.0  0.0   4152  3472 pts/0    Ss   08:47   0:00 /bin/bash
root         519  0.0  0.0   3888  2876 ?        S    08:48   0:00 /bin/bash /etc/haproxy/healthcheck NOT_USED NOT_USED 18.138.80.239 443
root         520  0.0  0.0   3888  1500 ?        S    08:48   0:00 /bin/bash /etc/haproxy/healthcheck NOT_USED NOT_USED 18.138.80.239 443
root         521  0.0  0.1 237636 10320 ?        Sl   08:48   0:00 /usr/local/bin/te-ping 18.138.80.239 -p 443 -c 50
root         522  0.0  0.0   8216  2604 ?        S    08:48   0:00 /usr/bin/awk /packet loss/ { print strtonum($6) }

my haproxy.cfg looks like:

global
    log stdout format raw local0 debug
    stats socket /tmp/socket mode 660 level admin expose-fd listeners
    stats timeout 2m
    insecure-fork-wanted
    insecure-setuid-wanted
    external-check
    nbthread 1

defaults
    log  global
    mode    tcp
    option  tcplog
    maxconn  8000
    retries  5
    timeout  queue 1m
    timeout  connect 10s
    timeout  client 1m
    timeout  server 1m
    timeout  check 10m

frontend....

backend data_us
    option log-health-checks
    option external-check
    external-check command /etc/haproxy/healthcheck
    server data_us proxy1-ap-southeast-1.aws.ca.thousandeyes.com:443 check inter 10m rise 5 fall 5 resolvers mynameservers resolve-prefer ipv4

I thought increasing timeout would help, but so far it hasn’t. I then thought it could be a process problem, where haproxy spawns multiple processes and kills them to save memory, so i decrease nbthread, but that didn’t work. My script is the following:

#!/bin/bash

set -x 

not_used_1=$1
not_used_2=$2
host=$3
port=$4

packet_loss=$(/usr/local/bin/te-ping $host -p $port -c 50 | /usr/bin/awk '/packet loss/ { print strtonum($6) }')
if [ "${packet_loss}" -lt 5 ]; then
  echo "${packet_loss}"
  exit 0;
else
  exit 1;
fi

Why does my script get killed before completing ? Why does haproxy start two main process? and why does it spawn multiple processes to do the external check even though i have a timeout specified of 10 min?

2

It doesn’t look like it gets killed, it looks like it crashes with a broken pipe:

Process 103392 exited with code 141 (Broken pipe)
3

Hi Lukas i found the problem, i was setting the timeout on the worng backend, and haproxy default is 2 sec, which means every two sec it would kill the process and spwan a new one. Once i noticed this i set the timeout accordingly and ass is fixed. long story short, config error