Need help reloading a whitelist.lst

Help!
1

Hi,

I’m having trouble with the whitelist acl rule, snippet below:

acl whitelist src -f /usr/local/etc/haproxy/ip_whitelist/whitelist.lst

I’m running haproxy within a docker container and using inotifywait to detect file changes and then using the below snippet to reload haproxy.

sudo /usr/local/sbin/haproxy -d -f /usr/local/etc/haproxy/haproxy.cfg -sf $HAPROXY_PID > /var/log/haproxy.log &; HAPROXY_PID=$!

While I can see from the logs that inotifywait is being hit and it’s running the snippet, it doesn’t seem to be reloading the whitelist correctly.

If I have my laptops IP within the list and send a request to the container I pass the whitelist fine, but then remove all IPs from the list without restarting the docker container I still pass the whitelist despite removing my IP from the list. If I then stop and start the container it corrects the mistake and fails the acl rule as it should. Presumably something is wrong with how I’m reloading haproxy, I’d appreciate any help I can get.

2

Here is my full entrypoint if it’s helpful:

#!/bin/bash

# mostly stolen from https://github.com/aamkye/seamless-reload-haproxy

export PS4='+ $0:$LINENO '

# Sleep just gives the other containers a moment to start up
sleep 1.5

HAPROXY_VALID_MSG=${HAPROXY_VALID_MSG:-"Looks good - reload completed. Haproxy output at /var/log/haproxy.log"}
HAPROXY_INVALID_MSG=${HAPROXY_INVALID_MSG:-"Invalid HAProxy configuration - check the logs, leaving old config.."}

ENABLE_SYN_DROP="sudo iptables -I INPUT -p tcp -m multiport --dport ${HAPROXY_PORTS} --syn -j DROP"
DISABLE_SYN_DROP="sudo iptables -D INPUT -p tcp -m multiport --dport ${HAPROXY_PORTS} --syn -j DROP"

HAPROXY_CONFIG_DIRECTORY=${HAPROXY_CONFIG_DIRECTORY:-"/usr/local/etc/haproxy"}
HAPROXY_CONFIG=${HAPROXY_CONFIG:-"/usr/local/etc/haproxy/haproxy.cfg"}
HAPROXY_PORTS=${HAPROXY_PORTS:-"80,443,8080"}

# May want to consider removing -d debug flag for production
HAPROXY_CMD="sudo /usr/local/sbin/haproxy -d -f ${HAPROXY_CONFIG}"
HAPROXY_CHECK_CONFIG_CMD="/usr/local/sbin/haproxy -c -- ${HAPROXY_CONFIG}"

function safe_run {
  if [[ ! -f "${HAPROXY_CONFIG}" ]]; then
    echo "No config.." >&2
    exit 1
  fi

  ${HAPROXY_CHECK_CONFIG_CMD} && RC=0 || RC=1
  if [[ ${RC} -ne 0 ]]; then
    echo $HAPROXY_INVALID_MSG
  else
    # ${ENABLE_SYN_DROP}
    sleep 0.2
    if [[ -n "$HAPROXY_PID" ]]; then
      echo "pid was found as $HAPROXY_PID"
      ${HAPROXY_CMD} -sf $HAPROXY_PID > /var/log/haproxy.log &
      HAPROXY_PID=$!
    else
      echo "pid was not found"
      ${HAPROXY_CMD} > /var/log/haproxy.log &
      HAPROXY_PID=$!
    fi
    # ${DISABLE_SYN_DROP}
    echo $HAPROXY_VALID_MSG
  fi
}

safe_run
echo 'Made it to the inotifywait?'
sudo inotifywait --monitor -r -e create,delete,modify,attrib,close_write,move /etc/hosts /etc/ssl/private "${HAPROXY_CONFIG_DIRECTORY}" | \
while read CHANGED;
do
  echo "inotifywait has been hit: $CHANGED"
  safe_run
done
3

At a glance, the script looks solid. I would probably try kill -1 $HAPROXY_PID as long as you’re running a newer version of HAProxy (if PID is unknown, I might would try killall -s SIGHUP haproxy). That’s the recommended way from HAProxy’s docker hub page.

You’re showing the ACL, but you’ve not shown how the ACL is in use. Do you have a line below it that is something like one of these?

    use backend not_whitelisted unless whitelisted
    http-request deny deny_status 403 unless whitelisted

If not, you’re only setting up the ACL but not actually using it.

4

Thanks I’ll give the kill command a try. And yes I’m using the ACL, I just didn’t want to drown my post in text, here:

    acl host_sub hdr(host) -m reg -i ^host\..+$
    acl guest_sub hdr(host) -m reg -i ^guest\..+$
    acl cms_sub hdr(host) -m reg -i ^cms\..+$
    acl display_sub hdr(host) -m reg -i ^display\..+$
    acl backend_sub hdr(host) -m reg -i ^backend\..+$
    acl whitelist_sub hdr(host) -m reg -i ^whitelist\..+$
    acl whitelist src -f /usr/local/etc/haproxy/ip_whitelist/whitelist.lst
    use_backend host_node_8081 if host_sub whitelist
    use_backend guest_node_8081 if guest_sub whitelist
    use_backend cms_node_8081 if cms_sub
    use_backend display_node_8081 if display_sub whitelist
    use_backend backend_node_8081 if backend_sub
    # Test if you pass whitelist using whitelist.localhost
    use_backend whitelist_pass if whitelist_sub whitelist
    # If not on whitelist but trying to access whitelist required endpoint, return whitelist fail response
    use_backend whitelist_fail if whitelist_sub !whitelist || host_sub !whitelist || guest_sub !whitelist || display_sub !whitelist
5

Seems I’ve got something weird going on with my permissions as neither the haproxy -sf flag or the kill command is actually killing the original haproxy instance. If I use htop I can see it’s just creating more and more instances whenever I edit my config. No idea how I’ve managed this screw up but it’s a direction to work in at least.

6

I tried swapping over to using GitHub - million12/docker-haproxy: Fully customisable HAProxy load balancer with HTTP/2 and ALPN support. Built on CentOS-7. instead, exact same issue. If I run docker top host-proxy I can see that every reload of the config is just starting up a new haproxy instance without killing the original.

So I then noticed that as an alternative to the -sf flag there’s an -st flag instead to terminate the previous instance instead of just telling it to finish. That also didn’t work, exactly the same as the -sf flag it just shows in top as a bunch of never ending processes.

In the end I figured out that removing sudo from my HAPROXY_CMD fixed the problem completely. No clue as to why but it is what it is. Problem solved.

1 Like
7

In my experience (limited to Docker), it’s unusual for sudo to be installed within a container. Typically the container runs with Docker’s root permissions unless otherwise specified either when the container was built or when it was launched. This means you won’t normally find sudo in a container because it’s either not needed or excluded by design. Good find!

8

Yep, strange that the popular million12/docker-haproxy project has this bug, they aren’t using sudo in their command so I don’t know what’s going on. From my usage it seems that the reloading of config is completely broken for that project. Maybe I was doing something else wrong there.