V1.6.10 - Soft reload not working under Centos 7.2

Arianrhod · February 7, 2017, 9:24am

Hi - I have two haproxy servers with identical haproxy.cfg files, one running under Ubuntu 16.04, the other Centos 7.2. I need the option to maintain existing connections while loading a new config file, so choose to use the soft reload option. On the Ubuntu server, I can run /etc/init.d/haproxy reload to ‘soft restart’ and it works perfectly, but on the Centos server, the same command doesn’t work.

Haproxy version is this on both:

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
OPTIONS =

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built without compression support (neither USE_ZLIB nor USE_SLZ are set)
Compression algorithms supported : identity(“identity”)
Built without OpenSSL support (USE_OPENSSL not set)
Built without PCRE support (using libc’s regex instead)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Error on trying a reload is this:

Feb 7 07:26:31 apl00086i3 haproxy[8092]: Proxy http_frontend started.
Feb 7 07:26:31 apl00086i3 haproxy[8092]: Proxy http_frontend started.
Feb 7 07:26:31 apl00086i3 haproxy[8092]: Proxy sspe started.
Feb 7 07:26:31 apl00086i3 haproxy[8092]: Proxy sspe started.
Feb 7 07:26:31 apl00086i3 haproxy[8092]: Proxy stats started.
Feb 7 07:26:31 apl00086i3 haproxy[8092]: Proxy stats started.
Feb 7 07:26:31 apl00086i3 systemd: Reloaded SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 7 07:26:31 apl00086i3 systemd: haproxy.service: main process exited, code=killed, status=9/KILL
Feb 7 07:26:31 apl00086i3 haproxy: Shutting down haproxy: [FAILED]
Feb 7 07:26:31 apl00086i3 systemd: Unit haproxy.service entered failed state.
Feb 7 07:26:31 apl00086i3 systemd: haproxy.service failed.

… but if I issue /etc/init.d/haproxy start, it starts correctly, even though the previous command had a ‘FAILED’ state for shutdown:

Feb 7 07:26:41 apl00086i3 systemd: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 7 07:26:41 apl00086i3 haproxy[8122]: Proxy http_frontend started.
Feb 7 07:26:41 apl00086i3 haproxy[8122]: Proxy http_frontend started.
Feb 7 07:26:41 apl00086i3 haproxy[8122]: Proxy sspe started.
Feb 7 07:26:41 apl00086i3 haproxy[8122]: Proxy sspe started.
Feb 7 07:26:41 apl00086i3 haproxy[8122]: Proxy stats started.
Feb 7 07:26:41 apl00086i3 haproxy[8122]: Proxy stats started.
Feb 7 07:26:41 apl00086i3 haproxy: Starting haproxy: [ OK ]
Feb 7 07:26:41 apl00086i3 systemd: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…

Any clues, please, or suggestions how this can be debugged further, since those log entries aren’t particularly useful?

thanks …

AaronWest · February 7, 2017, 10:35am

Okay I’m guessing here a little…

However you start it correctly with /etc/init.d/haproxy which would appear to be an old style init script, while the failed reload appears to be using “haproxy.service” suggesting a systemd style script.

Can I check do you have two scripts or is it just me misunderstanding? Normally the systemd script would live here if activated : /etc/systemd/system/multi-user.target.wants/haproxy.service

Also for CentOS 7 and systemd try checking the output of “journalctl -xn” as it may offer more info.

Arianrhod · February 7, 2017, 11:07am

Thanks for your input Aaron. There is no script at /etc/systemd/system/multi-user.target.wants/haproxy.service , only the /etc/init.d/haproxy one.

Reading through the journal log, I think the haproxy.service error was a red herrring, actually … when /etc/init.d/haproxy reload didn’t work, I tried the alternate systemctl reload haproxy command (given that it’s a systemd server), and that’s what threw the ‘Unit haproxy.service entered failed state’ error.

The order of commands I actually tried were:

/etc/init.d/haproxy reload
… this came back with an error telling me to do a systemctl daemon-reload, which I did:
systemctl daemon-reload

journalctl -u haproxy.service shows this error resulting from these two commands:

Feb 07 07:22:21 apl00086i3 systemd[1]: PID 7966 read from file /var/run/haproxy.pid does not exist or is a zombie.
Feb 07 07:22:21 apl00086i3 systemd[1]: Reloaded SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 07 07:22:21 apl00086i3 haproxy[7968]: Shutting down haproxy: [FAILED]

I then issued:
/etc/init.d/haproxy start

journalctl shows this worked:

Feb 07 07:23:42 apl00086i3 systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 07 07:23:42 apl00086i3 haproxy[8012]: Starting haproxy: [ OK ]
Feb 07 07:23:42 apl00086i3 systemd[1]: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…

I then tried again with a systemctl reload command, which again failed:
systemctl reload haproxy

Feb 07 07:25:10 apl00086i3 systemd[1]: PID 8037 read from file /var/run/haproxy.pid does not exist or is a zombie.
Feb 07 07:25:10 apl00086i3 systemd[1]: Reloaded SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 07 07:25:10 apl00086i3 haproxy[8039]: Shutting down haproxy: [FAILED]

… and then restarted:
/etc/init.d/haproxy start

Feb 07 07:25:26 apl00086i3 systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 07 07:25:26 apl00086i3 haproxy[8063]: Starting haproxy: [ OK ]
Feb 07 07:25:26 apl00086i3 systemd[1]: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…

I then tried again, back to the original script:
/etc/init.d/haproxy reload

Feb 07 07:26:31 apl00086i3 haproxy[8092]: Proxy http_frontend started.
Feb 07 07:26:31 apl00086i3 systemd[1]: Reloaded SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 07 07:26:31 apl00086i3 systemd[1]: haproxy.service: main process exited, code=killed, status=9/KILL
Feb 07 07:26:31 apl00086i3 haproxy[8095]: Shutting down haproxy: [FAILED]
Feb 07 07:26:31 apl00086i3 systemd[1]: Unit haproxy.service entered failed state.
Feb 07 07:26:31 apl00086i3 systemd[1]: haproxy.service failed.

… note that it still refers to ‘haproxy.service’ despite using an old-style script … ??

Finally clean start again:
/etc/init.d/haproxy start

Feb 07 07:26:41 apl00086i3 systemd[1]: Starting SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…
Feb 07 07:26:41 apl00086i3 haproxy[8118]: Starting haproxy: [ OK ]
Feb 07 07:26:41 apl00086i3 systemd[1]: Started SYSV: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited for high availability environments…

I may have muddied the waters a little by trying both the systemd command and the old-style script, but the fact is that reload under neither command works - and we need it to; this is a production service that needs to be able to take one load-balanced server out of the loop while it’s being upgraded without impact to the service. Restarting each time is no good, as that kills off connections; we need the reload command to work.

For info, this is the /etc/init.d/haproxy script that was installed with the haproxy installation:

#!/bin/sh

chkconfig: - 85 15

description: HA-Proxy is a TCP/HTTP reverse proxy which is particularly suited \

for high availability environments.

processname: haproxy

config: /etc/haproxy/haproxy.cfg

pidfile: /var/run/haproxy.pid

Script Author: Simon Matter simon.matter@invoca.ch

Version: 2004060600

Source function library.

if [ -f /etc/init.d/functions ]; then
. /etc/init.d/functions
elif [ -f /etc/rc.d/init.d/functions ] ; then
. /etc/rc.d/init.d/functions
else
exit 0
fi

Source networking configuration.

. /etc/sysconfig/network

Check that networking is up.

[ “${NETWORKING}” = “no” ] && exit 0

This is our service name

BASENAME=basename $0
if [ -L $0 ]; then
BASENAME=find $0 -name $BASENAME -printf %l
BASENAME=basename $BASENAME
fi

BIN=/usr/sbin/$BASENAME

CFG=/etc/$BASENAME/$BASENAME.cfg
[ -f $CFG ] || exit 1

PIDFILE=/var/run/$BASENAME.pid
LOCKFILE=/var/lock/subsys/$BASENAME

RETVAL=0

start() {
quiet_check
if [ $? -ne 0 ]; then
echo "Errors found in configuration file, check it with ‘$BASENAME check’."
return 1
fi

echo -n "Starting $BASENAME: "
daemon $BIN -D -f $CFG -p PIDFILE RETVAL=?
echo
[ $RETVAL -eq 0 ] && touch $LOCKFILE
return $RETVAL
}

stop() {
echo -n "Shutting down $BASENAME: "
killproc BASENAME -USR1 RETVAL=?
echo
[ $RETVAL -eq 0 ] && rm -f $LOCKFILE
[ $RETVAL -eq 0 ] && rm -f $PIDFILE
return $RETVAL
}

restart() {
quiet_check
if [ $? -ne 0 ]; then
echo "Errors found in configuration file, check it with ‘$BASENAME check’."
return 1
fi
stop
start
}

reload() {
if ! [ -s $PIDFILE ]; then
return 0
fi

quiet_check
if [ $? -ne 0 ]; then
echo "Errors found in configuration file, check it with ‘$BASENAME check’."
return 1
fi
$BIN -D -f $CFG -p PIDFILE -sf (cat $PIDFILE)
}

check() {
$BIN -c -q -V -f $CFG
}

quiet_check() {
$BIN -c -q -f $CFG
}

rhstatus() {
status $BASENAME
}

condrestart() {
[ -e $LOCKFILE ] && restart || :
}

See how we were called.

case "1" in start) start ;; stop) stop ;; restart) restart ;; reload) reload ;; condrestart) condrestart ;; status) rhstatus ;; check) check ;; *) echo "Usage: $BASENAME {start|stop|restart|reload|condrestart|status|check}"
exit 1
esac

exit $?

Arianrhod · February 10, 2017, 9:30am

Is this something that I should raise on the mailing list? We need this functionality to work, and it currently isn’t.

thanks

AaronWest · March 20, 2017, 7:20pm

I recently had to rebuilt my CentOS 7 HAproxy box, in order to use the systemd wrapper you’ll need to copy the binary which gets compiled with HAproxy to :

cp haproxy-systemd-wrapper /usr/local/sbin/haproxy-systemd-wrapper

Then paste the following systemd script into the following location :

/usr/lib/systemd/system/haproxy.service

[Unit]
Description=HAProxy Load Balancer
After=network.target

[Service]
ExecStartPre=/usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c -q
ExecStart=/usr/local/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always

[Install]
WantedBy=multi-user.target

Then issue a daemon reoload : systemctl daemon-reload

Followed by enabling and starting the service :

systemctl enable haproxy
systemctl start haproxy

It worked for me on a fresh install although there is probably an easier way…

Topic		Replies	Views
Unable to use systemctl reload haproxy (help enabling systemd?) Help!	6	4619	November 7, 2020
Safe reloads not working Help!	0	317	February 1, 2022
Seamless Reloads don't work with systemd Help!	19	17383	September 19, 2018
Systemctl reload not working - fails to start/stop Help!	3	1636	November 9, 2019
Haproxy 1.8 vs 2.7.3 Help!	11	835	February 28, 2023