I’m trying to make HAProxy 1.8.3 work on CentOS 7 and I want to enable seamless reloads. I’ve managed to make seamless reloads work if I run HAProxy in the shell, but it doesn’t want to work with systemd.
Here is the HAProxy version:
# haproxy -v
HA-Proxy version 1.8.3-205f675 2017/12/30
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>
Systemd unit file /usr/lib/systemd/system/haproxy.service:
[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target
[Service]
EnvironmentFile=/etc/sysconfig/haproxy
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"
ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q
ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $OPTIONS
ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
Type=notify
[Install]
WantedBy=multi-user.target
and the global section of the /etc/haproxy/haproxy.cfg looks like this:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats expose-fd listeners
When I start haproxy everything looks fine:
[root@hap18 ~]# systemctl start haproxy.service
[root@hap18 ~]# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2018-01-05 14:53:12 CET; 2s ago
Process: 4004 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
Process: 4003 ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
Process: 4038 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
Main PID: 4039 (haproxy)
CGroup: /system.slice/haproxy.service
├─4039 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
└─4041 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
Jan 05 14:53:12 hap18 systemd[1]: Starting HAProxy Load Balancer...
Jan 05 14:53:12 hap18 systemd[1]: Started HAProxy Load Balancer.
[root@hap18 ~]# ps auxf | grep haprox[y]
root 4039 0.0 0.2 76296 4344 ? Ss 14:53 0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy 4041 8.5 0.1 297808 2280 ? Ssl 14:53 0:00 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
then when I do a reload, this happens:
[root@hap18 ~]# systemctl reload haproxy.service
[root@hap18 ~]# systemctl status haproxy.service
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2018-01-05 14:53:12 CET; 46s ago
Process: 4058 ExecReload=/bin/kill -USR2 $MAINPID (code=exited, status=0/SUCCESS)
Process: 4057 ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
Process: 4038 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
Main PID: 4039 (haproxy)
CGroup: /system.slice/haproxy.service
├─4039 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4041 -x /var/lib/haproxy/stats
└─4060 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4041 -x /var/lib/haproxy/stats
Jan 05 14:53:12 hap18 systemd[1]: Starting HAProxy Load Balancer...
Jan 05 14:53:12 hap18 systemd[1]: Started HAProxy Load Balancer.
Jan 05 14:53:56 hap18 systemd[1]: Reloaded HAProxy Load Balancer.
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145312 (4039) : Reexecuting Master process
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145356 (4039) : Failed to connect to the old process socket '/var/lib/haproxy/stats'
Jan 05 14:53:56 hap18 haproxy[4039]: [ALERT] 004/145356 (4039) : Failed to get the sockets from the old process!
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145356 (4039) : Former worker 4041 exited with code 0
[root@hap18 ~]# ps auxf | grep haprox[y]
root 4039 0.0 0.2 76296 4352 ? Ss 14:53 0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4041 -x /var/lib/haproxy/stats
haproxy 4060 19.7 0.1 297808 2280 ? Ssl 14:53 0:00 \_ /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4041 -x /var/lib/haproxy/stats
as you can see HAProxy logs that it cannot connect to the socket:
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145312 (4039) : Reexecuting Master process
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145356 (4039) : Failed to connect to the old process socket '/var/lib/haproxy/stats'
Jan 05 14:53:56 hap18 haproxy[4039]: [ALERT] 004/145356 (4039) : Failed to get the sockets from the old process!
Jan 05 14:53:56 hap18 haproxy[4039]: [WARNING] 004/145356 (4039) : Former worker 4041 exited with code 0
but it does everything else correctly because we can see that it spawned a new child process with adding -sf 4041 -x /var/lib/haproxy/stats where 4041 is the PID of the old process.
I tried many things:
- removed
chroot - set user
user/groupto root - changed socket path
- changed socket permissions
but nothing helped.
The interesting thing is that if I run HAProxy from the shell, reload works (I just had to disable daemon mode in the config so that HAProxy logs everything into the console, and of course while working with systemd I tried with disabling daemon mode as well). Here are the steps to run it in the shell.
Run haproxy in tmux/screen:
[root@hap18 ~]# haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
then in a spearate session we list these processes:
[root@hap18 ~]# ps auxf | grep haprox[y]
root 4185 0.0 0.2 76296 4348 pts/1 S+ 15:06 0:00 \_ haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
haproxy 4186 0.0 0.1 76612 2024 ? Ss 15:06 0:00 \_ haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
Now we send that process the USR2 signal so that it starts the reload:
kill -USR2 $(pgrep -U 0 haproxy)
pgrep -U 0 haproxy finds the master HAProxy process.
Then if we go back to the tmux/screen session we can see this:
[root@hap18 ~]# haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
[WARNING] 004/150638 (4185) : Reexecuting Master process
[WARNING] 004/150741 (4185) : Former worker 4186 exited with code 0
If we list the haproxy processes again:
[root@hap18 ~]# ps auxf | grep haprox[y]
root 4185 0.0 0.2 76296 4348 pts/1 S+ 15:06 0:00 \_ haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4186 -x /var/lib/haproxy/stats
haproxy 4190 0.0 0.1 76612 2028 ? Ss 15:07 0:00 \_ haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 4186 -x /var/lib/haproxy/stats
We can see that the reload was done and there weren’t any errors about not being able to read the socket.
Any ideas what could be the reason behind this? It has to be something related to systemd, but I’m not sure what could it be.