HAProxy gets “Permission denied” error on server restart

I’m having issues with HAProxy not restarting when my Red Hat server restarts. I have a service script thats designed to call the haproxy start script on server reboot. Every time I try to start it with the server, it returns “Could not open configuration file /apps/haproxy/config/haproxy.cfg : Permission denied” in the nohup file. The service script is:

[Unit]
Description=HAProxy Load Balancer
After=syslog.target network.target

[Service]
User=user1
ExecStart=/apps/haproxy/bin/start.sh
ExecStop=/apps/haproxy/bin/stop.sh
Group=games
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

The start script it calls is:

#!/usr/bin/env sh

nohup haproxy -D -db -f /apps/haproxy/config/haproxy.cfg  -p /apps/haproxy/config/haproxy.pid -sf $(cat /apps/haproxy/config/haproxy.pid) >> /apps/haproxy/bin/nohup.out 2>&1 &

So far, I’ve tried giving the configuration file 777 permissions, but this has not worked. I’m using versio 1.5.18 of HAProxy, and my configuration is as follows:

global
  maxconn 4096
  quiet
  user root
  group root

#/installs version
defaults
  log     global
  mode    http
  retries 3
  timeout client 3600s
  timeout connect 3600s
  timeout server 3600s
  option tcplog
  balance  roundrobin
# Set up application listeners here.

listen admin
  bind 127.0.0.1:22005
  mode http
  stats enable
  stats show-node
  stats uri  /admin

listen  stats    :1939
  mode            http
  log             global

  maxconn 10

  timeout client      3600s
  timeout server      3600s
  timeout connect     3600s
  timeout queue   3600s

  stats enable
  stats hide-version
  stats show-node
  stats uri  /haproxy?stats

frontend http
  maxconn 2000
  bind 0.0.0.0:5050

  acl test-service path_reg /test*
  use_backend servers-test if test-service

backend servers-test
  server www.testserver.com 127.0.0.1:8000 maxconn 100

A few caveats to my issue:

  1. Upgrading HAProxy is not an option for me. I’m trying to get a newer version, but there is a process that needs to be followed for that, and it has currently stalled
  2. I can’t really provide any logs, as I’m also still trying to get that working rsyslog, so for the moment, I can only use the informaiton provided by nohup. As it is, though, I don’t think logs would provide any additional insight, as when the server reboots HAProxy is not even restarting - it just gives the “permission denied” error.

Exactly why is it that you need to use a homegrown script, as opposed to whatever ships with your OS? Can you explain what this configuration is supposed to do, so we get the big picture here? I assume that there are a million issues with this configuration and strongly advise you to not go down this rabbit hole.

To the technical aspect: are you saying that the startup script works when the server is already running, and it only fails at reboot? Or does it not actually work in fact?

I can see that you are running your startup script with a specific user (user1), but in the haproxy configuration you expect to use root?

To delay the startup further I suggest you use network-online.target instead:

-After=syslog.target network.target
+After=network-online.target
+Wants=network-online.target

otherwise: strace the failing startup procedure and review the syscalls.

I’m not sure what you mean by what ships with my OS - I using RHEL7, and had to install haproxy to the server. There was nothing on it previously.

If you mean the actual start up configuration that RHEL7 uses, this is it - it uses systemd to restart services on server reboot, but that requires a homegrown script to tell it where the service start.sh script is. Thats what the first block of code is for, with ExecStart

The configuration file I gave simply serves as a front for a simple web service that runs on port 8000. I made this specifically to test the restart logic for haproxy. We use a separate configuration to front a series of web services, but that is not remotely ready, and I’m using this much more simplified configuration file to track down the restart issue.

Start.sh simply runs the haproxy command in daemon mode, with that configuration file, outputs a pid file for the haproxy instance it creates, and redirects any output to nohup. This allows users do not have to continuously type in the command present in the file. This script runs fine when I login as user1 and manually call it with ./start.sh.

haproxy.service is a systemd script that is supposed to run the aforementioned start.sh on server reboot, to ensure that there is no downtime. However, as I said, running it seems to bring about the issue where the user does not have permissions to read the config file. This does not make sense, as I set the running user as User1, who created and owns the config file and the start.sh script that haproxy.service calls.

The start.sh script works fine when run manually. Its only during server reboot that we get this error for “Permission denied”.

I originally had it setup to have user1 and games in the config file, as user1 created and owns the file, and games is what the user belongs to, but that generates a warning:

[WARNING] 173/172540 (11385) : [haproxy.main()] Failed to drop supplementary groups. Using ‘gid’/‘group’ without ‘uid’/‘user’ is generally useless.

which I could not find any information on at all. Changing the user and group to root got rid of the warning and did not affect manually running the file. I’ll change it back to the proper user and group, though.

I’ll try this next, and see what I get.

Also, I apologize if it takes me time to reply with results - I do not own the server this is on, and I do not have root permissions, so it takes a little more time to run and tests code changes.

You are wrong.

On a freshly installed RHEL7 you need 2 commands:

yum install haproxy

To install haproxy as well as:

systemctl enable haproxy

To enable haproxy at start automatically. That’s it.

You absolutely do NOT write your own startup scripts, that is just asking for trouble. Everything is there already, you just have to use it.