Upgrade from 1.5 to 2.1 - Active: inactive (dead)

[harporb]

Running Centos 7

dev reshaproxy02.dev:root[/opt/haproxy-2.1.1]# cat /etc/redhat-release
CentOS Linux release 7.7.1908 (Core)

Upgraded HAPROXy from 1.5 to 2.1 and started having issue where the service wont start.

dev reshaproxy02.dev:root[/opt/haproxy-2.1.1]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: inactive (dead) since Tue 2019-12-17 13:38:04 MST; 58min ago
Process: 102700 ExecStart=/usr/sbin/haproxy-systemd-wrapper -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid $OPTIONS (code=exited, status=0/SUCCESS)
Main PID: 102700 (code=exited, status=0/SUCCESS)

Dec 17 13:38:04 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Started HAProxy Load Balancer.
Dec 17 13:38:04 reshaproxy02.dev.wdc1.wildblue.net haproxy-systemd-wrapper[102700]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Dec 17 13:38:04 reshaproxy02.dev.wdc1.wildblue.net haproxy-systemd-wrapper[102700]: haproxy-systemd-wrapper: exit, haproxy RC=0

My upgrade procedure

tar -zxvf haproxy-2.1.1.tar.gz
cd haproxy-2.1.1
yum install gcc pcre-static pcre-devel openssl-devel -y
make TARGET=linux-glibc USE_OPENSSL=1 USE_PCRE=1 USE_ZLIB=1
make install
cp /usr/local/sbin/haproxy /usr/bin
systemctl enable haproxy.service
systemctl start haproxy
systemctl status haproxy

Ive checked a few links from other upgrades that were posted on this site but none seem to help.

Haproxy 1.7.10 upgrade to 1.8.15 fails to start

Hello, We currently run Haproxy 1.7.10 on Debian 9.4 Stretch. We have used Haproxy for Load Balancing for about 4 to 5 years now without much of an issue. We recently read that starting in 1.8 band Haproxy can be multi threaded when ran vs single threaded, so we want to try to take advantage of this. Unfortunately I am running into some errors when upgrading to 1.8. I do not get any configuration errors, so I believe our haproxy.cfg file is good, but I will post the rest of the errors I am get…

Anyone have any pointers that may help me get this fixed?

Thanks in advance.

[lukastribus]

You should uninstall haproxy 1.5 cleanly, so that when you do make a mistake while installing the new release, you get a proper error message instead of strange behavior.

When you did that, double check that there are no files like /usr/lib/systemd/system/haproxy.service, haproxy and especially /usr/sbin/haproxy-systemd-wrapper.

@harporb

cp /usr/local/sbin/haproxy /usr/bin

Please don’t do copy the executable around, what’s wrong with /usr/local/sbin/?

Once you have a clear situation, install the system unit file that ships with haproxy as per:

Failing to start with systemd and unexplained issues

Do not use the init.d file. You are using systemd, you have use the systemd unit file and wrapper, not the init.d script. rm /etc/init.d/haproxy cd contrib/systemd cp haproxy.service.in /lib/systemd/system/ systemctl daemon-reload systemctl enable haproxy systemctl start haproxy systemctl status haproxy Note: Haproxy 1.8 and later needs to be compiled with USE_SYSTEMD=1 for the provided systemd unit file to work (needs systemd header files, that would be libsystemd-dev on Debian/Ubuntu or sys…

Once you have done all this, follow the procedure here to setup systemd:

[harporb]

@lukastribus

Please don’t do copy the executable around, what’s wrong with /usr/local/sbin/ ?

This was in a book Im reading by Nick Ramirez, it was for HAProxy 1.5 to 1.6. I have since removed this and followed everything you have posted but now am getting a new error

dev reshaproxy02.dev:root[/opt/haproxy-2.1.1]# systemctl status haproxy
Unit haproxy.service could not be found.

My procedure thus far…

Backup your config file to be safe.
Remove old 1.5.x haproxy
	yum remove haproxy
Verify your config file exists, may have been renamed when 1.5 was removed.
	ll /etc/haproxy/
	    haproxy.cfg.rpmsave
Rename it back to original.
    mv /etc/haproxy/haproxy.cfg.rpmsave /etc/haproxy/haproxy.cfg
Verify that "haproxy.service, haproxy" does not exist in /usr/lib/systemd/system/. If so, remove
	ll /usr/lib/systemd/system/ | grep haproxy
Verify "haproxy-systemd-wrapper" does not exit in /usr/sbin. If so, remove.
	ll /usr/sbin/ | grep haproxy
Verify haproxy is not using init.d. If so remove.
	ll /etc/init.d/ | grep haproxy

Download source to /opt
	cd /opt
	tar -zxvf haproxy-2.1.1.tar.gz
	cd haproxy-2.1.1/
	yum install gcc pcre-static pcre-devel openssl-devel systemd-devel -y
	make TARGET=linux-glibc USE_OPENSSL=1 USE_PCRE=1 USE_ZLIB=1 USE_SYSTEMD=1
	make install
	cp /opt/haproxy-2.1.1/contrib/systemd/haproxy.service.in /lib/systemd/system/
	systemctl daemon-reload
	systemctl enable haproxy
	systemctl start haproxy
	systemctl status haproxy

Near the end for the copy statement of haproxy.service.in, should this be renamed to haproxy.service?

[lukastribus]

I don’t know all the CentOs specifics, maybe systemd doesn’t like that path in Centos.

Move the unit file to where the previous file was:

mv /lib/systemd/system/haproxy.service.in /usr/lib/systemd/system/haproxy.service
systemctl daemon-reload
systemctl status haproxy

[harporb]

@lukastribus

aproxy.service

You posted before i could edit. That looks like it helped but still an error im googling for now. Appreciate the help.

dev reshaproxy02.dev:root[/opt/haproxy-2.1.1]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: error (Reason: Invalid argument)
   Active: inactive (dead)

Dec 17 13:38:04 reshaproxy02.dev.wdc1.wildblue.net haproxy-systemd-wrapper[102700]: haproxy-systemd-wrapper: executing /usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -Ds
Dec 17 13:38:04 reshaproxy02.dev.wdc1.wildblue.net haproxy-systemd-wrapper[102700]: haproxy-systemd-wrapper: exit, haproxy RC=0
Dec 18 07:23:58 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: [/usr/lib/systemd/system/haproxy.service:9] Executable path is not absolute, ignoring: @SBINDIR@/haproxy -f $CONFIG -c -q $EXTRAOPTS
Dec 18 07:23:58 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: [/usr/lib/systemd/system/haproxy.service:10] Executable path is not absolute, ignoring: @SBINDIR@/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS
Dec 18 07:23:58 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: [/usr/lib/systemd/system/haproxy.service:11] Executable path is not absolute, ignoring: @SBINDIR@/haproxy -f $CONFIG -c -q $EXTRAOPTS
Dec 18 07:23:58 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service lacks both ExecStart= and ExecStop= setting. Refusing.
Dec 18 07:23:58 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: [/usr/lib/systemd/system/haproxy.service:9] Executable path is not absolute, ignoring: @SBINDIR@/haproxy -f $CONFIG -c -q $EXTRAOPTS
Dec 18 07:23:58 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: [/usr/lib/systemd/system/haproxy.service:10] Executable path is not absolute, ignoring: @SBINDIR@/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS
Dec 18 07:23:58 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: [/usr/lib/systemd/system/haproxy.service:11] Executable path is not absolute, ignoring: @SBINDIR@/haproxy -f $CONFIG -c -q $EXTRAOPTS
Dec 18 07:23:58 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service lacks both ExecStart= and ExecStop= setting. Refusing.

[lukastribus]

You can replace @SBINDIR@ with /usr/local/sbin/ manually, or remove the unit file and use the makefile in contrib/systemd.

Always systemctl daemon-reload after changes to the unit files.

[harporb]

Alright, and yet another issue. This one with less detail.
My config file is good
dev reshaproxy01.dev:root[~]# haproxy -c -f /etc/haproxy/haproxy.cfg
Configuration file is valid

dev reshaproxy01.dev:root[~]# journalctl -xe
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: Failed to start HAProxy Load Balancer.
> – Subject: Unit haproxy.service has failed
> – Defined-By: systemd
> – Support: systemd-devel Info Page
> –
> – Unit haproxy.service has failed.
> –
> – The result is failed.
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: Unit haproxy.service entered failed state.
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: haproxy.service failed.
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: haproxy.service holdoff time over, scheduling restart.
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: Stopped HAProxy Load Balancer.
> – Subject: Unit haproxy.service has finished shutting down
> – Defined-By: systemd
> – Support: systemd-devel Info Page
> –
> – Unit haproxy.service has finished shutting down.
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: start request repeated too quickly for haproxy.service
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: Failed to start HAProxy Load Balancer.
> – Subject: Unit haproxy.service has failed
> – Defined-By: systemd
> – Support: systemd-devel Info Page
> –
> – Unit haproxy.service has failed.
> –
> – The result is failed.
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: Unit haproxy.service entered failed state.
> Dec 18 08:21:10 reshaproxy01.dev.wdc1.wildblue.net systemd[1]: haproxy.service failed.

[lukastribus]

There is zero information in the journal about why it failed.

Can you provide the output of:

which haproxy
haproxy -vv
systemctl status haproxy
cat /usr/lib/systemd/system/haproxy.service

[harporb]

@lukastribus

which haproxy haproxy -vv systemctl status haproxy cat /usr/lib/systemd/system/haproxy.service

dev reshaproxy02.dev:root[~]# which haproxy
/usr/local/sbin/haproxy

dev reshaproxy02.dev:root[~]# haproxy -vv
HA-Proxy version 2.1.1 2019/12/11 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2021.
Known bugs: http://www.haproxy.org/bugs/bugs-2.1.1.html
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
  OPTIONS = USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2
            fcgi : mode=HTTP       side=BE        mux=FCGI
       <default> : mode=HTTP       side=FE|BE     mux=H1
       <default> : mode=TCP        side=FE|BE     mux=PASS

Available services : none

Available filters :
	[SPOE] spoe
	[CACHE] cache
	[FCGI] fcgi-app
	[TRACE] trace
	[COMP] compression

dev reshaproxy02.dev:root[~]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2019-12-18 08:12:23 MST; 57min ago
 Main PID: 122435 (code=exited, status=1/FAILURE)

Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service: main process exited, code=exi...URE
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Failed to start HAProxy Load Balancer.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Unit haproxy.service entered failed state.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service failed.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service holdoff time over, scheduling ...rt.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Stopped HAProxy Load Balancer.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: start request repeated too quickly for haproxy...ice
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Failed to start HAProxy Load Balancer.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Unit haproxy.service entered failed state.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

dev reshaproxy02.dev:root[~]# cat /usr/lib/systemd/system/haproxy.service
[Unit]
Description=HAProxy Load Balancer
After=network.target

[Service]
EnvironmentFile=-/etc/default/haproxy
EnvironmentFile=-/etc/sysconfig/haproxy
Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid" "EXTRAOPTS=-S /run/haproxy-master.sock"
ExecStartPre=/usr/local/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecStart=/usr/local/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS
ExecReload=/usr/local/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS
ExecReload=/bin/kill -USR2 $MAINPID
KillMode=mixed
Restart=always
SuccessExitStatus=143
Type=notify

# The following lines leverage SystemD's sandboxing options to provide
# defense in depth protection at the expense of restricting some flexibility
# in your setup (e.g. placement of your configuration files) or possibly
# reduced performance. See systemd.service(5) and systemd.exec(5) for further
# information.

# NoNewPrivileges=true
# ProtectHome=true
# If you want to use 'ProtectSystem=strict' you should whitelist the PIDFILE,
# any state files and any other files written using 'ReadWritePaths' or
# 'RuntimeDirectory'.
# ProtectSystem=true
# ProtectKernelTunables=true
# ProtectKernelModules=true
# ProtectControlGroups=true
# If your SystemD version supports them, you can add: @reboot, @swap, @sync
# SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io

[Install]
WantedBy=multi-user.target

[lukastribus]

As always systemd completely hides the actual errors the application provides and makes it impossible to troubleshoot.

A shot in the dark: are you sure that no old haproxy instance is running? Please killall haproxy and retry.

Otherwise, you will have to start it manually and check if there are any errors:

/usr/local/sbin/haproxy -f /etc/haproxy/haproxy.cfg -db

[harporb]

Yeah, I dont see any process running for haproxy but oddly enough it has a main pid when running systemctl status haproxy. Maybe next option is to uninstall and try again? May have to remove the files manually as im not sure make uninstall will work. Thoughts? I also dont have to use haproxy 2.1 it was the most recent stable version is why I chose it. Wish there was an rpm for the install.

dev reshaproxy02.dev:root[/opt/haproxy-2.1.1]# ps -ef | grep haproxy
root      12468  11428  0 06:30 pts/0    00:00:00 grep --color=auto haproxy
dev reshaproxy02.dev:root[/opt/haproxy-2.1.1]#
dev reshaproxy02.dev:root[/opt/haproxy-2.1.1]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
   Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2019-12-18 08:12:23 MST; 22h ago
 Main PID: 122435 (code=exited, status=1/FAILURE)

Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service: main process exited, code=exited, status=1/FAILURE
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Failed to start HAProxy Load Balancer.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Unit haproxy.service entered failed state.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service failed.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service holdoff time over, scheduling restart.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Stopped HAProxy Load Balancer.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: start request repeated too quickly for haproxy.service
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Failed to start HAProxy Load Balancer.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Unit haproxy.service entered failed state.
Dec 18 08:12:23 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: haproxy.service failed.

[harporb]

You hinted to this before but not sure why this is the case. It seems as if some file was not being installed, even after a few attempts at the make install. HAproxy is now working. After I posted above I went and did a “make uninstall”

To fix the unknown issue this is what i did…
cd /opt/haproxy-2.1.1/
make uninstall
make TARGET=linux-glibc USE_OPENSSL=1 USE_PCRE=1 USE_ZLIB=1 USE_SYSTEMD=1
make install
cp /opt/haproxy-2.1.1/contrib/systemd/haproxy.service.in /lib/systemd/system/haproxy.service
vi /lib/systemd/system/haproxy.service #replace @SBINDIR@ with /usr/local/sbin/
systemctl daemon-reload
systemctl enable haproxy
systemctl start haproxy
systemctl status haproxy

And all is good now. Unbelievable. Again my only thought here is that there was a missing/corrupt/stale file for some reason.
dev reshaproxy02.dev:root[/opt/haproxy-2.1.1]# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/haproxy.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-12-19 06:51:25 MST; 12min ago
Process: 14057 ExecStartPre=/usr/local/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 14060 (haproxy)
CGroup: /system.slice/haproxy.service
├─14060 /usr/local/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
└─14062 /usr/local/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

Dec 19 06:51:25 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Starting HAProxy Load Balancer...
Dec 19 06:51:25 reshaproxy02.dev.wdc1.wildblue.net haproxy[14060]: [NOTICE] 352/065125 (14060) : New worker #1 (14062) forked
Dec 19 06:51:25 reshaproxy02.dev.wdc1.wildblue.net systemd[1]: Started HAProxy Load Balancer.

@ lukastribus Thank you for the assistance.