HAProxy fails on restart

marcusm · February 3, 2019, 7:03pm

If i restart our Ubuntu 18.04 Server with HAProxy 1.8.8 it fails with the error:

service haproxy status

**●** haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: **failed** (Result: exit-code) since Sun 2019-02-03 19:29:25 CET; 2min 43s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 1047 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS **(code=exited, status=1/FAILURE)**
Feb 03 19:29:25 lb02 systemd[1]: haproxy.service: Service hold-off time over, scheduling restart.
Feb 03 19:29:25 lb02 systemd[1]: haproxy.service: Scheduled restart job, restart counter is at 5.
Feb 03 19:29:25 lb02 systemd[1]: Stopped HAProxy Load Balancer.
Feb 03 19:29:25 lb02 systemd[1]: **haproxy.service: Start request repeated too quickly.**
Feb 03 19:29:25 lb02 systemd[1]: **haproxy.service: Failed with result 'exit-code'.**
Feb 03 19:29:25 lb02 systemd[1]: **Failed to start HAProxy Load Balancer.**

And haproxy.log contains:

Feb  3 19:53:41 lb02 haproxy[869]: [ALERT] 033/195341 (869) : parsing [/etc/haproxy/haproxy.cfg:82] : 'mailer smtp1' : invalid address: 'subdomain.hostname.com' in 'subdomain.hostname.com:25'
Feb  3 19:53:41 lb02 haproxy[869]: [ALERT] 033/195341 (869) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Feb  3 19:53:41 lb02 haproxy[869]: [ALERT] 033/195341 (869) : Fatal errors found in configuration.

In haproxy.conf my config looks like:

mailers mymailers01
	 mailer smtp1 subdomain.hostname.com:25

If i manually do “service haproxy restart” it’s working again.

haproxy -vv
HA-Proxy version 1.8.8-1ubuntu0.3 2019/01/11
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -g -O2 -fdebug-prefix-map=/build/haproxy-_XfIfg/haproxy-1.8.8=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0g  2 Nov 2017
Running on OpenSSL version : OpenSSL 1.1.0g  2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace

Any idea?

AaronWest · February 4, 2019, 7:06pm

I feel like it’s a looking up DNS type issue, do you have a “resolvers” section?

marcusm · February 4, 2019, 7:22pm

No, i don’t have one. Where would i configure this, do you mean: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.3.2 ?

Normal dig query works.

AaronWest · February 4, 2019, 9:06pm

Yes, that’s exactly what I mean.

I think I’d try putting in a valid resolvers section, it maybe shouldn’t need it as it’s fine when you start the service but it’s clearly trying and failing to lookup on the restart.

The other option is switch to using an IP address for SMTP mailer and see if the problem persists, i have a feeling it wont…

AaronWest · February 4, 2019, 9:07pm

Something like:

resolvers mydns
  nameserver google 8.8.8.8:53
  nameserver opendns 208.67.222.222:53

lukastribus · February 7, 2019, 10:02pm

I don’t understand the difference between:

and:

What are you saying? When you are rebooting the entire OS, haproxy fails to startup?

marcusm · February 7, 2019, 10:15pm

Yes, that’s the difference / problem. It only works if you manually issue “service haproxy restart” after OS reboot.

I will try the suggested solution from @AaronWest

lukastribus · February 7, 2019, 10:18pm

Ok, then that’s a different issue than what Aaron is trying to address.

Haproxy is trying to start before network / and name resolution is ready.

Can you please share how your startup script looks like?

cat /lib/systemd/system/haproxy.service

marcusm · February 8, 2019, 6:07am

Of course, but for me it looks OK:

cat /lib/systemd/system/haproxy.service

[Unit]

Description=HAProxy Load Balancer

Documentation=man:haproxy(1)

Documentation=file:/usr/share/doc/haproxy/configuration.txt.gz

After=network.target rsyslog.service

[Service]

EnvironmentFile=-/etc/default/haproxy

Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"

ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS

ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS

ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS

ExecReload=/bin/kill -USR2 $MAINPID

KillMode=mixed

Restart=always

Type=notify

# The following lines leverage SystemD's sandboxing options to provide

# defense in depth protection at the expense of restricting some flexibility

# in your setup (e.g. placement of your configuration files) or possibly

# reduced performance. See systemd.service(5) and systemd.exec(5) for further

# information.

# NoNewPrivileges=true

# ProtectHome=true

# If you want to use 'ProtectSystem=strict' you should whitelist the PIDFILE,

# any state files and any other files written using 'ReadWritePaths' or

# 'RuntimeDirectory'.

# ProtectSystem=true

# ProtectKernelTunables=true

# ProtectKernelModules=true

# ProtectControlGroups=true

# If your SystemD version supports them, you can add: @reboot, @swap, @sync

# SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io

[Install]

WantedBy=multi-user.target

lukastribus · February 8, 2019, 5:44pm

Try replacing the ExecStartPre in there with the following:

ExecStartPre=/bin/bash -c 'until host subdomain.hostname.com; do sleep 1; done' && /usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS

Replace the subdomain.hostname.com with the real hostname you are trying to resolve.

Completely stolen from:

(other workarounds in there may also work).

marcusm · February 17, 2019, 4:14pm

Thank you, this helped me to solve the issue. I ended up with using

[Unit]
After=network-online.target

Instead of

[Unit]
After=network.target

See: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

ExecStartPre=/bin/bash -c 'until host subdomain.hostname.com; do sleep 1; done' && /usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS

Might work too.

lukastribus · February 17, 2019, 7:51pm

I did not know about network-online.target, thanks. It’s certainly the better choice if it works.

Looping through the host command until it succeeds is a workaround.

Topic		Replies	Views
Haproxy 1.8.30 start failure Help!	2	704	December 16, 2021
Failed to start HAProxy Load Balancer Help!	16	62811	November 10, 2021
HAProxy Fails to Start Ubuntu 16.04 Help!	5	18529	January 31, 2018
2.8.2 to 2.9.6 HAPROXY Wont start after update Help!	7	475	May 15, 2024
Failed to start HAProxy Load Balancer .Failed with result 'exit-code' Help!	1	3417	May 2, 2022

HAProxy fails on restart

Related Topics