HAProxy community

HAProxy fails on restart


#1

If i restart our Ubuntu 18.04 Server with HAProxy 1.8.8 it fails with the error:

service haproxy status

**●** haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: **failed** (Result: exit-code) since Sun 2019-02-03 19:29:25 CET; 2min 43s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 1047 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS **(code=exited, status=1/FAILURE)**
Feb 03 19:29:25 lb02 systemd[1]: haproxy.service: Service hold-off time over, scheduling restart.
Feb 03 19:29:25 lb02 systemd[1]: haproxy.service: Scheduled restart job, restart counter is at 5.
Feb 03 19:29:25 lb02 systemd[1]: Stopped HAProxy Load Balancer.
Feb 03 19:29:25 lb02 systemd[1]: **haproxy.service: Start request repeated too quickly.**
Feb 03 19:29:25 lb02 systemd[1]: **haproxy.service: Failed with result 'exit-code'.**
Feb 03 19:29:25 lb02 systemd[1]: **Failed to start HAProxy Load Balancer.**

And haproxy.log contains:

Feb  3 19:53:41 lb02 haproxy[869]: [ALERT] 033/195341 (869) : parsing [/etc/haproxy/haproxy.cfg:82] : 'mailer smtp1' : invalid address: 'subdomain.hostname.com' in 'subdomain.hostname.com:25'
Feb  3 19:53:41 lb02 haproxy[869]: [ALERT] 033/195341 (869) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Feb  3 19:53:41 lb02 haproxy[869]: [ALERT] 033/195341 (869) : Fatal errors found in configuration.

In haproxy.conf my config looks like:

mailers mymailers01
	 mailer smtp1 subdomain.hostname.com:25

If i manually do “service haproxy restart” it’s working again.

haproxy -vv
HA-Proxy version 1.8.8-1ubuntu0.3 2019/01/11
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -g -O2 -fdebug-prefix-map=/build/haproxy-_XfIfg/haproxy-1.8.8=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0g  2 Nov 2017
Running on OpenSSL version : OpenSSL 1.1.0g  2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace

Any idea?


#2

I feel like it’s a looking up DNS type issue, do you have a “resolvers” section?


#3

No, i don’t have one. Where would i configure this, do you mean: https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#5.3.2 ?

Normal dig query works.


#4

Yes, that’s exactly what I mean.

I think I’d try putting in a valid resolvers section, it maybe shouldn’t need it as it’s fine when you start the service but it’s clearly trying and failing to lookup on the restart.

The other option is switch to using an IP address for SMTP mailer and see if the problem persists, i have a feeling it wont…


#5

Something like:

resolvers mydns
  nameserver google 8.8.8.8:53
  nameserver opendns 208.67.222.222:53

#6

I don’t understand the difference between:

and:

What are you saying? When you are rebooting the entire OS, haproxy fails to startup?


#7

Yes, that’s the difference / problem. It only works if you manually issue “service haproxy restart” after OS reboot.

I will try the suggested solution from @AaronWest


#8

Ok, then that’s a different issue than what Aaron is trying to address.

Haproxy is trying to start before network / and name resolution is ready.

Can you please share how your startup script looks like?

cat /lib/systemd/system/haproxy.service

#9

Of course, but for me it looks OK:

cat /lib/systemd/system/haproxy.service

[Unit]

Description=HAProxy Load Balancer

Documentation=man:haproxy(1)

Documentation=file:/usr/share/doc/haproxy/configuration.txt.gz

After=network.target rsyslog.service

[Service]

EnvironmentFile=-/etc/default/haproxy

Environment="CONFIG=/etc/haproxy/haproxy.cfg" "PIDFILE=/run/haproxy.pid"

ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS

ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS

ExecReload=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS

ExecReload=/bin/kill -USR2 $MAINPID

KillMode=mixed

Restart=always

Type=notify

# The following lines leverage SystemD's sandboxing options to provide

# defense in depth protection at the expense of restricting some flexibility

# in your setup (e.g. placement of your configuration files) or possibly

# reduced performance. See systemd.service(5) and systemd.exec(5) for further

# information.

# NoNewPrivileges=true

# ProtectHome=true

# If you want to use 'ProtectSystem=strict' you should whitelist the PIDFILE,

# any state files and any other files written using 'ReadWritePaths' or

# 'RuntimeDirectory'.

# ProtectSystem=true

# ProtectKernelTunables=true

# ProtectKernelModules=true

# ProtectControlGroups=true

# If your SystemD version supports them, you can add: @reboot, @swap, @sync

# SystemCallFilter=~@cpu-emulation @keyring @module @obsolete @raw-io

[Install]

WantedBy=multi-user.target

#10

Try replacing the ExecStartPre in there with the following:

ExecStartPre=/bin/bash -c 'until host subdomain.hostname.com; do sleep 1; done' && /usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS

Replace the subdomain.hostname.com with the real hostname you are trying to resolve.

Completely stolen from:

(other workarounds in there may also work).


#11

Thank you, this helped me to solve the issue. I ended up with using

[Unit]
After=network-online.target

Instead of

[Unit]
After=network.target

See: https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/

ExecStartPre=/bin/bash -c 'until host subdomain.hostname.com; do sleep 1; done' && /usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS

Might work too.


#12

I did not know about network-online.target, thanks. It’s certainly the better choice if it works.

Looping through the host command until it succeeds is a workaround.