Health checks not using changing IP addresses

Help!
1

Hi there,

I’m looking for a solution for dynamically changing IP addresses behind hostnames.
Actual use case is the RDS in an AWS environment which due to maintenance sometimes gets a new IP address.

Now, it seems the forwarding of traffic seems to recognize the new IP address without restarting the HAProxy service - the health checks are not.
It seems health checks are using the IP address the first time the service starts up and resolves the hostnames - and that is not quite good of a health check…

DNS works fine in our system, IP address is being updated, but still the health check is looking into the old IP address causing the entry to go down and never come up again.

Is it possible to set a reset for the health check to periodically resolve the hostname again?
Or is the solution to not use health checks at all for hostnames that have dynamic IPs?

Thanks for the help!

PS: It’s about TCP traffic only… Maybe thats important, idk.

2

Share you configuration and output of haproxy -vv please.

3 
global
   log /dev/log local0
   log /dev/log local1 notice
   chroot /var/lib/haproxy
   stats timeout 30s
   pidfile /var/run/haproxy.pid
   maxconn 4000
   user haproxy
   group haproxy
   daemon
   stats socket /var/lib/haproxy/stats

defaults
   timeout connect 5m
   timeout client 60m
   timeout server 60m
   default-server init-addr libc,none
   log global
   mode tcp
   option tcplog
   option tcp-check
   maxconn 3000
   source xxx

listen Test_1234
   bind *:1234
   server test.123456789.aws.net:1521 test.123456789.aws.net:1521 check inter 10s fall 3 rise 2

sh-4.2$ sudo haproxy -vv
HAProxy version 2.6.13-234aa6d 2023/05/02 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.13.html
Running on: Linux 4.14.311-233.529.amzn2.aarch64 #1 SMP Thu Mar 23 09:54:09 UTC 2023 aarch64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS =
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY-LUA -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER -OPENSSL -OT -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -QUIC +RT +SLZ -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.3.1 20180712 (Red Hat 7.3.1-15)

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
4

There is not even a resolver configured here.

Not only will the health check stay at the old IP address, production traffic will too.

Perhaps you shared an incomplete configuration?

5

We are using the DNS settings configured on the operating system where HAProxy is running.
No resolver config was needed for now - what is your suggestion?

6

Please read section 5.3 Server IP address resolution using DNS of the configuration manual:

http://docs.haproxy.org/2.6/configuration.html#5.3

7

Checked that section, it is even mentioning that use case, thank you for that.
But it doesn’t really help me further implementing continuous renewing of IP of hostnames.

I added the following to the config and still if I run my tests the server never comes back to UP again:

resolvers customdnssettings
  resolve_retries 1800
  timeout resolve 1s
  timeout retry   1s
  hold other      10s
  hold refused    10s
  hold nx         10s
  hold timeout    10s
  hold valid      10s

listen Test_1234
   bind *:1234
   server test.123456789.aws.net:1521 test.123456789.aws.net:1521 check resolvers customdnssettings inter 10s fall 3 rise 2

Test:

  1. Have a target behind test.123456789.aws.net to be up and running. HAProxy stats page lights up green.
  2. Make this server go down.
  3. Stats page: Entry turns red.
  4. Change the DNS entry to something available.
  5. Waiting for the DNS of the OS to update - once ping comes back from correct new location look at the stats page:
  6. Entry never comes back to green UP status ever…

telnet works fine from that server to the new IP & Port.
If I boot up the “old” server it goes again to green. Which means that it is only health checking against the first IP it got when starting the HAProxy service.

8

There are no nameservers configured in the section. You need to at at least parse-resolv-conf in the section, or specify nameservers manually.

I also recommend setting accepted_payload_size to 8192.

Your safest bet is to configure nameservers manually in TCP mode with accepted_payload_size set to 65535, to avoid using truncated DNS responses:

resolvers customdnssettings
  nameserver dns1 tcp@8.8.8.8
  nameserver dns2 tcp@1.1.1.1
 accepted_payload_size 65535

Also I’d recommend removing libc from it altogether, so that you don’t have to wait around to see if the resolver works or not, but if the haproxy resolver is not actually used or doesn’t actually work, then the configuration won’t work at all (as opposed to working for a bit due to startup resolution, but then fail to update):

default-server init-addr none
1 Like
9

In the end the following settings needed to be set to make it work:

default-server init-addr none

resolvers customdnssettings
  nameserver dns1 10.1.2.3:53
  resolve_retries 1800
  timeout resolve 1s
  timeout retry   1s
  hold other      10s
  hold refused    10s
  hold nx         10s
  hold timeout    10s
  hold valid      10s

parse-resolv-conf did not work at all, although the same server is mentioned in the resolv.conf as nameserver (but without port 53). It needs to be explicitly mentioned in the HAProxy config itself.

Thank you Lukas for your efforts!

1 Like