I am using HAProxy to provide a proxy around internal containers that are part of an HA control plane. Only one of these nodes is valid to respond to web ui queries at a time. The other one returns a redirect if you try to access it. I am using HAProxy to manage this. It all works fine until the control plane nodes are restarted or fail. They change ip addresses at that point. Because HAProxy caches the DNS entry at startup it can never resolve the new ip addresses.
After some searching I found that I needed to specify a resolver. Hoping for the best I started down that path.
My first attempt started out well, but after a few seconds the primary node dropped out with an error
“Server webui-backend/manager-0 is going DOWN for maintenance (DNS NX status)”
With some research I found that it was likely failing because the resolver can’t handle anything besides FQDN. So I made that change. Now I get a different error:
Server webui-backend/manager-0 is going DOWN for maintenance (unspecified DNS error).
I haven’t had much luck with Googling so I thought I would ask here. Help! 
Config:
global
pidfile /var/run/haproxy.pid
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
maxconn 4096
resolvers dns
parse-resolv-conf
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold valid 10s
hold obsolete 30s
defaults
default-server init-addr last,libc,none
mode http
balance leastconn
retries 3
log global
option httplog
option http-server-close
option dontlognull
timeout connect 30ms
timeout check 1000ms
timeout client 30000ms
timeout server 30000ms
option httpchk GET /overview HTTP/1.0\r\nHost:\ manager.example.com
http-check expect status 200
frontend webui
bind *:8081
http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Proto https
default_backend webui-backend
backend webui-backend
server manager-0 manager-0.manager.default.svc.cluster.local:9081 check resolvers dns
server manager-1 manager-1.manager.default.svc.cluster.local:9081 check resolvers dns
haproxy -vv:
HA-Proxy version 1.9.0 2018/12/19 - https://haproxy.org/
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
: mode=HTX side=FE|BE
: mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace