Issues proxying in K8s

I am using HAProxy to provide a proxy around internal containers that are part of an HA control plane. Only one of these nodes is valid to respond to web ui queries at a time. The other one returns a redirect if you try to access it. I am using HAProxy to manage this. It all works fine until the control plane nodes are restarted or fail. They change ip addresses at that point. Because HAProxy caches the DNS entry at startup it can never resolve the new ip addresses.

After some searching I found that I needed to specify a resolver. Hoping for the best I started down that path.

My first attempt started out well, but after a few seconds the primary node dropped out with an error
“Server webui-backend/manager-0 is going DOWN for maintenance (DNS NX status)”

With some research I found that it was likely failing because the resolver can’t handle anything besides FQDN. So I made that change. Now I get a different error:
Server webui-backend/manager-0 is going DOWN for maintenance (unspecified DNS error).

I haven’t had much luck with Googling so I thought I would ask here. Help! :slight_smile:

pidfile /var/run/
log local0
log local1 notice
maxconn 4096

resolvers dns
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold valid 10s
hold obsolete 30s

default-server init-addr last,libc,none
mode http
balance leastconn
retries 3
log global
option httplog
option http-server-close
option dontlognull
timeout connect 30ms
timeout check 1000ms
timeout client 30000ms
timeout server 30000ms
option httpchk GET /overview HTTP/1.0\r\nHost:\
http-check expect status 200

frontend webui
bind *:8081

http-request set-header X-Forwarded-For %[src]
http-request set-header X-Forwarded-Proto https

default_backend webui-backend

backend webui-backend
server manager-0 manager-0.manager.default.svc.cluster.local:9081 check resolvers dns
server manager-1 manager-1.manager.default.svc.cluster.local:9081 check resolvers dns

haproxy -vv:

HA-Proxy version 1.9.0 2018/12/19 -
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
: mode=HTX side=FE|BE
: mode=TCP|HTTP side=FE|BE

Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace

That’s most likely the DNS regression in 1.8.15 and 1.9.0.

I suggest you use 1.8.16 if you don’t need any 1.9 specific features or wait for 1.9.1, which should be released in a relative short period of time.

That appears to have fixed the problem. I lost my parse-resolv-conf config option, but I can hard code that for now.


1.9.1 has been released with the fix for this bug.

Started seeing this “(unspecified DNS error)” error after several requests.
This is weird, the haproxy 1.8 that comes with ubuntu:18.04 had been working fine.
Anyway, installing 1.9.8 using this repo fixed that mysterious problem.

1 Like