We are using the haproxy resolvers feature. It works awesome, but I would be grateful to get some clarifications on what haproxy does if it can’t reache the DNS server(s) or DNS servers don’t provide the requested dns entry.
The main issue we seem to have is that haproxy stops to ask the DNS server if it can resolve a hostname forever in some scenarios.
backend servers
mode http
timeout connect 10s
timeout server 1m
balance roundrobin
server host1 virt-api-service:8183 resolvers kubernetes
“$DNS” points to the kubernetes dns server, “virt-api-service” is sooner or later resolvable over this $DNS server, but when we start the application we can’t guarantee that the DNS server is already reachable and if it is, we can’t guarantee that it already knows about “virt-api-server”.
How would you configure the resolvers to deal with such situations?
You miss the “resolvers kubernetes” statement on your server line, that’s why you don’t see any DNS resolution during runtime of HAProxy.
As well, I would recommend using HAProxy 1.7 and set an “init-addr none” statement on the server lien too.
That way, if ${KUBERNETES_SERVICE_HOST} can’t be resolved when HAProxy starts up, it will still do the resolution during runtime until it can get an IP. Of course, the server will be DOWN up to that point.
Hi thanks for you response. Thanks for the “inet-addr none” hint but KUBERNETES_SERVICE_HOST is an IP anyway. I will also update haproxy.
But that my main problem is this:
I have the resolver in the backend section below the shown section above (the file is truncated). It also works like I expect it. HAPROXY comes up even if the service is not there yet, but it seems that when I want to access it before my DNS server is there, HAPROXY is not refeshing anymore and just ignoring that DNS server forever.
Also, could you please share here (or to my mail bedis9@gmail.com) how you setup your HAProxy container in Kubernetes?
I’m working on improving DNS in HAProxy with scaling feature and your help will allow me to save a lot of time!
Hm, on 1.7 when I add check, haproxy looses the connection after a few seconds and reports that there are no more backends there.
On 1.6, everything works as expected, until my backend has no more servers (e.g. because of a redeploy).
Even if dns is then reporting all required dns names with the new ip properly, haproxy still insists on not seeing any servers.
Hi @rmohr - Did you figured out how to solve this?
I’m facing exactly the same problem on my HA Proxy (1.7) towards an Redis Statefulsets implementation in a Kubernetes cluster.
It works, but after some seconds I face: Error: Server closed the connection
I’m not sure - but for me there is a problem because:
kubernetes create service name like <service>.<namespace>.svc.cluster.local
for libc initial resolver - it’s enought to use <service> as server name in backend - because there is “search” entry in /etc/resolver.conf
for dns resolver included in haproxy - that backend after few seconds is not available anymore, becase it doesn’t support “search” directive and do requests for <service>
Workaround - is use full name of server name (like <service>.<namespace>.svc.cluster.local)
