Resolvers inside kubernetes


#1

Hi,

We are using the haproxy resolvers feature. It works awesome, but I would be grateful to get some clarifications on what haproxy does if it can’t reache the DNS server(s) or DNS servers don’t provide the requested dns entry.

The main issue we seem to have is that haproxy stops to ask the DNS server if it can resolve a hostname forever in some scenarios.

Our setup looks like this:

resolvers kubernetes
nameserver skydns ${DNS}:53
resolve_retries 10
timeout retry 2s
hold valid 30s

backend servers
mode http
timeout connect 10s
timeout server 1m
balance roundrobin
server host1 virt-api-service:8183 resolvers kubernetes

“$DNS” points to the kubernetes dns server, “virt-api-service” is sooner or later resolvable over this $DNS server, but when we start the application we can’t guarantee that the DNS server is already reachable and if it is, we can’t guarantee that it already knows about “virt-api-server”.

How would you configure the resolvers to deal with such situations?

If I understand https://cbonte.github.io/haproxy-dconv/configuration-1.6.html#5.3.2 correct, it might be that it is not possible. Could that be?

Best Regards,
Roman


#2

Here is the full config:


#3

Hi,

You miss the “resolvers kubernetes” statement on your server line, that’s why you don’t see any DNS resolution during runtime of HAProxy.

As well, I would recommend using HAProxy 1.7 and set an “init-addr none” statement on the server lien too.
That way, if ${KUBERNETES_SERVICE_HOST} can’t be resolved when HAProxy starts up, it will still do the resolution during runtime until it can get an IP. Of course, the server will be DOWN up to that point.

Baptiste


#4

Hi thanks for you response. Thanks for the “inet-addr none” hint but KUBERNETES_SERVICE_HOST is an IP anyway. I will also update haproxy.

But that my main problem is this:

I have the resolver in the backend section below the shown section above (the file is truncated). It also works like I expect it. HAPROXY comes up even if the service is not there yet, but it seems that when I want to access it before my DNS server is there, HAPROXY is not refeshing anymore and just ignoring that DNS server forever.

Thank you,

Roman


#5

As a reminder, this is your current server line:

server host1 ${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT} check ssl ca-file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Well, why do you need DNS resolution then??

I’m sorry, but with the server line above, HAProxy won’t enable DNS resolution at run time.
You need the following one:

server host1 ${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT} check ssl ca-file /var/run/secrets/kubernetes.io/serviceaccount/ca.crt resolvers kubernetes

#6

Ah my bad, sorry, I looked at the wrong backend :slight_smile:

You have to enable “check” on the server line to enable DNS resolution at run time:

server host1 virt-api-service:8183 check resolvers kubernetes

#7

Also, could you please share here (or to my mail bedis9@gmail.com) how you setup your HAProxy container in Kubernetes?
I’m working on improving DNS in HAProxy with scaling feature and your help will allow me to save a lot of time!


#8

This is how we build the docker image:


This is how we deploy it to kubernetes:

Hope this helps you a little bit.


#9

Added the “check” also to the server line.

Hm, on 1.7 when I add check, haproxy looses the connection after a few seconds and reports that there are no more backends there.

On 1.6, everything works as expected, until my backend has no more servers (e.g. because of a redeploy).
Even if dns is then reporting all required dns names with the new ip properly, haproxy still insists on not seeing any servers.


#10

Hi @rmohr - Did you figured out how to solve this?
I’m facing exactly the same problem on my HA Proxy (1.7) towards an Redis Statefulsets implementation in a Kubernetes cluster.
It works, but after some seconds I face: Error: Server closed the connection


#11

I’m not sure - but for me there is a problem because:

  1. kubernetes create service name like <service>.<namespace>.svc.cluster.local

  2. for libc initial resolver - it’s enought to use <service> as server name in backend - because there is “search” entry in /etc/resolver.conf

  3. for dns resolver included in haproxy - that backend after few seconds is not available anymore, becase it doesn’t support “search” directive and do requests for <service>

Workaround - is use full name of server name (like <service>.<namespace>.svc.cluster.local)


#12

Yes, HAProxy runtime resolver supports fqdn only for now.