Consistent hash and server-template via dns resolver

Help!
1

Hello community,

We’re using HAProxy in Kubernetes as a consistent hashing load balancer in front of a deployment of five total pods (real HAProxy, not the ingress controller version of it). Here’s our config:

global
	daemon
	maxconn 10000
	stats socket /usr/local/etc/haproxy/admin.sock mode 600 level admin
	log /dev/log local0
	
defaults
	mode http
	timeout connect 5000ms
	timeout client 30000ms
	timeout server 30000ms

resolvers kubernetes
	nameserver skydns kube-dns.kube-system:53
	resolve_retries 10
	timeout retry 2s
	hold valid 5s

frontend http-in
	bind *:80
	log /dev/log local0
	option httplog
	default_backend servers

backend servers
	balance uri depth 3
	hash-type consistent
	option httpchk GET /health
	http-check expect status 200
	option tcp-check
	server-template pod 5 pod.namespace.svc.cluster.local:8080 check resolvers kubernetes inter 500

As you can see we’re leveraging the server-templates and k8s dns resolvers to create the backend servers dynamically.

Now since we’re using consistent hashing we assume that we can just scale the haproxy frontend to multiple pods and have the same load balancing across the backend servers. However, when we try that out, we observe that the routing is consistent on one haproxy, but not on the others. So it seems that the order of the backend servers is different on each of the instances of haproxy.

My main question here is: is there a way to guarantee ordering of the server-template via the dns resolver in this case? A simple sort on the resulting IPs would probably good enough for us, since we don’t control the DNS server and the replies.

Thanks,
Thomas

2 Likes
2

Hi Thomas, did you ever find a solution to this problem? I am having the same problem in nginx… I thought to try HAProxy instead of nginx to solve this problem, but it seems (according to your post here) the problem also exists in HAProxy.

It seems consistent hashing of DNS replies depends on order of entries in the response… somewhat sadly.

My current best thinking is to layer an additional DNS on top of the existing DNS which might re-order the entries and be able to respond in a non round robin fashion.

Any thoughts/ideas? Did you get this working?

Best,
Ian

3

Any solution for this problem? We are thinking of using a manual dns solution using /etc/hosts values for each host and update it periodically

4

In haproxy 2.4 you will be able to use the srvkey addr option on your stick-table line to use the address rather than the server name as the key for stick tables (see HAProxy version 2.4-dev10 - Configuration Manual).

But with stable releases I don’t know of a solution better than dynamically generating the haproxy config yourself.

1 Like