We’re using HAProxy in Kubernetes as a consistent hashing load balancer in front of a deployment of five total pods (real HAProxy, not the ingress controller version of it). Here’s our config:
global daemon maxconn 10000 stats socket /usr/local/etc/haproxy/admin.sock mode 600 level admin log /dev/log local0 defaults mode http timeout connect 5000ms timeout client 30000ms timeout server 30000ms resolvers kubernetes nameserver skydns kube-dns.kube-system:53 resolve_retries 10 timeout retry 2s hold valid 5s frontend http-in bind *:80 log /dev/log local0 option httplog default_backend servers backend servers balance uri depth 3 hash-type consistent option httpchk GET /health http-check expect status 200 option tcp-check server-template pod 5 pod.namespace.svc.cluster.local:8080 check resolvers kubernetes inter 500
As you can see we’re leveraging the server-templates and k8s dns resolvers to create the backend servers dynamically.
Now since we’re using consistent hashing we assume that we can just scale the haproxy frontend to multiple pods and have the same load balancing across the backend servers. However, when we try that out, we observe that the routing is consistent on one haproxy, but not on the others. So it seems that the order of the backend servers is different on each of the instances of haproxy.
My main question here is: is there a way to guarantee ordering of the server-template via the dns resolver in this case? A simple sort on the resulting IPs would probably good enough for us, since we don’t control the DNS server and the replies.