Trouble with DNS Resolvers sticking to single IP address

Baptiste · June 12, 2017, 8:15am

Yes, please, give a try to 1.8dev2. The responses are now stored in a local cache and each time a record is consumed, it’s moved to the back of the list:
http://git.haproxy.org/?p=haproxy.git;a=commitdiff;h=8ea0bcc911809e77560bdd937c02a0b832526ef7

Note that it’s a bit limited in some use-cases:

does not look for records in other IP family (under dev)
when adding a new IP in the response, only 1 server will use it (we don’t change many servers at the mean time unless there is a good reason for it)

Baptiste

z0mb1ek · June 13, 2017, 12:12am

when adding a new IP in the response, only 1 server will use it (we don’t change many servers at the mean time unless there is a good reason for it)

what does it mean?

Baptiste · June 13, 2017, 3:06pm

Let’s take a backend like the one below:
backend my_app server s1 myapp.domain.com:80 resolvers mydns server s2 myapp.domain.com:80 resolvers mydns server s3 myapp.domain.com:80 resolvers mydns server s4 myapp.domain.com:80 resolvers mydns server s5 myapp.domain.com:80 resolvers mydns server s6 myapp.domain.com:80 resolvers mydns

if myapp.domain.com returns 10.0.0.1 and 10.0.0.2, then 3 servers will be affected to each IP.
Now, if you add one more record to myapp.domain.com (10.0.0.1, 10.0.0.2 and 10.0.0.3), then only 1 server will pick up this third IP address.

We can’t do better for now, but we’ll work on improving this situation.

z0mb1ek · June 13, 2017, 5:23pm

if myapp.domain.com returns 10.0.0.1 and 10.0.0.2, then 3 servers will be affected to each IP.

s1,s2,s3 for 10.0.0.1 and s4,s5,s6 for 10.0.0.2? And then s1,s2,s3-10.0.0.1; s4,s5-10.0.0.2 and s6-10.0.0.3 this way?

Thus if a have two ips for one dns record I need two same server record like this?

backend my_app
server s1 myapp.domain.com:80 resolvers mydns
server s2 myapp.domain.com:80 resolvers mydns

or can only have one record?

z0mb1ek · June 19, 2017, 1:55am

@Baptiste can you answer please?)

Baptiste · June 19, 2017, 9:23am

you need 2 lines. Actually, you need the number you expect to have.
Again, we’re working on improving this situation. Stay tuned.
(I mean that soon, the number of UP server will match the number of records in the DNS response, without any duplication unless you allow, for backward compatibility).

Baptiste

z0mb1ek · July 2, 2017, 12:52pm

@Baptiste big thx.

Can I use dev version in production?

boostrack · October 4, 2017, 10:47am

Hello list,

This are quite exciting news and I am trying to give it at shot with 1.8-dev2

So far i have created a resolvers entry on my haproxy.cfg and some basic frontend/backend

global
    log /dev/log daemon
    log /dev/log daemon notice
    maxconn 100

defaults
    mode http
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

listen stats
    bind *:1936
    stats enable
    stats hide-version
    stats refresh 5s
    stats show-node
    stats realm HAProxy\ Statistics
    stats uri /
    http-request set-log-level silent

resolvers dns-consul
  nameserver dns1 127.0.0.1:8600
  resolve_retries 3
  hold valid 100ms

frontend http
  bind *:80
  default_backend site-backend

backend site-backend
  balance roundrobin
  server site frontend-api-frontend.service.dc1.consul check resolvers dns-consul resolve-prefer ipv

I can see some messages of DNS requests in Consul (my dns srv source)

    2017/10/04 12:44:04 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 28 1} (139.981µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:04 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 1 1} (107.348µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:05 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 28 1} (121.594µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:05 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 1 1} (57.816µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:06 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 28 1} (124.103µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:06 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 1 1} (78.518µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:07 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 28 1} (128.503µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:07 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 1 1} (109.668µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:08 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 28 1} (190.55µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:08 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 1 1} (113.934µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:09 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 28 1} (123.446µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:09 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 1 1} (91.523µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:09 [DEBUG] http: Request GET /v1/agent/self (568.817µs) from=127.0.0.1:53393
    2017/10/04 12:44:10 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 28 1} (127.082µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:10 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 1 1} (108.02µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:11 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 28 1} (133.022µs) from client 127.0.0.1:63649 (udp)
    2017/10/04 12:44:11 [DEBUG] dns: request for {frontend-api-frontend.service.dc1.consul. 1 1} (97.165µs) from client 127.0.0.1:63649 (udp)

I’ve also verified that I see the list of servers on my dns request.

dig @127.0.0.1 -p 8600 frontend-api-frontend.service.consul SRV

; <<>> DiG 9.11.2 <<>> @127.0.0.1 -p 8600 frontend-api-frontend.service.consul SRV
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 750
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;frontend-api-frontend.service.consul. IN SRV

;; ANSWER SECTION:
frontend-api-frontend.service.consul. 0	IN SRV	1 1 21587 7f000001.addr.dc1.consul.
frontend-api-frontend.service.consul. 0	IN SRV	1 1 22242 7f000001.addr.dc1.consul.

;; ADDITIONAL SECTION:
7f000001.addr.dc1.consul. 0	IN	A	127.0.0.1
7f000001.addr.dc1.consul. 0	IN	A	127.0.0.1

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Wed Oct 04 12:44:55 CEST 2017
;; MSG SIZE  rcvd: 155

But i can not get the haproxy backend to be dynamic. Not sure what I am doing wrong and how to debug the haproxy itself.

So far starting it like this:

haproxy -f haproxy.cfg -d -V

But not errors messages or hints have spit into stdout.
Any helps or hints are appreciated.

Thanks in advance

Baptiste · October 5, 2017, 9:34pm

Hi,

You must use the latest -dev from git (code has been commited after dev2
has been released).
Second, your request is not sent as SRV since it does not follow-up the
RFC. The “fqdn” must be ...
Consul accepts the only (as kubernetes does), but it shouldn’t.
Last, you should use the srv-template directive to provision X server with
the same configuration.
It would be something like this:

server-template red 20 _http._tcp.red.default.svc.cluster.local:8080
inter 1s resolvers kube resolve-prefer ipv4 check

Baptiste

adnaan · October 13, 2017, 1:02pm

If anyone wants to try out 1.8-dev2 dns resolvers, I have setup a docker demo:

github.com

adnaan/haproxy-consul-service-discovery/blob/master/haproxy/haproxy.cfg

global
    maxconn 2000
    pidfile /var/run/haproxy.pid
    log     127.0.0.1 local0
    log     127.0.0.1 local1 notice

    # echo "" | nc -U /var/run/haproxy.sock
    stats   socket /var/run/haproxy.sock mode 777

resolvers consuldns
  nameserver dns 172.19.0.7:8600
  resolve_retries 3
#   timeout retry 1s
#   hold other 30s
#   hold refused 30s
#   hold nx 30s
#   hold timeout 30s
  hold valid 100ms

defaults

This file has been truncated. show original

You would probably need to reconfigure the hard coded IP addresses. Will get around to making it env based soon.

lukastribus · October 18, 2017, 2:58pm

Asking @willy to release 1.8-dev3, so that all those changes can be easier tested in the field.

jaschaio · May 4, 2018, 5:03am

Asking myself the same question.

From the Docker Docs:

To bypass the routing mesh, you can start a service using DNS Round Robin (DNSRR) mode, by setting the --endpoint-mode flag to dnsrr. You must run your own load balancer in front of the service. A DNS query for the service name on the Docker host returns a list of IP addresses for the nodes running the service. Configure your load balancer to consume this list and balance the traffic across the nodes.

How do I actually do that with HAProxy?

Baptiste · May 10, 2018, 6:21am

Well, either you set multiple server line with the same name:
backend myapp

[…]

server s1 myapp.domain.com:80 check resolvers mydns

server s2 myapp.domain.com:80 check resolvers mydns

server s3 myapp.domain.com:80 check resolvers mydns

Or you use server-template directive:

backend myapp

[…]

server s 3 myapp.domain.com:80 check resolvers mydns

Adjust the number of servers to your need.

In each case, the resolvers should turn on one server per IP found in the response.

danih · June 25, 2018, 5:39pm

Baptiste, I tried to use multiple servers but the requests are not balanced evenly between the servers.

I built an example using Docker and docker-compose. The backend servers count each request they receive and print the number of requests received after 60 seconds.

Code:

gist.github.com

https://gist.github.com/danihodovic/c89d685b95b1023fc16591b6e74d1134

app.js

const http = require('http');
const os = require('os');

let num = 0;

setTimeout(() => {
  console.log(os.hostname(), num);
  process.exit(0);
}, 60000);

This file has been truncated. show original

docker-compose.yml

---
version: '2'
services:
  haproxy:
    image: haproxy
    ports:
      - '80:80'
    volumes:
      - './haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg'

This file has been truncated. show original

haproxy.cfg

resolvers docker
  nameserver default 127.0.0.11:53
  hold valid 1000ms

frontend proxy
  mode http
  bind 0.0.0.0:80 maxconn 10000
  option http-server-close
  option forwardfor
  timeout client 5000

This file has been truncated. show original

The output is

api_4      | f8e1414ea551 0
api_1      | 860eb040651c 0
api_2      | 6af96d901ea8 179
api_5      | 1f15abd0d461 60
api_3      | 271ae04ff5cc 60

One API server receives 180 requests, two receive 60 each and another two receive 0.

Is it possible to round robin to servers which were resolved with DNS?

Baptiste · June 29, 2018, 8:34am

Did you start 5 instances of ‘api’ service?

danih · June 29, 2018, 8:52am

Yes. Five instances of the API service using.

docker-compose up --scale api=5

Baptiste · July 3, 2018, 9:33am

Hi,

You’re missing the “resolvers docker” statement on your server-template line.

With this enabled, I have the following result:

docker-compose up --scale api=5

Starting debug_api_1 …

Starting debug_api_1 … done

Starting debug_api_2 … done

Starting debug_api_3 … done

Starting debug_api_4 … done

Starting debug_api_5 … done

Attaching to debug_haproxy_1, debug_api_1, debug_api_2, debug_api_3, debug_api_4, debug_api_5

api_3 | cd51492adab9 63

api_4 | f5532b40ea80 61

debug_api_3 exited with code 0

debug_api_4 exited with code 0

api_2 | a81602129221 61

api_1 | f8202d903d1b 61

debug_api_2 exited with code 0

debug_api_1 exited with code 0

api_5 | 02f56990bbd4 62

debug_api_5 exited with code 0

danih · July 3, 2018, 4:13pm

You are correct @Baptiste. Apologies!

jamshid · July 22, 2018, 8:52pm

Thank you so much @z0mb1ek and @baptiste for explaining this I never would have guessed I need multiple server lines!

I’ve spent all weekend trying to figure out why haproxy wouldn’t load balance my docker service even though simple curls show it cycling among the different ip’s that my service name resolves to.

In my case I don’t know how large my service will be scaled. How do I know how many duplicate service lines to use? Is there any downside to using whatever maximum I expect?

What is the upcoming better way to do this is there a github issue I can follow or blog post explaining it?

supermathie · May 19, 2022, 8:38pm

I’ve right now implementing something similar on our infrastructure.

In our case, we have it pointing at an AWS ALB. Since those IPs can change, we don’t want it to hold onto that IP forever.

The final configuration looks something like this:

resolvers default
  parse-resolv-conf
  timeout resolve 1m

backend be_gw
  mode http
  http-request set-header host aws-gw.contoso.com
  option httpchk GET /srv/status HTTP/1.1
  http-check send hdr "host" "aws-gw.contoso.com"

  default-server init-addr none resolvers default check downinter 2s fastinter 2s inter 3m ssl ca-file /usr/local/etc/haproxy/ca-certificates.crt
  server-template gw4- 1-3 aws-gw.contoso.com:443 backup resolve-prefer ipv4
  server-template gw6- 1-3 aws-gw.contoso.com:443 resolve-prefer ipv6

Here’s the options applied to these servers and why:

init-addr none: the server starts off blank and then gets populated by the DNS lookup
resolvers default: use the “default” resolver group defined at the top of the file
ssl ca-file …: use TLS and validate with the specified CA file
resolve-opts prevent-dup-ip: one IP per server
backup: only use IPv4 as backup; this is the year of BOTH the Linux desktop and IPv6
resolve-prefer ipv[46]: use these addresses for these servers

The combination of these options gives us the following internal configuration:

'show servers state be_gw' | socat STDIO UNIX-CONNECT:/tmp/haproxy.sock
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord srv_use_ssl srv_check_port srv_check_addr srv_agent_addr srv_agent_port
5 be_gw 1 gw4-1 192.0.2.53 2 0 1 1 5 1 0 0 0 0 0 0 aws-gw.contoso.com 443 - 1 0 - - 0
5 be_gw 2 gw4-2 198.51.100.42 2 0 1 1 5 1 0 0 0 0 0 0 aws-gw.contoso.com 443 - 1 0 - - 0
5 be_gw 3 gw4-3 203.0.113.60 2 0 1 1 5 1 0 0 0 0 0 0 aws-gw.contoso.com 443 - 1 0 - - 0
5 be_gw 4 gw6-1 2001:db8:0:1:bf56:d653:4d5f:5254 2 0 1 1 5 1 0 0 0 0 0 0 aws-gw.contoso.com 443 - 1 0 - - 0
5 be_gw 5 gw6-2 2001:db8:0:2:37ce:beb1:dbd1:78a6 2 0 1 1 5 1 0 0 0 0 0 0 aws-gw.contoso.com 443 - 1 0 - - 0
5 be_gw 6 gw6-3 2001:db8:0:3:92b5:98b9:a514:9f8e 2 0 1 1 5 1 0 0 0 0 0 0 aws-gw.contoso.com 443 - 1 0 - - 0

Note that we’re using two separate sections for IPv4 and IPv6 addresses - this is due to resolve-prefer (default: ipv6) causing only IPv6 addresses to be used (tested on 2.5.7)