Haproxy doesn't resolve a domain to all the IPs

Help!
1

I’ve created a Docker Compose project with haproxy and 4 replicas of a web server. I’m using server-template to connect haproxy to the replicas:

server-template s 4 app:80 check

But:

$ docker compose exec haproxy host -t a app
app has address 192.168.48.5
app has address 192.168.48.4
app has address 192.168.48.3
app has address 192.168.48.2

$ echo show servers state \
    | docker compose exec -T haproxy socat /var/lib/haproxy/haproxy.sock stdio
1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord srv_use_ssl srv_check_port srv_check_addr srv_agent_addr srv_agent_port
2 http 1 s1 192.168.48.2 2 0 1 1 21 15 3 4 6 0 0 0 app 80 - 0 0 - - 0
2 http 2 s2 192.168.48.2 2 0 1 1 21 15 3 4 6 0 0 0 app 80 - 0 0 - - 0
2 http 3 s3 192.168.48.3 2 0 1 1 21 15 3 4 6 0 0 0 app 80 - 0 0 - - 0
2 http 4 s4 192.168.48.5 2 0 1 1 21 15 3 4 6 0 0 0 app 80 - 0 0 - - 0

Sometimes there are no duplicates, but usually there’s a couple of them.

I thought that there’s some sort of race condition here (when haproxy starts the DNS resolver responds with only some IPs), but the behavior is identical after reload:

$ docker compose exec haproxy kill -HUP 1

What am I doing wrong? How do I make haproxy connect to all the servers?

On a side note, set server http/s1 state ready always returns “No such server.” But the command works with http/s2, http/s3, http/s4. Anything wrong on my part here?

2

hello, if you have a resolvers section, try adding accepted_payload_size 8192 because the default of 512 bytes may be too small to fit more SRV records to populate your template.

See here how to create a resolvers section if you don’t have one HAProxy version 2.8.10 - Configuration Manual

good luck

1 Like
3

In the OP I gave a link to a minimal reproducible example (GitHub gist). And yes, I don’t have the resolvers section. The reason I think it has nothing to do with accepted_payload_size is because it sometimes resolves to all 4 IPs. It’s just that usually it doesn’t. I tried it with 2 and 3 IPs, and it still doesn’t always resolve to all of them. And I don’t use SRV records. Can’t it be solved without a resolvers section? What resolvers section corresponds to the default settings? To be frank it potentially looks like a bug. At least I fail to understand what going on here.

4

The libc resolution is not designed for this. It will resolve the hostnames 4 times and use the first address in the answer. As such in this particular configuration you get completely random behaviour. You could have 4 times the same IP.

You need to switch to resolvers and disable libc resolution. It’s 2 lines after the global section like this:

defaults
 default-server init-addr last,none resolvers default

Or if you want a dedicate section:

resolvers mydns
 parse-resolv-conf
 accepted_payload_size 8192

defaults
 default-server init-addr last,none resolvers mydns
2 Likes
5

I wrote my answer before I saw the @lukastribus reply so I guess I post it as is:

Okay, without the resolvers section haproxy calls getaddrinfo() 4 times, takes the first address and with luck obtains 4 different IPs. I’m not sure why it doesn’t call it once, because getaddrinfo() returns all the 4 IP addresses. There’s probably some reason, but it escapes me.

With a resolvers section:

    server-template s 4 app:80 check resolvers docker
    ...

resolvers docker
    parse-resolv-conf

it starts to resolve the hostname continuously and in this case it succeeds. Unless allow-dup-ip is set. Which is by default not. It succeeds because it doesn’t let duplicates in this case.

6

That’s right, the configuration to do service discovery via DNS must use resolvers without allow-dup-ip.

7

I guess I’m starting to understand. init-addr is responsible for initial resolution that happens during initialization. And if I attach resolvers it starts to periodically reresolve the domain (at run time).

By default init-addr is last,libc and none if resolvers are set.

And I like your “trick” about using the default section.

In my case I’d prefer to define it inline though:

server-template s 4 app:80 check init-addr libc,none resolvers docker

Since at the moment I have only one listen section. But not that it matters much.

Don’t you know by any chance why libc resolution can’t resolve the domain once and use all the IPs? I think that’s more expected and can be useful. To me the libc method looks similar to what the default resolvers section does. I’m not sure if the latter uses getaddrinfo(), but at least they use the same DNS server.

8

libc resolution is was always supported, however it is a synchronous API so it cannot be used during runtime (because haproxy is a event-loop based application).

Usually people doing service discovery via DNS also need runtime updates, it’s not enough to just resolve the IPs once at startup.

Therefor to allow for service discovery via DNS a internal resolver was written that supports all this.

libc resolution itself was never really upgraded to support multiple answers, because we already have the resolver for this, and like I said it’s very rare that someone needing service discovery is ok with just resolving DNS records once.