Lua HttpClient - load balance using DNS multiple A records

Help!
1

Hello all, hope you’re well.

Here is my context:
I have a Lua script that registers an action on http_req, and make a HTTP call to a third party on each incoming request.
This works fine.

The issue is that the third-party host I’m reaching has multiple IPs defined in the DNS record, and I need to distribute the load of incoming requests to all those IP.
If I don’t, the load will is too large for one single LB of the third-party.

By default, as the DNS is resolved only once at startup, and the connections are kept open, my instance of haproxy sticks to one single IP, and overloads it, resulting in timeouts to get a response.

How can I properly load-balance between differents IP ?
I tried to use socket.dns.toip("third-party-host.com") to get a different IP to use, but this generates a certificate issue (using https) - that can not be disabled with the current httpclient implementation.

I’ve tried multiple things, but the basic problem is how to properly use multiple A records for the same FQDN in Lua Http client ?

All ideas are welcome !
Thanks a lot !

2

I wouldn’t recommend using socket.* API because it is probably leveraging blocking functions which are not compatible with haproxy Lua’s event driven engine. When using external libraries, you should ensure that functions you rely on don’t wait for events and give the hand back to haproxy, see: https://www.arpalert.org/haproxy-lua.html#h204

By default, as the DNS is resolved only once at startup, and the connections are kept open, my instance of haproxy sticks to one single IP, and overloads it, resulting in timeouts to get a response.

If an explicit “resolvers” section is defined within haproxy, and Lua httpclient makes use of it, runtime resolution will be supported, so if the record is updated for a given DNS it should work as expected:
https://docs.haproxy.org/dev/configuration.html#resolvers%20(The%20resolvers%20section)
https://docs.haproxy.org/dev/configuration.html#httpclient.resolvers.id

However if multiple records are returned for the same DNS at a given time, haproxy will only consider the first returned entry. So you may force it to renew it’s cache by playing with the “timeout resolve” from resolvers section, but it may not suit your needs.

In this case, if you can’t deal with the builtin resolver’s behavior, and if you are able to query the records for a given DNS through an HTTP webservice at a defined address for example, you could leverage that webservice using the lua httpclient (or leverage core.tcp for a TCP service), and then iterate over raw IPs (from Lua) to make the actual httpclient request the server with the expected A record (and fallback to other IPs until one request succeeds for instance).

Another solution is to define another listener/proxy in haproxy and use DNS autodiscovery (with server-template), so that HAProxy discovers all IPs for a given DNS entry. You may then choose “roundrobin” load-balancing policy for that backend and leverage this “local” proxy directly from Lua’s httpclient (ie: forward requests to local listener which performs the discovery and load-balancing instead of requesting using the endpoint DNS from Lua directly). See: DNS for Service Discovery in HAProxy

1 Like
3

Thanks a lot for your answer.

The solution lies in using haproxy dns discovery and server-template.
This is better than using some hazardous lua implementation.

For reference, here is the config i’m validating:
haproxy.cfg:

resolvers mydns
  nameserver ns1 x.x.x.x:53
  
frontend my-third-party-api-unix
  mode tcp
  bind unix@/tmp/my-third-party-api-unix.sock
  default_backend my-third-party-api
  
backend my-third-party-api
  mode tcp
  server-template my-third-party-api1 10 third-party.com:443 check resolvers mydns

and in lua, using the dst param:

local res = httpclient:post { url, headers,  body, timeout,
    dst="unix@/tmp/my-third-party-api-unix.sock"
  }
1 Like