Dynamic source address

Help!
1

Hi all!

For avoiding the problem of port exhaustion when connecting to upstream servers I have configured several IPs in my HAProxy machine. My intention is HAProxy to use those IPs as source addresses when connecting to the upstream servers (outgoing connections).

For selecting the source address for outgoing connections I have found the “source” option. The problem is that there is not an easy way of using several source addresses using this option.

The only way I have found for doing this is to duplicate the server definitions, using a different source address for each of them. For example, for 2 real servers (app1 and app2) and 3 outgoing IPs:

backend my-backend
        balance leastconn
        server app1_src1 app1.test.com source 192.168.1.1
        server app1_src2 app1.test.com source 192.168.1.2
        server app1_src3 app1.test.com source 192.168.1.3
        server app2_src1 app2.test.com source 192.168.1.1
        server app2_src2 app2.test.com source 192.168.1.2
        server app2_src3 app2.test.com source 192.168.1.3

As you can see, I have had to create 6 “virtual” servers in my backend, one for each application-sourceIP combination.

Although this solutions works, it is not ideal, as we are not balancing the load properly between the two real servers, but on the 6 “virtual/artificial” servers, combinations of real server and source IP. Also, disabling a specific real server from the backend is more complex, because we need to disable all the possible combinations of that server.

I have to say that this can be implemented in Nginx in a very elegant way. For example:

http {
    upstream backend {
        server app1.test.com:80;
        server app2.test.com:80;
    }

    server {
        location / {
            proxy_pass http://backend;
            proxy_bind $split_ip;
        }
    }

    split_clients "$remote_addr$remote_port" $split_ip {
        33%  192.168.1.1;
        33%  192.168.1.2;
        *    192.168.1.3;
    }
}

For implementing something like that in HAProxy we would need to use dynamic variables or converters (maps) in the “source” option. For example:

backend my-backend
        balance leastconn
        source %[rand(3),map(myips.map)]
        server app1 app1.test.com
        server app2 app2.test.com

Unfortunately, it seems that the “source” option does not expand a logformat string, so we cannot use variables or converters with it.

Is there a better way of doing this?

Regards.

2

You are right, haproxy does not have an abstraction like nginx has. I doubt it ever will.

Deployments where this is really needed are very rare and abstracting away some configuration lines is usually not a problem in those specific deployments. This is basically a cosmetic issue.

Before going down this road I really suggest you take a look at why you are in this situation in the first place, and analyze that situation.

The source keyword in haproxy or the proxy_bind keyword in nginx usually make source port exhaustion worse, unless both the haproxy/nginx as well as the libc and the linux kernel support and enable IP_BIND_ADDRESS_NO_PORT.

If any one of those components in your setup is not at least at the indicated version number, you will actually actively cause source port exhaustion because of the fact that you are binding the socket to a specific source IP:

  • linux kernel 4.2
  • libc 2.23
  • nginx 1.11.4
  • haproxy 1.7

Also all this problems go away if the server closes first.

3

Hi @lukastribus !

Thanks for replying, I really appreciate it.

My use case is that I want to maximise the number of simultaneous connections between HAProxy and each of the upstream servers. I have already increase the range of ephemeral ports, and I was also trying to implement the dynamic binding of source addresses.

Thanks for pointing out the necessity of the IP_BIND_ADDRESS_NO_PORT option. It seems there is a problem with that option in the operating system I’m using (CentOS 7.7). Is there any way of checking if the option is configured/enabled/working in my HAProxy package? I don’t see the option when I enter: haproxy -vv

Best regards.

4

You’d have to run haproxy through strace -tt and look the calltrace when establishing a backend connections.

It’s possibile that this was backported to CentOs 7, but to confirm a calltrace of a backend connection establishing is needed.

FWIW it should be backported in RHEL/CentOs 7, but haproxy also has to have been compiled with the libc update, before it will use it. You should therefor strace it, to make sure.