HAProxy 2.4 with stick-tables on swarm

Help!
1

Hi,
I’m trying to setup HAProxy on Docker Swarm (the container is haproxytech/haproxy-debian:2.4 ) and the deploy mode is global (3 replicas, as I have 3 workers) with the network mode as host, so I am avoiding the docker load balancer and I’m sure which instance of the HAProxy container is forwarding the http traffic.

Behind the HAProxy there is a replicated service which requires the session persistence, so by googling up I came to the conclusion that the approach I should use is to have a stick table where I should track the cookie and redirect the connection to the right server.

As HAProxy runs on a container, I created 3 (almost identical) config files, where the only change is the line where the local peer (node#) has its listening IP to 127.0.0.1 instead of the VM IP address.

I’m providing the configuration file here:

global
    log          fd@2 local2 debug
    chroot       /var/lib/haproxy
    pidfile      /var/run/haproxy.pid
    maxconn      4000
    user         haproxy
    group        haproxy
    stats socket /var/lib/haproxy/stats expose-fd listeners
    master-worker

resolvers docker
    nameserver dns1 127.0.0.11:53
    resolve_retries 3
    timeout resolve 1s
    timeout retry   1s
    hold other      10s
    hold refused    10s
    hold nx         10s
    hold timeout    10s
    hold valid      10s
    hold obsolete   10s

defaults
    timeout connect 10s
    timeout client 30s
    timeout server 30s
    log global
    mode http
    option httplog

peers mypeers
    peer node1 127.0.0.1:2000
    peer node2 <node2 IP>:2000
    peer node3 <node3 IP>:2000

frontend  fe_web
    bind *:80
    use_backend stat if { path -i /my-stats }
    default_backend configurator

backend configurator
    balance roundrobin
    option persist
    option redispatch
    server-template configurator- 6 <service name>:443 check resolvers docker init-addr libc,none
    cookie SERVERID insert indirect nocache maxidle 30m maxlife 8h
    stick-table type string len 32 size 100k expire 1h  peers mypeers srvkey addr
    stick store-response res.cook(PHPSESSID)
    stick on req.cook(PHPSESSID)
    tcp-request content track-sc0 req.cook(PHPSESSID)
    http-request track-sc0 src


backend stat
    stats enable
    stats uri /my-stats
    stats refresh 15s
    stats show-legends
    stats show-node

The service is run with the endpoint_mode: dnsrr option, so HAProxy can get all instances, like described in the guide for swarm load balancing . The first issue is that docker gives a list of IP in a random order, so the stick-table over the 3 nodes is not consistent over the server_id key.

Two possible solutions came to my mind:

  1. Install HAProxy on the VM and, via systemd, ensure that it gets restarted in case of any issue. The problem with this solution is that the HAProxy instance on each node is not able to resolv the containers name with the Docker DNS server, as the latter is available only inside the docker overlay network. So, I discarded this option as it seemed unfeasable to solve my problem
  2. Use the srv_key addr option on the stick table. With this solution, the stick-table contains correctly the Cookie as the value for the key, the server_id field and the server_key one:
# table: configurator, type: string, size:102400, used:2
0x556b582e4f20: key=10.2.0.198 use=0 exp=3533310 server_id=0 server_key=-
0x556b582f34f0: key=6qql68sgo3q33k9qfdesud9a7i use=0 exp=3582937 server_id=1 server_key=172.0.10.73:80

The first line shows up when I’m at the login page, and the second one when I’m successfully logged in.
If I use the same HAProxy, there is no problem, but if I change the HAProxy instance (giving the same PHPSESSID on the request to the other HAProxy container), the session persistence is not mantained, as data is routed to the server with id 1 for the second HAProxy instance.
The solution shown is to associate the server name with an IP, but this is not possible with docker swarm, as the IP address is changed once a container is recreated.

i.e. what described here is not doable:

[...]
    server lamp1 10.0.0.1:80 check
    server lamp2 10.0.0.2:80 check

So, going to the real question: is there a way to tell HAProxy to use what is stored with srvkey addr as the server address where to redirect the traffic?

Thanks in advance