I have multiple backends for a service that should have all traffic going to a single server. I want connections to be killed when the active server goes down (handles itself), but HAProxy to kill the connections when it comes back up.
It sounds like ‘no-marked-up shutdown-backup-sessions’ is what I’m after, but this doesn’t seem to work when you have multiple backup servers.
Our standard config has all the servers marked as backup, and allbackups disabled. This correctly causes all traffic to go to the first server in the list (what we want).
If we shutdown that server (redis1 in the example), traffic goes to the second server in the list (redis2).
However, if the first server comes back online (docker-compose start redis1), connections do not get killed.
The setup works if redis1 is not marked as a backup, but then it doesn’t work when redis2 fails over to redis3.
Example config:
global
stats socket /var/run/api.sock user haproxy group haproxy mode 660 level admin expose-fd listeners
log stdout format raw local0 info
defaults
mode http
timeout client 10s
timeout connect 5s
timeout server 10s
timeout http-request 10s
log global
frontend stats
bind *:1337
stats enable
stats uri /
stats refresh 10s
listen redis
bind *:6379
balance roundrobin
maxconn 20000
mode tcp
no log
no option allbackups
option redispatch
option tcp-check
option dontlog-normal
option tcpka
option clitcpka
tcp-check send "PING\r\n"
tcp-check expect string "+PONG"
tcp-check send "QUIT\r\n"
tcp-check expect string "+OK"
timeout connect 4s
timeout server 330s
timeout client 330s
server redis1 redis1:6379 backup check on-marked-up shutdown-backup-sessions on-marked-down shutdown-sessions
server redis2 redis2:6379 backup check on-marked-up shutdown-backup-sessions on-marked-down shutdown-sessions
server redis3 redis3:6379 backup check on-marked-up shutdown-backup-sessions on-marked-down shutdown-sessions
docker-compose setup
services:
redis1:
image: eqalpha/keydb:x86_64_v6.3.1
redis2:
image: eqalpha/keydb:x86_64_v6.3.1
redis3:
image: eqalpha/keydb:x86_64_v6.3.1
haproxy:
image: haproxytech/haproxy-alpine:2.6
ports:
- 6379:6379
- 1337:1337
volumes:
- .:/usr/local/etc/haproxy:ro
Demo:
docker-compose up -d
redis-cli monitor (this will show pings fine)
docker-compose stop redis1
# redis-cli disconnects (as expected)
redis-cli monitor (start it up again, it'll be pinging)
docker-compose start redis1
Expected behaviour is that redis-cli disconnects.
Actual behaviour is redis-cli stays connected to the redis2, ignoring the ‘on-marked-up shutdown-backup-sessions’.