Resurrecting backend servers with health checks

Help!
1

I have the following haproxy config

global
    daemon
    maxconn 256

defaults
    mode http
    timeout connect 4s
    timeout client 30s
    timeout server 30s
    retry-on all-retryable-errors

frontend http
    bind :8080
    http-request use-service prometheus-exporter if { path /metrics }
    default_backend stats

backend stats
    mode http
    stats enable
    stats uri /
    stats refresh 5s
    stats show-legends
    stats admin if TRUE

resolvers k8s
    parse-resolv-conf
    hold other           10s
    hold refused         10s
    hold nx              10s
    hold timeout         10s
    hold valid           10s
    hold obsolete        10s

frontend redis-read
    bind *:6380
    default_backend redis-online

frontend redis-write
    bind *:6379
    default_backend redis-primary

frontend redis-sentinel
    bind *:26379
    default_backend redis-sentinel

backend redis-primary
    mode http
    balance first
    option tcp-check
    tcp-check connect port 6379
    tcp-check send info\ replication\r\n
    tcp-check expect string role:master
    server redis-0 redis-node-0.redis-headless.dbms.svc.cluster.local check inter 5s
    server redis-1 redis-node-1.redis-headless.dbms.svc.cluster.local check inter 5s
    server redis-2 redis-node-2.redis-headless.dbms.svc.cluster.local check inter 5s

backend redis-online
    mode http
    balance roundrobin
    option tcp-check
    tcp-check connect port 6379
    tcp-check send PING\r\n
    tcp-check expect string +PONG
    server redis-0 redis-node-0.redis-headless.dbms.svc.cluster.local check inter 5s
    server redis-1 redis-node-1.redis-headless.dbms.svc.cluster.local check inter 5s
    server redis-2 redis-node-2.redis-headless.dbms.svc.cluster.local check inter 5s

backend redis-sentinel
    mode http
    balance roundrobin
    option tcp-check
    tcp-check connect port 26379
    tcp-check send PING\r\n
    tcp-check expect string +PONG
    server redis-0 redis-node-0.redis-headless.dbms.svc.cluster.local check inter 5s
    server redis-1 redis-node-1.redis-headless.dbms.svc.cluster.local check inter 5s
    server redis-2 redis-node-2.redis-headless.dbms.svc.cluster.local check inter 5s

If all three of the servers go down, haproxy indicates that all are down and backend has no servers available.

However, if the servers come back, haproxy never discovers this and I have to restart haproxy.

As far as I can tell, it should be retrying after 5s to see if the server is back (inter time) but isn’t.

Ideas on how to get haproxy to discover that the servers are back?

2

I think I figured it out - haproxy was caching the IP of each of the servers and when the pods / service came back up, it had different IPs. Added a resolvers section to flush the cache