We are running into a snag in our deployment process. While removing servers from rotation, persistent connections are dropped. We are currently using cookie based persistence. We’d like the connections to gracefully move to another server, instead of being dropped.
Here is what we are doing to remove a server from rotation:
- Change the state of the server to DRAIN (via socat command). This command disallows any new connections to be on the server, however persistent connections are still hitting our server.
- Change “health.html” contents to “DOWN”. This marks the server as “DOWN”, but all connections are dropped and users bounced to another server.
We are unable to determine the step we are missing between #1 & #2. We have tried the following:
- Incorporating the “MAINT” status
- Setting the maxconn value on a server to -1
- Renaming the “health.html” file instead of changing the contents. This causes the server to be marked as “NOLB”
Does anyone have any suggestions?
Below is the HA Proxy config:
global
maxconn 30000
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
nbthread 48
tune.bufsize 32768
tune.ssl.cachesize 30000
tune.ssl.lifetime 600
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
ssl-default-bind-options no-sslv3
stats socket ipv4@127.0.0.1:9999 level admin
stats socket /var/run/haproxy.sock mode 666 level admin
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 121000
timeout client 121000
timeout server 121000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
frontend fe_main
bind :80
bind :443 ssl crt /etc/cc-ssl/[redacted].pem crt /etc/cc-ssl/[redacted].pem
reqadd X-Forwarded-Proto:\ https
http-request redirect scheme https unless { ssl_fc }
default_backend be-https
frontend stats
bind *:8404
stats enable
stats uri /stats
backend be-https
balance roundrobin
cookie NUMID insert indirect nocache
option httpchk GET /health.html HTTP/1.1\r\nHost:\ www
http-check disable-on-404
http-check expect string UP
default-server inter 3s fall 2 rise 2 slowstart 5m
server s1 10.10.10.1:443 ssl verify none check cookie 1
server s2 10.10.10.2:443 ssl verify none check cookie 2
server s3 10.10.10.3:443 ssl verify none check cookie 3
server s4 10.10.10.4:443 ssl verify none check cookie 4