Flapping status after having disable server

Help!
1

Hello,

After have put a server in maintenance state, after some http requests, the status change to MAINT to UP, like flapping.

I Put my server in maintenance with

 
echo "disable server bk_eol/eolcache4" | socat unix-connect://run/haproxy/admin.sock stdio

After, i verify

while true ; do sleep 5 ; echo "show stat" | socat unix-connect://run/haproxy/admin.sock stdio | grep "eolcache4" ; done
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT,1,1,0,0,1,124,124,,2,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT,1,1,0,0,1,129,129,,2,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,231,0,,3,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT,1,1,0,0,1,139,139,,2,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,241,0,,3,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,246,0,,4,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,251,0,,4,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,256,0,,4,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,UP,1,1,0,0,0,261,0,,3,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,
bk_eol,eolcache4,0,0,0,0,,0,0,0,,0,,0,0,0,0,MAINT,1,1,0,0,1,169,169,,2,3,1,,0,,2,0,,0,L4OK,,0,0,0,0,0,0,0,0,,,,0,0,,,,,-1,,,0,0,0,0,

Why the server don’t stay in MAINT state ? sometime it take the UP status !! it’s dangerous

Here is the log

Server bk_eol/eolcache4 is going DOWN for maintenance. 0 active and 1 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue

Here is my configuration

global 
        log 127.0.0.1   syslog debug
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 1m
        stats bind-process 4
        user haproxy
        group haproxy
        daemon
        nbproc 4
        maxconn 8192
        #ulimit-n 16416
        ulimit-n 16419
defaults
        log     global
        mode    http
        option  httplog
        timeout connect 10s
        timeout client  1m
        timeout server  1m
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
        # close connections that are not closed properly, avoiding connection living till timeout
        #option httpclose
frontend fo_eol
    bind *:80
    option          forwardfor
    mode http
    default_backend bk_eol
backend bk_eol
    mode http
    server  eolcache4 127.0.0.1:82 check inter 5s
    server  maintenance1 maintenance1:80 check backup inter 5s

And my version

haproxy -v
HA-Proxy version 1.5.8 2014/10/31
Copyright 2000-2014 Willy Tarreau 

eolcache4:/etc/haproxy# lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 7.9 (wheezy)
Release:        7.9
Codename:       wheezy

thanks in advence !!!

Ronron

2

You are using nbproc 4, meaning you have 4 haproxy processes, but you only have 1 stats socket for one process.

So, you are disabling the backend for 1 random haproxy process only, and than querying random haproxy processes about the status of the backend. When you accidentally hit the same process where you disabled haproxy, you will see MAINT, when you hit the other one you will see UP.

If you don’t need nbproc, I suggest you disable it an use one process only (if you don’t have SSL, chances are you don’t need it).

Otherwise I suggest you drop stats bind-process, and configure 4 specific stats socket for each process:

global 
        stats socket /run/haproxy/admin.sock-proc1 mode 660 level admin process 1
        stats socket /run/haproxy/admin.sock-proc2 mode 660 level admin process 2
        stats socket /run/haproxy/admin.sock-proc3 mode 660 level admin process 3
        stats socket /run/haproxy/admin.sock-proc4 mode 660 level admin process 4
3

Lukastribus,

Very fast and good response !!!

Ok, i understand better the random side, thank you :slight_smile:
Unfortunaly, i must use SSL. ok for your solution based on the multiple “stats socket”.

Ronron