HAProxy community

Backup keyword not working correctly

I’m setting up a localhost backup webserver for specific content in the event all primaries are down. When all the primaries are down, the backup isn’t switched to and a 503 is displayed instead.

If i remove the keyword backup, and then take down the other primary, the content that is expected is displayed. I can’t figure out why for the life of me.

backend default
balance leastconn
option httpchk
server app1 ssl cookie app1 weight 10 check verify none on-marked-up shutdown-backup-sessions
server app2 ssl cookie app2 weight 10 check verify none on-marked-up shutdown-backup-sessions
server backup localhost:8080 backup check port 8080

is there something wrong with the way the keyword is being used?

Provide the full configuration, the output of haproxy -vv and most importantly: the haproxy logs.

HA-Proxy version 1.8.23 2019/11/25
Copyright 2000-2019 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-null-dereference -Wno-unused-label -Wno-stringop-overflow
  OPTIONS = USE_LINUX_TPROXY=1 USE_CRYPT_H=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.1c FIPS  28 May 2019
Running on OpenSSL version : OpenSSL 1.1.1c FIPS  28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
    [SPOE] spoe
    [COMP] compression
    [TRACE] trace

and the config file

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2 debug

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon


    ## Security settings
    # Use more bits for Diffie-Hellman exchange
    tune.ssl.default-dh-param 2048
    # Use only recent versions of TLS
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    # Use only recent ciphers
    ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP
#    ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256

 # turn on stats unix socket
    stats socket /var/lib/haproxy/stats level admin
    # We would like this but it is only supported in newer versions of HAProxy.
    server-state-file /var/lib/haproxy/state

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch 3
    retries                 4
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 10000

#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend  main
    bind *:80
    bind *:443 ssl crt /etc/pki/tls/certs/star.pem
    redirect scheme https if !{ ssl_fc }
    #acl url_static       path_beg       -i /static /images /javascript /stylesheets
    #acl url_static       path_end       -i .jpg .gif .png .css .js

    acl host_wwwt hdr(host) -i www-test.site.com

    # Test fail over using number of available servers
    #acl host_wwwt-health_fail nbsrv(wwwt) lt 1
    #monitor-uri /
    #monitor fail if host_wwwt-health_fail
    #use_backend wwwt_failover if host_wwwt host_wwwt-health_fail
    #use_backend wwwt_failover if host_wwwt-health_fail
    use_backend wwwt if host_wwwt

    #    use_backend accounts if host_ba_prod

    default_backend  www

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
#backend static
 #   balance     roundrobin
  #  server      static 127.0.0.1:4331 check

backend wwwt
    balance leastconn
    #errorfile 503 /etc/haproxy/errorfiles/503.http
    option httpchk
    server app1 192.168.0.89 ssl cookie app1 weight 10 check verify none on-marked-up shutdown-backup-sessions
    server backup1 localhost:8080 backup check port 8080

and the log file

Jun  9 09:00:06 localhost haproxy[417741]: Proxy main started.
Jun  9 09:00:06 localhost haproxy[417741]: Proxy wwwt started.
Jun  9 09:00:06 localhost haproxy[417741]: Proxy wwwt_failover started.
Jun  9 09:00:06 localhost haproxy[417741]: Proxy www started.
Jun  9 09:00:26 localhost haproxy[417742]: 172.17.0.125:54256 [09/Jun/2020:09:00:26.410] main~ wwwt/app1 0/0/5/66/71 302 344 - - ---- 2/2/0/0/0 0/0 "GET / HTTP/1.1"
Jun  9 09:01:09 localhost haproxy[417819]: Proxy main started.
Jun  9 09:01:09 localhost haproxy[417819]: Proxy wwwt started.
Jun  9 09:01:09 localhost haproxy[417819]: Proxy wwwt_failover started.
Jun  9 09:01:09 localhost haproxy[417819]: Proxy www started.
Jun  9 09:04:56 localhost haproxy[1873]: Proxy main started.
Jun  9 09:04:56 localhost haproxy[1873]: Proxy wwwt started.
Jun  9 09:04:56 localhost haproxy[1873]: Proxy wwwt_failover started.
Jun  9 09:04:56 localhost haproxy[1873]: Proxy www started.
Jun  9 09:04:56 localhost haproxy[1874]: Backup Server wwwt/backup1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun  9 09:04:57 localhost haproxy[1874]: Server wwwt_failover/backup is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun  9 09:04:57 localhost haproxy[1874]: backend wwwt_failover has no server available!
Jun  9 09:05:39 localhost haproxy[1874]: 172.17.0.125:54436 [09/Jun/2020:09:05:39.498] main~ wwwt/app1 0/0/5/65/70 302 344 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
Jun  9 09:06:09 localhost haproxy[1874]: 172.17.0.125:54476 [09/Jun/2020:09:06:05.662] main~ wwwt/app1 0/3004/-1/-1/4006 503 212 - - SC-- 2/2/0/0/4 0/0 "GET / HTTP/1.1"
Jun  9 09:06:13 localhost haproxy[1874]: 172.17.0.125:54477 [09/Jun/2020:09:06:09.748] main~ wwwt/app1 0/3004/-1/-1/4006 503 212 - - SC-- 1/1/0/0/4 0/0 "GET /favicon.ico HTTP/1.1"
Jun  9 09:06:19 localhost haproxy[1874]: 172.17.0.125:54485 [09/Jun/2020:09:06:15.678] main~ wwwt/app1 0/3004/-1/-1/4007 503 212 - - SC-- 2/2/0/0/4 0/0 "GET / HTTP/1.1"
Jun  9 09:06:23 localhost haproxy[1874]: 172.17.0.125:54487 [09/Jun/2020:09:06:19.796] main~ wwwt/app1 0/3003/-1/-1/4006 503 212 - - SC-- 1/1/0/0/4 0/0 "GET /favicon.ico HTTP/1.1"

here is the log file after removing just the backup key word.

Jun 18 11:35:41 localhost haproxy[657179]: Proxy main started.
Jun 18 11:35:41 localhost haproxy[657179]: Proxy wwwt started.
Jun 18 11:35:41 localhost haproxy[657179]: Proxy wwwt_failover started.
Jun 18 11:35:41 localhost haproxy[657179]: Proxy www started.
Jun 18 11:36:05 localhost haproxy[657180]: 100.14.14.193:40734 [18/Jun/2020:11:36:05.233] main/2: SSL handshake failure
Jun 18 11:36:05 localhost haproxy[657180]: 100.14.14.193:40742 [18/Jun/2020:11:36:05.302] main/2: SSL handshake failure
Jun 18 11:36:05 localhost haproxy[657180]: 100.14.14.193:40744 [18/Jun/2020:11:36:05.394] main/2: SSL handshake failure
Jun 18 11:36:05 localhost haproxy[657180]: 100.14.14.193:40740 [18/Jun/2020:11:36:05.462] main/2: SSL handshake failure
Jun 18 11:36:09 localhost haproxy[657180]: 100.14.14.193:40746 [18/Jun/2020:11:36:09.961] main/2: SSL handshake failure
Jun 18 11:36:09 localhost haproxy[657180]: 100.14.14.193:40748 [18/Jun/2020:11:36:09.961] main/2: SSL handshake failure
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40754 [18/Jun/2020:11:36:10.409] main/2: SSL handshake failure
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40752 [18/Jun/2020:11:36:10.413] main/2: SSL handshake failure
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40756 [18/Jun/2020:11:36:10.465] main~ wwwt/backup1 0/0/1/1/2 200 20018 - - ---- 2/2/1/0/0 0/0 "GET / HTTP/1.1"
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40756 [18/Jun/2020:11:36:10.674] main~ wwwt/backup1 0/0/1/1/2 200 8202 - - ---- 2/2/1/0/0 0/0 "GET /images/reverse-logo-recovery-footer.png HTTP/1.1"
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40756 [18/Jun/2020:11:36:10.930] main~ wwwt/backup1 0/0/0/0/1 404 410 - - ---- 2/2/1/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40764 [18/Jun/2020:11:36:10.933] main/2: SSL handshake failure
Jun 18 11:36:13 localhost haproxy[657180]: 100.14.14.193:40750 [18/Jun/2020:11:36:10.019] main~ wwwt/backup1 0/3006/0/1/3007 200 20018 - - ---- 2/2/0/0/+3 0/0 "GET / HTTP/1.1"
Jun 18 11:36:24 localhost haproxy[657180]: 100.14.14.193:40768 [18/Jun/2020:11:36:23.994] main/2: SSL handshake failure
Jun 18 11:36:24 localhost haproxy[657180]: 100.14.14.193:40766 [18/Jun/2020:11:36:23.993] main/2: SSL handshake failure
Jun 18 11:36:27 localhost haproxy[657180]: 100.14.14.193:40770 [18/Jun/2020:11:36:24.047] main~ wwwt/backup1 0/3007/0/1/3008 200 20018 - - ---- 1/1/0/0/+3 0/0 "GET / HTTP/1.1"
Jun 18 11:36:27 localhost haproxy[657180]: 100.14.14.193:40776 [18/Jun/2020:11:36:27.302] main/2: SSL handshake failure

I don’t see how health checks are supposed to work in the first place, you did not specify any port for your primary server, in fact haproxy warns about it:

[ALERT] 169/182923 (24505) : parsing […/cert/mcdonaldr-backup-issue.cfg:100] : server app1 has neither service port nor check port nor tcp_check rule ‘connect’ with port information. Check has been disabled.

What exactly are you doing do simulate the primary server not working? Do you actually specify the port in the real configuration?

for simulating the failure, I am just stopping httpd on the backend server “app1”.

I did not have the check port 443 on the primary; added it and am now testing it.

changing the “check” to “check port 80” has made it work.

Without the check port 80, the backend (app1) site was being displayed if it was up. Is that normal, or was i being silly?

What you need to do is specify the port in general, not only for health checks. Same for the backup server.

<ip>:<port> instead of <ip> check port <port>

Try:

server app1 192.168.0.89:443 ssl cookie app1 weight 10 check verify none on-marked-up shutdown-backup-sessions

You configured:

check port 80

which means port 443 is no longer health check, which is not what you want since you are actually using port 443 for your real traffic.