I’m setting up a localhost backup webserver for specific content in the event all primaries are down. When all the primaries are down, the backup isn’t switched to and a 503 is displayed instead.
If i remove the keyword backup, and then take down the other primary, the content that is expected is displayed. I can’t figure out why for the life of me.
backend default
balance leastconn
option httpchk
server app1 ssl cookie app1 weight 10 check verify none on-marked-up shutdown-backup-sessions
server app2 ssl cookie app2 weight 10 check verify none on-marked-up shutdown-backup-sessions
server backup localhost:8080 backup check port 8080
is there something wrong with the way the keyword is being used?
Provide the full configuration, the output of haproxy -vv and most importantly: the haproxy logs.
HA-Proxy version 1.8.23 2019/11/25
Copyright 2000-2019 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-null-dereference -Wno-unused-label -Wno-stringop-overflow
OPTIONS = USE_LINUX_TPROXY=1 USE_CRYPT_H=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.1c FIPS 28 May 2019
Running on OpenSSL version : OpenSSL 1.1.1c FIPS 28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
and the config file
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
# to have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2 debug
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
## Security settings
# Use more bits for Diffie-Hellman exchange
tune.ssl.default-dh-param 2048
# Use only recent versions of TLS
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
# Use only recent ciphers
ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP
# ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
# turn on stats unix socket
stats socket /var/lib/haproxy/stats level admin
# We would like this but it is only supported in newer versions of HAProxy.
server-state-file /var/lib/haproxy/state
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch 3
retries 4
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 10000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend main
bind *:80
bind *:443 ssl crt /etc/pki/tls/certs/star.pem
redirect scheme https if !{ ssl_fc }
#acl url_static path_beg -i /static /images /javascript /stylesheets
#acl url_static path_end -i .jpg .gif .png .css .js
acl host_wwwt hdr(host) -i www-test.site.com
# Test fail over using number of available servers
#acl host_wwwt-health_fail nbsrv(wwwt) lt 1
#monitor-uri /
#monitor fail if host_wwwt-health_fail
#use_backend wwwt_failover if host_wwwt host_wwwt-health_fail
#use_backend wwwt_failover if host_wwwt-health_fail
use_backend wwwt if host_wwwt
# use_backend accounts if host_ba_prod
default_backend www
#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
#backend static
# balance roundrobin
# server static 127.0.0.1:4331 check
backend wwwt
balance leastconn
#errorfile 503 /etc/haproxy/errorfiles/503.http
option httpchk
server app1 192.168.0.89 ssl cookie app1 weight 10 check verify none on-marked-up shutdown-backup-sessions
server backup1 localhost:8080 backup check port 8080
and the log file
Jun 9 09:00:06 localhost haproxy[417741]: Proxy main started.
Jun 9 09:00:06 localhost haproxy[417741]: Proxy wwwt started.
Jun 9 09:00:06 localhost haproxy[417741]: Proxy wwwt_failover started.
Jun 9 09:00:06 localhost haproxy[417741]: Proxy www started.
Jun 9 09:00:26 localhost haproxy[417742]: 172.17.0.125:54256 [09/Jun/2020:09:00:26.410] main~ wwwt/app1 0/0/5/66/71 302 344 - - ---- 2/2/0/0/0 0/0 "GET / HTTP/1.1"
Jun 9 09:01:09 localhost haproxy[417819]: Proxy main started.
Jun 9 09:01:09 localhost haproxy[417819]: Proxy wwwt started.
Jun 9 09:01:09 localhost haproxy[417819]: Proxy wwwt_failover started.
Jun 9 09:01:09 localhost haproxy[417819]: Proxy www started.
Jun 9 09:04:56 localhost haproxy[1873]: Proxy main started.
Jun 9 09:04:56 localhost haproxy[1873]: Proxy wwwt started.
Jun 9 09:04:56 localhost haproxy[1873]: Proxy wwwt_failover started.
Jun 9 09:04:56 localhost haproxy[1873]: Proxy www started.
Jun 9 09:04:56 localhost haproxy[1874]: Backup Server wwwt/backup1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 9 09:04:57 localhost haproxy[1874]: Server wwwt_failover/backup is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jun 9 09:04:57 localhost haproxy[1874]: backend wwwt_failover has no server available!
Jun 9 09:05:39 localhost haproxy[1874]: 172.17.0.125:54436 [09/Jun/2020:09:05:39.498] main~ wwwt/app1 0/0/5/65/70 302 344 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
Jun 9 09:06:09 localhost haproxy[1874]: 172.17.0.125:54476 [09/Jun/2020:09:06:05.662] main~ wwwt/app1 0/3004/-1/-1/4006 503 212 - - SC-- 2/2/0/0/4 0/0 "GET / HTTP/1.1"
Jun 9 09:06:13 localhost haproxy[1874]: 172.17.0.125:54477 [09/Jun/2020:09:06:09.748] main~ wwwt/app1 0/3004/-1/-1/4006 503 212 - - SC-- 1/1/0/0/4 0/0 "GET /favicon.ico HTTP/1.1"
Jun 9 09:06:19 localhost haproxy[1874]: 172.17.0.125:54485 [09/Jun/2020:09:06:15.678] main~ wwwt/app1 0/3004/-1/-1/4007 503 212 - - SC-- 2/2/0/0/4 0/0 "GET / HTTP/1.1"
Jun 9 09:06:23 localhost haproxy[1874]: 172.17.0.125:54487 [09/Jun/2020:09:06:19.796] main~ wwwt/app1 0/3003/-1/-1/4006 503 212 - - SC-- 1/1/0/0/4 0/0 "GET /favicon.ico HTTP/1.1"
here is the log file after removing just the backup key word.
Jun 18 11:35:41 localhost haproxy[657179]: Proxy main started.
Jun 18 11:35:41 localhost haproxy[657179]: Proxy wwwt started.
Jun 18 11:35:41 localhost haproxy[657179]: Proxy wwwt_failover started.
Jun 18 11:35:41 localhost haproxy[657179]: Proxy www started.
Jun 18 11:36:05 localhost haproxy[657180]: 100.14.14.193:40734 [18/Jun/2020:11:36:05.233] main/2: SSL handshake failure
Jun 18 11:36:05 localhost haproxy[657180]: 100.14.14.193:40742 [18/Jun/2020:11:36:05.302] main/2: SSL handshake failure
Jun 18 11:36:05 localhost haproxy[657180]: 100.14.14.193:40744 [18/Jun/2020:11:36:05.394] main/2: SSL handshake failure
Jun 18 11:36:05 localhost haproxy[657180]: 100.14.14.193:40740 [18/Jun/2020:11:36:05.462] main/2: SSL handshake failure
Jun 18 11:36:09 localhost haproxy[657180]: 100.14.14.193:40746 [18/Jun/2020:11:36:09.961] main/2: SSL handshake failure
Jun 18 11:36:09 localhost haproxy[657180]: 100.14.14.193:40748 [18/Jun/2020:11:36:09.961] main/2: SSL handshake failure
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40754 [18/Jun/2020:11:36:10.409] main/2: SSL handshake failure
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40752 [18/Jun/2020:11:36:10.413] main/2: SSL handshake failure
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40756 [18/Jun/2020:11:36:10.465] main~ wwwt/backup1 0/0/1/1/2 200 20018 - - ---- 2/2/1/0/0 0/0 "GET / HTTP/1.1"
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40756 [18/Jun/2020:11:36:10.674] main~ wwwt/backup1 0/0/1/1/2 200 8202 - - ---- 2/2/1/0/0 0/0 "GET /images/reverse-logo-recovery-footer.png HTTP/1.1"
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40756 [18/Jun/2020:11:36:10.930] main~ wwwt/backup1 0/0/0/0/1 404 410 - - ---- 2/2/1/0/0 0/0 "GET /favicon.ico HTTP/1.1"
Jun 18 11:36:10 localhost haproxy[657180]: 100.14.14.193:40764 [18/Jun/2020:11:36:10.933] main/2: SSL handshake failure
Jun 18 11:36:13 localhost haproxy[657180]: 100.14.14.193:40750 [18/Jun/2020:11:36:10.019] main~ wwwt/backup1 0/3006/0/1/3007 200 20018 - - ---- 2/2/0/0/+3 0/0 "GET / HTTP/1.1"
Jun 18 11:36:24 localhost haproxy[657180]: 100.14.14.193:40768 [18/Jun/2020:11:36:23.994] main/2: SSL handshake failure
Jun 18 11:36:24 localhost haproxy[657180]: 100.14.14.193:40766 [18/Jun/2020:11:36:23.993] main/2: SSL handshake failure
Jun 18 11:36:27 localhost haproxy[657180]: 100.14.14.193:40770 [18/Jun/2020:11:36:24.047] main~ wwwt/backup1 0/3007/0/1/3008 200 20018 - - ---- 1/1/0/0/+3 0/0 "GET / HTTP/1.1"
Jun 18 11:36:27 localhost haproxy[657180]: 100.14.14.193:40776 [18/Jun/2020:11:36:27.302] main/2: SSL handshake failure
I don’t see how health checks are supposed to work in the first place, you did not specify any port for your primary server, in fact haproxy warns about it:
[ALERT] 169/182923 (24505) : parsing […/cert/mcdonaldr-backup-issue.cfg:100] : server app1 has neither service port nor check port nor tcp_check rule ‘connect’ with port information. Check has been disabled.
What exactly are you doing do simulate the primary server not working? Do you actually specify the port in the real configuration?
for simulating the failure, I am just stopping httpd on the backend server “app1”.
I did not have the check port 443 on the primary; added it and am now testing it.
changing the “check” to “check port 80” has made it work.
Without the check port 80, the backend (app1) site was being displayed if it was up. Is that normal, or was i being silly?
What you need to do is specify the port in general, not only for health checks. Same for the backup server.
<ip>:<port> instead of <ip> check port <port>
Try:
server app1 192.168.0.89:443 ssl cookie app1 weight 10 check verify none on-marked-up shutdown-backup-sessions
You configured:
check port 80
which means port 443 is no longer health check, which is not what you want since you are actually using port 443 for your real traffic.