HAProxy servers staying down after failed health checks

Help!
1

I am using HAProxy to front 2 (or more) 3rd party Databases as a Service servers. My initial setup requirements include a basic round-robin and adding custom basic auth HTTP headers that are unique to each backend server. I have implemented this as shown below in the config file.

The problem is that whenever a server goes down from failing a health check it will never come back up. I must “restart” the server (I am using Docker to run HAProxy so I actually reload the config file using docker kill -s HUP <haproxy-container>).

This seems to be something simple I have a misunderstanding of. Any suggestions??

Details: HAProxy 1.7 using Docker. HTTPS cert using Let’s Encrypt SSL cert. Roundrobin between 2 servers, which are database as a service servers needing unique basic auth headers added to their requests.

haproxy.cfg:

global
    daemon
    maxconn 256
    log /dev/log local0

defaults
    mode http
    log global
    option httplog
    option log-health-checks
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

userlist auth_api
    user proxyuser password "${PROXY_PWD}"

#---------------------------------------------------------------------
# HTTPS in
#---------------------------------------------------------------------
frontend https-in
    bind :443 ssl crt /usr/local/etc/haproxy/certs/"${DOMAIN_NAME}".pem
    reqadd X-Forwarded-Proto:\ https
    
    acl stats url_beg /stats
    use_backend stats if stats
    
    acl letsencrypt-acl path_beg /.well-known/acme-challenge/
    use_backend letsencrypt-backend if letsencrypt-acl
    
    default_backend proxy

#---------------------------------------------------------------------
# Proxy Backend
#---------------------------------------------------------------------
backend proxy
    acl auth http_auth(auth_api)
    http-request allow if auth
    http-request auth

    option httpchk
    default-server inter 3s fall 3 rise 2

    balance roundrobin
    server proxy1    0.0.0.0:8080 check
    server proxy2    0.0.0.0:8081 check

#---------------------------------------------------------------------
# Elastic servers
#---------------------------------------------------------------------
listen proxy1
    bind *:8080
    reqidel '^Authorization:.*'
    reqidel '^Host:.*'
    reqadd "Authorization: Basic ${PROXY1_CREDS}"
    reqadd "Host: ${PROXY1_HOSTNAME}"
    rspadd "X-Proxy1-Backend: ${PROXY1_HOSTNAME}"
    server db1 "${PROXY1_HOSTNAME}" check ssl verify none

listen proxy2
    bind *:8081
    reqidel '^Authorization:.*'
    reqidel '^Host:.*'
    reqadd "Authorization: Basic ${PROXY2_CREDS}"
    reqadd "Host: ${PROXY2_HOSTNAME}"
    rspadd "X-Proxy2-Backend: ${PROXY2_HOSTNAME}"
    server db2 "${PROXY2_HOSTNAME}" check ssl verify none

#---------------------------------------------------------------------
# Stats server: /stats
#---------------------------------------------------------------------
backend stats
   mode http
   stats enable
   stats hide-version
   stats realm Haproxy\ Statistics
   stats uri /
   stats auth "${STATS_USER}":"${STATS_PWD}"

#---------------------------------------------------------------------
# Let's Encrypt ACME Challenges Handler
#---------------------------------------------------------------------
backend letsencrypt-backend
   server letsencrypt 127.0.0.1:54321