Can't get dynamic custom error pages to work (HAProxy 2.3.5 on RHEL 8)

I am trying to implement a country based custom error page dependent of the status code like its shown here Serve Dynamic Custom Error Page - my implementation is based on the scenario where they use

http-response return status 404 errorfiles site1 if { status 404 } { var(txn.host) site1.com }

Little about my setup. I have an F5 load-balancer at the edge which does irule based traffic routing. I have three main services that are served as web services (separate irule for each) - lets call them site.no, site.se and site.fi and a bunch of supportin services which run on separate servers and have internal URLs but are not accessible from the outside and only through one of the main three websites. So for example I have service “subservice A” that can be accessed through the three main web sites (for some the paths are different across the NO,SE,FI realm, others may be identical)

https://site.no/some/path/to/subservices/subserviceA
https://site.se/subservices/subservice/A
https://site.fi/some/alternate/path/to/subservices/subserviceA

As I said the proxying trough the main sites is done on the edge reverse proxy/load balancer. After the previously mentioned lb/reverse proxy I have HAProxy as an ingress controller for a container environment (just a bunch of unorchestrated containers doing header enrichment and such). Sometimes those containers are down and HAProxy can detect them and shows the default 50x errors (I have seen 500, 503, 504). In HAProxy I have separate frontends and backends for each of the three main sites and all of the subservices. Custom error pages for the main sites works with

http-errors site-no {
   errorfile 500 /path/to/errorfiles/error-500-NO.http
   ...
   errorfile 504 /path/to/errorfiles/error-504-NO.http
} 

http-errors site-se {
   errorfile 500 /path/to/errorfiles/error-500-SE.http
   ...
   errorfile 504 /path/to/errorfiles/error-504-SE.http
}

http-errors site-fi {
   errorfile 500 /path/to/errorfiles/error-500-FI.http
   ...
   errorfile 504 /path/to/errorfiles/error-504-FI.http
}

frontend main-site-no {
    ...
    errorfiles site-no
    ...
}

frontend main-site-se {
    ...
    errorfiles site-se
    ...
}

frontend main-site-fi {
    ...
    errorfiles site-fi
    ...
}

But now with the subservice, as it is pan-NO,SE,FI I have to display the error based on status code and the header Host value - this is where the linked tutorial comes in. I did:

frontend sub-service-A {
    ...
    http-request set-var(txn.host) req.hdr(host)

    http-response return status 500 errorfiles site-no if { status 500 } { var(txn.host) site.no }
    ...
    http-response return status 504 errorfiles site-no if { status 504 } { var(txn.host) site.no }

    http-response return status 500 errorfiles site-se if { status 500 } { var(txn.host) site.se }
    ...
    http-response return status 504 errorfiles site-se if { status 504 } { var(txn.host) site.se }

    http-response return status 500 errorfiles site-fi if { status 500 } { var(txn.host) site.fi }
    ...
    http-response return status 504 errorfiles site-fi if { status 504 } { var(txn.host) site.fi }
    ...
}

And the problem is that the conditional errors are never triggered - I shut down the subserviceA and HAProxy marks it as DOWN. When I do

curl -I https://haproxy.subserviceA.frontend.ip.address:port -H "Host: site.no"

I get the default error page and not the custom one. The error status is correct - in this case 503 (also I noticed HTTP/2 and HTTP/1.1 show the error status a little differently i.e. HTTP/2 does not show the “Service Unavailable” text - could also be a curl thing)

I have no idea how to debug this. I added logging of req.hdr(host), var(txn.host) and %ST

htt-request capture req.hdr(Host) len 10
log-format "Host: %[capture.req.hdr(0)], Txn.Host: %[var(txn.host)], Status: %ST"

and they all show the pre-conditions to drigger the result:

Host: site.no, Txn.Host: site.no, Status: 503

Does anyone have ideas on how to proceed. I am running HAProxy 2.3.5 on RHEL 8

Post your unredacted, full configuration.

I removed the passwords and all the non-relevant frontends and backends. The subservice frontend/backends are basically identical.

global
    pidfile     /var/run/haproxy/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    master-worker
    stats socket /var/run/haproxy/haproxy.sock user haproxy group haproxy mode 660 level admin expose-fd listeners

   ## modern configuration
   ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
   ssl-default-bind-options prefer-client-ciphers ssl-min-ver TLSv1.2 no-tls-tickets

   ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
   ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets

  log stdout format raw local0
  log stderr format raw local0 notice

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

    unique-id-format        %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid

userlist haproxy-dataplaneapi
    user dataplaneapi password <password removed>

program api
    command /opt/haproxy/api/dataplaneapi --host 0.0.0.0 --port 5555 --haproxy-bin /opt/haproxy/sbin/haproxy --config-file /opt/haproxy/conf/haproxy.cfg --reload-cmd "kill -SIGUSR2 1" --reload-delay 5 --userlist haproxy-dataplaneapi --maps-dir=/opt/haproxy/maps --ssl-certs-dir=/opt/haproxy/tls --spoe-dir=/opt/haproxy/spoe
no option start-on-reload

frontend stats
    bind 0.0.0.0:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats realm Ingress\ Controller\ #01\ Statistics
    stats auth admin:<password removed>
    stats admin if TRUE

# Error pages #
http-errors duckasylum-prodno
    errorfile 403 /opt/haproxy/errors/page-403_NO.http
    errorfile 500 /opt/haproxy/errors/page-500_NO.http
    errorfile 502 /opt/haproxy/errors/page-502_NO.http
    errorfile 503 /opt/haproxy/errors/page-503_NO.http
    errorfile 504 /opt/haproxy/errors/page-504_NO.http

http-errors duckasylum-prodse
    errorfile 403 /opt/haproxy/errors/page-403_SE.http
    errorfile 500 /opt/haproxy/errors/page-500_SE.http
    errorfile 502 /opt/haproxy/errors/page-502_SE.http
    errorfile 503 /opt/haproxy/errors/page-503_SE.http
    errorfile 504 /opt/haproxy/errors/page-504_SE.http

http-errors duckasylum-prodfi
    errorfile 403 /opt/haproxy/errors/page-403_FI.http
    errorfile 500 /opt/haproxy/errors/page-500_FI.http
    errorfile 502 /opt/haproxy/errors/page-502_FI.http
    errorfile 503 /opt/haproxy/errors/page-503_FI.http
    errorfile 504 /opt/haproxy/errors/page-504_FI.http

# F5 --> <LB ingress> --> agw #
frontend ingress_duckasylum-pno_external_A
   mode                http
   option              httplog
   bind                0.0.0.0:20001 ssl crt /opt/haproxy/tls/fullchain.pem.crt alpn h2,http/1.1

   # 'pretty' error pages #
   errorfiles duckasylum-pno

   default_backend     agw_duckasylum-pno_A

frontend ingress_duckasylum-pse_external_A
    mode                http
    option              httplog
    bind                0.0.0.0:20002 ssl crt /opt/haproxy/tls/fullchain.pem.crt alpn h2,http/1.1

    acl                 site_dead nbsrv(agw_duckasylum-pse_A) lt 1
    tcp-request         connection reject if site_dead

   #  'pretty' error pages #
   errorfiles duckasylum-pse

   default_backend     agw_duckasylum-pse_A

frontend ingress_duckasylum-pfi_external_A
   mode                http
   option              httplog
   bind                0.0.0.0:20003 ssl crt /opt/haproxy/tls/fullchain.pem.crt alpn h2,http/1.1

  # 'pretty' error pages #
  errorfiles duckasylum-pfi

  default_backend     agw_duckasylum-pfi_A


frontend ingress_subserviceA-ppan_external_A
    mode                http
    option              httplog
    bind                0.0.0.0:20006 ssl crt /opt/haproxy/tls/fullchain.pem.crt alpn h2,http/1.1

    # 'pretty' error pages #
    http-request set-var(txn.host) req.hdr(host)

    http-response return status 403 errorfiles duckasylum-pno if { status 403 } { var(txn.host) duckasylum.no }
    http-response return status 403 errorfiles duckasylum-pse if { status 403 } { var(txn.host) duckasylum.se }
    http-response return status 403 errorfiles duckasylum-pfi if { status 403 } { var(txn.host) duckasylum.fi }

    http-response return status 500 errorfiles duckasylum-pno if { status 500 } { var(txn.host) duckasylum.no }
    http-response return status 500 errorfiles duckasylum-pse if { status 500 } { var(txn.host) duckasylum.se }
    http-response return status 500 errorfiles duckasylum-pfi if { status 500 } { var(txn.host) duckasylum.fi }

    http-response return status 502 errorfiles duckasylum-pno if { status 502 } { var(txn.host) duckasylum.no }
    http-response return status 502 errorfiles duckasylum-pse if { status 502 } { var(txn.host) duckasylum.se }
    http-response return status 502 errorfiles duckasylum-pfi if { status 502 } { var(txn.host) duckasylum.fi }

    http-response return status 503 errorfiles duckasylum-pno if { status 503 } { var(txn.host) duckasylum.no }
    http-response return status 503 errorfiles duckasylum-pse if { status 503 } { var(txn.host) duckasylum.se }
    http-response return status 503 errorfiles duckasylum-pfi if { status 503 } { var(txn.host) duckasylum.fi }

    http-response return status 504 errorfiles duckasylum-pno if { status 504 } { var(txn.host) duckasylum.no }
    http-response return status 504 errorfiles duckasylum-pse if { status 504 } { var(txn.host) duckasylum.se }
    http-response return status 504 errorfiles duckasylum-pfi if { status 504 } { var(txn.host) duckasylum.fi }


   default_backend     agw_subserviceA-ppan_A


backend agw_duckasylum-pno_A
    balance     roundrobin

    acl xrid_exists req.hdr(X-Request-ID) -m found
    http-request set-header X-Request-ID %[unique-id] unless xrid_exists

    http-response set-header X-Response-ID %[unique-id]

    acl xrip_exists req.hdr(X-Real-IP) -m found
    http-request set-header X-Real-IP %[hdr(x-forwarded-for)] unless xrip_exists

    http-request replace-value X-Forwarded-For ^ " %[hdr(x-forwarded-for)], %[src]"

    option httpchk GET /health-check
    http-check expect status 200
    default-server inter 3s fall 3 rise 2

    server      agw_duckasylum-pno_A-01 172.30.160.1:443 ssl verify none check port 8080

backend agw_duckasylum-pse_A
    balance     roundrobin

   acl xrid_exists req.hdr(X-Request-ID) -m found
   http-request set-header X-Request-ID %[unique-id] unless xrid_exists

   http-response set-header X-Request-ID %[unique-id]

   acl xrip_exists req.hdr(X-Real-IP) -m found
   http-request set-header X-Real-IP %[hdr(x-forwarded-for)] unless xrip_exists

   http-request replace-value X-Forwarded-For ^ " %[hdr(x-forwarded-for)], %[src]"

   option httpchk GET /health-check
   http-check expect status 200
   default-server inter 3s fall 3 rise 2

  server      agw_duckasylum-pse_A-01 172.30.160.11:443 ssl verify none check port 8080

backend agw_duckasylum-pfi_A
   balance     roundrobin

   acl xrid_exists req.hdr(X-Request-ID) -m found
   http-request set-header X-Request-ID %[unique-id] unless xrid_exists
   http-response set-header X-Response-ID %[unique-id]

   acl xrip_exists req.hdr(X-Real-IP) -m found
   http-request set-header X-Real-IP %[hdr(x-forwarded-for)] unless xrip_exists

   http-request replace-value X-Forwarded-For ^ " %[hdr(x-forwarded-for)], %[src]"

   option httpchk GET /health-check
   http-check expect status 200
   default-server inter 3s fall 3 rise 2

   server      agw_ibank-flt_A-01 172.30.160.21:443 ssl verify none check port 8080

backend agw_subserviceA-ppan_A
   balance     roundrobin

   acl xrid_exists req.hdr(X-Request-ID) -m found
   http-request set-header X-Request-ID %[unique-id] unless xrid_exists
   http-response set-header X-Response-ID %[unique-id]

   acl xrip_exists req.hdr(X-Real-IP) -m found
   http-request set-header X-Real-IP %[hdr(x-forwarded-for)] unless xrip_exists

   http-request replace-value X-Forwarded-For ^ " %[hdr(x-forwarded-for)], %[src]"

   option httpchk GET /health-check
   http-check expect status 200
   default-server inter 3s fall 3 rise 2

  server      agw_subserviceA-ppan_A-01 172.30.160.51:443 ssl verify none check port 8080