HAProxy 2.5.1 occasional default backend selected instead of correct backend

I’ve been using HAProxy 1.7 to front Obsidian successfully for many years.
I have a couple of HAProxy servers which were running 2.5 fronting Obsidian in a production environment for months with no problems.
I recently upgraded them to 2.51 with no problems.

I’ve deployed 2 new HAProxy 2.51 servers in a development environment.
I see very strange behavior.
I’ve tested with many browsers from different locations.
I’ve even tried using HTTP 1.1 instead of 2.0 to no avail.

When it works correctly the logs look like this:

Feb  9 15:09:13 149.97.134.35:22159 [09/Feb/2022:15:08:42.499] main_ssl~ ssl_backend-obs/<NOSRV> 30601/-1/-1/-1/30600
302 90 - - LR-- 1/1/0/0/3 0/0 {} "GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb  9 15:09:13 149.97.134.35:22159 [09/Feb/2022:15:09:13.098] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 74/0/0/1/74
302 108 - - ---- 1/1/0/0/0 0/0 {} "GET https://tndci-obs.cloud.micropact.com/obsidian HTTP/2.0"

tndci-obs.cloud.micropact.com is sent via 302 to the ssl_backend-obs and “/obsidian” is appended to the path.

When it fails, the logs look like this:

Feb  9 13:26:24 209.112.9.82:65220 [09/Feb/2022:13:26:23.879] main_ssl~ ssl_backend-vr/<NOSRV> 1/-1/-1/-1/191
302 86 - - LRNN 3/3/0/0/3 0/0 {} "GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb  9 13:26:24 209.112.9.82:65220 [09/Feb/2022:13:26:24.071] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 56/0/0/5/67
200 38818 - - --VN 3/3/0/0/0 0/0 {} "GET https://tndci-obs.cloud.micropact.com/le5/ HTTP/2.0"

tndci-obs.cloud.micropact.com is sent to the default backend ssl_backend-vr and “/le5” is appended to the path.

Since this is HTTP 2.0 the URL is absolute and it clearly contains “tndci-obs”.

Frontend:

acl obs             hdr_dom(host)   tndci-obs.
use_backend ssl_backend-obs if obs

Backend:

backend ssl_backend-obs
balance     roundrobin
option httpchk GET /obsidian/
http-check disable-on-404
acl down nbsrv() eq 0
acl path_root  path /
http-response set-header Cache-Control no-cache,must-revalidate
http-response set-header Pragma no-cache
http-response set-header Expires 0
http-request allow if { src -f /etc/CONFIG/haproxy/whitelist.lst } || { ssl_c_used }
http-request deny
redirect location /obsidian         if path_root !down

I assume the frontend occasionally fails to match.
How do I debug this?

Provide the full configuration please.

# ---------------------------------------------------------------------
# See the full configuration options online:
#  https://cbonte.github.io/haproxy-dconv/2.4/configuration.html
# ---------------------------------------------------------------------
# includes from /etc/CONFIG/haproxy/conf.d/
global
    # To have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    user        haproxy
    group       haproxy
    master-worker
    nbthread    4
    daemon
    ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-bind-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256
#   ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECD
HE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:!AECDH 
    ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
    ssl-default-server-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256
    stats socket /var/lib/haproxy/stats mode 600 level admin
    stats timeout 2m
    tune.ssl.default-dh-param 2048
    userlist UsersFor_Java
      user java insecure-password xxxxx
    userlist UsersFor_Logi
      user logi insecure-password xxxxxx
defaults
    default_backend         ssl_backend-vr
    errorfile 408 /dev/null
    log    global
    maxconn                 5000
    mode   http
    option abortonclose
    option dontlognull
    option forwardfor       except 127.0.0.0/8
    option httplog
    option http-ignore-probes
    option redispatch
    retries                 3
    timeout check           10s
    timeout client          1m
    timeout connect         10s
    timeout http-keep-alive 10s
    timeout http-request    10s
    timeout queue           1m
    timeout server          5m
peers mypeers
# include hap_servers-haproxy declarations
    peer ip-10-160-17-161 10.160.17.161:1024
    peer ip-10-160-17-64 10.160.17.64:1024
# include frontend declarations
frontend  main
    bind 0.0.0.0:80
    acl lvr             hdr_dom(host)   lvr.
    acl ltndci-vr       hdr_dom(host)   ltndci-vr.
    acl solr            hdr_dom(host)   ltndci-solr. ltndci-access.
    acl search          hdr_dom(host)   ltndci-search.
    acl apigw           hdr_dom(host)   ltndci-apigw.
    acl ci              hdr_dom(host)   ci.
    http-request redirect scheme https code 301 if !{ ssl_fc } !lvr !ltndci-vr !solr !search !apigw !ci
    use_backend backend-solr   if solr
#   use_backend backend-search if search
#   use_backend backend-apigw  if apigw
    use_backend backend-ci     if ci
frontend  main_ssl
# Bind SSL port with PFS-enabling cipher suite
#   bind :443  ssl crt /etc/CONFIG/haproxy/certs/cert.pem alpn h2,http/1.1
    bind :443  ssl crt /etc/CONFIG/haproxy/certs/cert.pem
#   bind :8443 ssl crt /etc/CONFIG/haproxy/certs/cert.pem ca-file /apps/Config/certs/clients.cert.pem verify optional crt-ignore-err all
    http-request set-header X-Forwarded-Proto https if { ssl_fc } 
    http-request set-header X-Forwarded-Port 443    if { ssl_fc }
    capture request header  X-Forwarded-For len 15
# Rate limit
    stick-table  type ip size 100k expire 30s store http_req_rate(10s)
    http-request track-sc0 src
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 150 } !{ src -f /etc/CONFIG/haproxy/cidr.lst }
# Distinguish between secure and insecure requests
    acl secure dst_port eq 443
# Filter nasty input
    option http-buffer-request
    acl log4shell url,url_dec -m reg \${[^}]*\${
    acl log4shell url,url_dec -m reg \${jndi:(?:ldaps?|iiop|dns|rmi)://
    acl log4shell url,url_dec -i -m reg \${[\w${}\-:]*j[\w${}\-:]*n[\w${}\-:]*d[\w${}\-:]*i[\w${}\-:]*:.*}
    acl log4shell req.hdrs -m reg \${[^}]*\${
    acl log4shell req.hdrs -m reg \${jndi:(?:ldaps?|iiop|dns|rmi)://
    acl log4shell req.hdrs -i -m reg \${[\w${}\-:]*j[\w${}\-:]*n[\w${}\-:]*d[\w${}\-:]*i[\w${}\-:]*:.*}
    acl log4shell_form req.body,url_dec -m reg \${[^}]*\${
    acl log4shell_form req.body,url_dec -m reg \${jndi:(?:ldaps?|iiop|dns|rmi)://
    acl log4shell_form req.body,url_dec -i -m reg \${[\w${}\-:]*j[\w${}\-:]*n[\w${}\-:]*d[\w${}\-:]*i[\w${}\-:]*:.*}
    http-request deny if log4shell
    http-request deny if { req.fhdr(content-type) -m str application/x-www-form-urlencoded } log4shell_form
    acl missing_cl hdr_cnt(Content-length) eq 0
    acl METH_PUT method PUT
    acl METH_GET method GET HEAD
    acl METH_PATCH method PATCH
    acl METH_DELETE method DELETE
    http-request deny if HTTP_URL_STAR !METH_OPTIONS || METH_PUT missing_cl
    http-request deny if METH_GET HTTP_CONTENT
    http-request deny unless METH_GET or METH_POST or METH_OPTIONS or METH_PATCH or METH_DELETE or METH_PUT
# Mark all cookies as secure if sent over SSL
    http-response replace-header Set-Cookie (.*)[Hh]ttp[Oo]nly;?(.*) \1\2 if secure
    http-response replace-header Set-Cookie (.*)[Ss]ecure;?(.*) \1\2 if secure
    http-response replace-header Set-Cookie (.*) \1;HttpOnly;Secure; if secure
#   http-response replace-header Set-Cookie (.*) \1;HttpOnly;Secure;sameSite=strict; if secure
# Add the HSTS header with a 1 year max-age
    http-response set-header Strict-Transport-Security max-age=31536000;includeSubDomains;preload if secure
# Add cache headers
#   http-response set-header Cache-Control no-cache,must-revalidate
#   http-response set-header Pragma no-cache
#   http-response set-header Expires 0
# Add additional security headers
    http-response set-header Referrer-Policy no-referrer-when-downgrade
    http-response set-header X-Frame-Options SAMEORIGIN
    http-response set-header Content-Security-Policy "frame-ancestors 'self';"
    http-response set-header Access-Control-Allow-Origin *
    http-response set-header X-XSS-Protection 1;report=https://micropact.report-uri.io/r/d/xss/enforce
    http-response set-header X-Content-Type-Options nosniff
#   http-response set-header Feature-Policy "vibrate self"
    http-response set-header Expect-CT max-age=0;report=https://micropact.report-uri.io/r/d/ct/reportOnly
    http-response set-header Expect-Staple report=https://micropact.report-uri.io/r/d/staple/reportOnly
    http-response set-header Report-To {"group":"default","max_age":31536000,"endpoints":[{"url":"https://micropact.report-uri.com/a/d/g"}],"include_subdomains":true}
    http-response set-header NEL {"report_to":"default","max_age":31536000,"include_subdomains":true}
    http-response del-header Server
    http-response del-header X-Powered-By
    http-response del-header X-Runtime
# include front_ssl declarations
    acl search             hdr_dom(host)   tndci-access.
    acl apigw              path_beg        /apigw
    acl solr               path_beg        /solr
    use_backend backend-solr   if solr
    use_backend ssl_backend-apigw  if apigw
    use_backend ssl_backend-search if search
    acl idm     hdr_beg(host)   -i tndci-idm.
    use_backend ssl_backend-idm if idm
    acl logi            hdr_dom(host)   logi.
    use_backend ssl_backend-logi if logi
    acl logitest        hdr_dom(host)   logitest.
    use_backend ssl_backend-logitest if logitest
    acl lrs             hdr_dom(host)   ltndci-rs.
    use_backend ssl_backend-rs if lrs
    acl lvr             hdr_dom(host)   tndci-lvr.
    use_backend ssl_backend-lvr if lvr
    acl obs             hdr_dom(host)   tndci-obs.
    use_backend ssl_backend-obs if obs
    acl rs             hdr_dom(host)   tndci-rs.
    use_backend ssl_backend-rs if rs
    acl vo              hdr_dom(host)   tndci-vo.
    use_backend ssl_backend-vo if vo
    acl vomaster                hdr_dom(host)   vomaster.
    use_backend ssl_backend-vomaster if vomaster
    acl vopreprod               hdr_dom(host)   tndci-vopreprod.
    use_backend ssl_backend-vopreprod if vopreprod
    acl votest          hdr_dom(host)   votest.
    use_backend ssl_backend-votest if votest
    acl vr              hdr_dom(host)   tndci-vr.
    use_backend ssl_backend-vr if vr
    acl vrmaster                hdr_dom(host)   vrmaster.
    use_backend ssl_backend-vrmaster if vrmaster
    acl vrpreprod               hdr_dom(host)   tndci-vrpreprod.
    use_backend ssl_backend-vrpreprod if vrpreprod
    acl vrtest          hdr_dom(host)   vrtest.
    use_backend ssl_backend-vrtest if vrtest
# include backend declarations
backend backend-ci
    balance     roundrobin
    option httpchk GET /login
    http-check disable-on-404
    http-request allow if { src -f /etc/CONFIG/haproxy/ci-whitelist.lst }
    http-request deny
#   acl path_root  path /
#   redirect location /rashome.aspx  if path_root
# include servers-ci declarations
backend backend-solr
    balance     roundrobin
    option httpchk GET /solr/versa/admin/ping
    http-check disable-on-404
    cookie SRV insert indirect nocache httponly secure
# include servers-solr declarations
    server i-076e2e764968716eb-9034 10.160.17.160:9034 check cookie i-076e2e764968716eb-9034
# include back_ssl declarations
backend ssl_backend-apigw
    balance     roundrobin
    option httpchk GET /
    http-check disable-on-404
    cookie SRV insert indirect nocache httponly secure
# include ssl_servers-apigw declarations
    server i-076e2e764968716eb-9054 10.160.17.160:9054 check cookie i-076e2e764968716eb-9054
backend ssl_backend-idm
    balance     roundrobin
    option httpchk GET /IronDataMobile/about.txt
    http-check disable-on-404
    cookie SRV insert indirect nocache httponly secure
    acl path_form path_beg /form
    acl path_root  path /
    redirect location /IronDataMobile/formBuilder/index-prod.html  if path_form
    redirect location /IronDataMobile/latest/index-cache.html  if path_root
# include ssl_servers-idm declarations
    server i-07837e30d48328986 10.160.17.51:9003 check cookie i-07837e30d48328986
backend ssl_backend-lrs
    balance     roundrobin
    option httpchk GET /entellitrak/web-pub/udm/images/right-msdn.gif
    http-check disable-on-404
    cookie SRV insert indirect nocache httponly secure
    acl path_root  path /
    redirect location /entellitrak      if path_root
# include ssl_servers-rs declarations
    server i-0f4cc998ff5c18cf4 10.160.17.42:9006 check cookie i-0f4cc998ff5c18cf4
backend ssl_backend-lvr
    balance     roundrobin
    option httpchk GET /le5/about.txt
    http-check disable-on-404
    http-request allow if { src -f /etc/CONFIG/haproxy/whitelist.lst } || { ssl_c_used }
    http-request deny
    acl path_root path /
    redirect location /le5/     if path_root
# include ssl_servers-vr declarations
    server i-0548140935b42ced2 10.160.17.28:9001 check cookie i-0548140935b42ced2
backend ssl_backend-obs
    balance     roundrobin
    option httpchk GET /obsidian/
    http-check disable-on-404
    acl down nbsrv() eq 0
    acl path_root  path /
    http-response set-header Cache-Control no-cache,must-revalidate
    http-response set-header Pragma no-cache
    http-response set-header Expires 0
    http-request allow if { src -f /etc/CONFIG/haproxy/whitelist.lst } || { ssl_c_used }
    http-request deny
    redirect location /obsidian         if path_root !down
# include servers-obs declarations
    server i-07507641ad1b1365c 10.160.17.17:9016 check cookie i-07507641ad1b1365c
backend ssl_backend-rs
    balance     roundrobin
    option httpchk GET /entellitrak/web-pub/udm/images/right-msdn.gif
    http-check disable-on-404
    cookie SRV insert indirect nocache httponly secure
    acl path_root  path /
    redirect location /entellitrak      if path_root
# include ssl_servers-rs declarations
    server i-0f4cc998ff5c18cf4 10.160.17.42:9006 check cookie i-0f4cc998ff5c18cf4
backend ssl_backend-search
    balance     roundrobin
    option httpchk GET /locales/en/locale.json
    http-check disable-on-404
    cookie SRV insert indirect nocache httponly secure
# include ssl_servers-search declarations
    server i-076e2e764968716eb-9064 10.160.17.160:9064 check cookie i-076e2e764968716eb-9064
backend ssl_backend-vo
    balance     roundrobin
    option httpchk GET /datamart/about.txt
    http-check disable-on-404
    cookie SRV insert indirect nocache httponly secure
    acl path_status path /status
    acl AuthOkay_Java http_auth(UsersFor_Java)
    acl path_admin path_beg /admin /staff
    acl path_root  path /
    acl path_data  path /datamart
    http-request replace-path /datamart /datamart/languageChoice.do if path_data
    http-request replace-path /(admin|staff)(.*) /datamart/\1/languageChoice.do\2 if path_admin
    http-request auth realm Java if path_status !AuthOkay_Java
    http-request allow if { src -f /etc/CONFIG/haproxy/vo-whitelist.lst } || { ssl_c_used }
    http-request deny
    redirect location /datamart/languageChoice.do if path_root
# include ssl_servers-vo declarations
    server i-07837e30d48328986 10.160.17.51:9002 check cookie i-07837e30d48328986
backend ssl_backend-vr
    balance     roundrobin
    option httpchk GET /le5/about.txt
    http-check disable-on-404
    cookie SRV insert indirect nocache httponly secure
    acl path_status path /status
    acl AuthOkay_Java http_auth(UsersFor_Java)
    acl path_root path /
    http-request auth realm Java if path_status !AuthOkay_Java
    http-request allow if { src -f /etc/CONFIG/haproxy/vr-whitelist.lst } || { ssl_c_used }
    http-request deny
    redirect location /le5/     if path_root
# include ssl_servers-vr declarations
    server i-0548140935b42ced2 10.160.17.28:9001 check cookie i-0548140935b42ced2
# include listen declarations
listen stats
    bind 0.0.0.0:9999
    mode http
    balance
    timeout client 5000
    timeout connect 4000
    timeout server 30000
    stats hide-version
    stats uri /haproxy_stats 
    stats auth admin: xxxxx
# include resolvers declarations
resolvers mydns
  parse-resolv-conf
  resolve_retries       3
  timeout resolve       1s
  timeout retry         1s
  hold other           30s
  hold refused         30s
  hold nx              30s
  hold timeout         30s
  hold valid           10s
  hold obsolete        30s
# include proxy declarations

have you tired making your match case-insensitive or using hdr_beg instead of hdr_dom since you are matching the subdomain?

I assume because the pattern always ended in a “.”, hdr_dom worked correctly for the last 7 years.
On your suggestion, I changed:

    acl obs             hdr_dom(host)   tndci-obs.
    use_backend ssl_backend-obs if obs

To:

    acl obs             hdr_beg(host)   -i tndci-obs.
    use_backend ssl_backend-obs if obs

After making this change, I wasn’t able to repeat the problem for 3 days but now it has reared its ugly head again!

Log extract from Feb 15 around 16:05 CST:

Feb 15 16:05:58 149.97.134.35:65389 [15/Feb/2022:16:05:58.741] main_ssl~ ssl_backend-obs/<NOSRV> 0/-1/-1/-1/0 302 90 - - LR-- 1/1/0/0/3 0/0 {} "GET / HTTP/1.1" -
Feb 15 16:05:58 149.97.134.35:65389 [15/Feb/2022:16:05:58.813] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/1/0/1 302 108 - - ---- 1/1/0/0/0 0/0 {} "GET /obsidian HTTP/1.1" -
Feb 15 16:05:58 149.97.134.35:65389 [15/Feb/2022:16:05:58.940] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/1/1 304 111 - - ---- 1/1/0/0/0 0/0 {} "GET /obsidian/ HTTP/1.1" -
Feb 15 16:05:59 149.97.134.35:65389 [15/Feb/2022:16:05:59.301] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/7/7 304 110 - - ---- 1/1/0/0/0 0/0 {} "GET /obsidian/assets/custom.css HTTP/1.1" -
Feb 15 16:05:59 149.97.134.35:65388 [15/Feb/2022:16:05:58.638] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 582/0/0/6/690 404 231 - - --NI 1/1/0/0/0 0/0 {} "GET https://tndci-obs.cloud.micropact.com/obsidian/assets/main.js HTTP/2.0"

The 1st line detects the correct backend ssl_backend-obs and does a 302 redirect to the correct server.
This is followed by two 304 Not Modified redirects to add “/” to the end of the path and then modify the path.
Finally the traffic is sent to the default backend and thus the wrong server resulting in a 404 Not Found.
The log time appears to skip backward from 16:05:59.301 to 16:05:58.638 and the port changes
from 65389 to 65388 ?!

by best guess is that health check takes the backend offline, which is then causing haproxy to select the default backend

The Obsidian backend is UP and doesn’t flap:

[TNDCI-LB-64]# $CBIN/haproxy-state
# pxname,svname,status
main,FRONTEND,OPEN
main_ssl,FRONTEND,OPEN
ssl_backend-apigw,i-076e2e764968716eb-9054,UP
ssl_backend-idm,i-07837e30d48328986,UP
ssl_backend-lrs,i-0f4cc998ff5c18cf4,UP
ssl_backend-lsolr,i-076e2e764968716eb-9034,UP
ssl_backend-lvr,i-0548140935b42ced2,UP
ssl_backend-obs,i-07507641ad1b1365c,UP
ssl_backend-rs,i-0f4cc998ff5c18cf4,UP
ssl_backend-search,i-076e2e764968716eb-9064,UP
ssl_backend-solr,i-076e2e764968716eb-9034,UP
ssl_backend-vo,i-07837e30d48328986,UP
ssl_backend-vopreprod,i-08995fc9329e978e0,UP
ssl_backend-vr,i-0548140935b42ced2,UP
ssl_backend-vrpreprod,i-08995fc9329e978e0,UP
stats,FRONTEND,OPEN

your backend configs are different than what I’ve used in the past. I’ve never seen the use of nbsrv on the backend. Backends are more complicated than what I’ve ever used!

perhaps try taking out your monitor or moving it to the frontend. I’m out of ideas. Hopefully someone with more experience will be able to help!

You configuration is so complex, it’s very hard to understand what it’s supposed to do. With a configuration that complex you should probably avoid run bleeding edge haproxy releases.

I’d suggest running LTS release trains instead.

The host header is not constructed out of the absolute URI though. Host header is host header. So I’d suggest you start logging the host header together with the rest.

So in frontend main_ssl, let’s add Host header logging:

http-request capture req.hdr(Host) len 30

Then check your logs again and let’s see what the Host header actually contains.

I added the following:

http-request capture req.hdr(Host) len 100

Now when the request works correctly I see the host between the “{}”:

Feb 17 10:37:33 209.112.9.82:65114 [17/Feb/2022:10:37:33.232] main_ssl~ ssl_backend-obs/<NOSRV> 0/-1/-1/-1/0 302 90 - - LR-- 6/6/0/0/3 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb 17 10:37:33 209.112.9.82:65114 [17/Feb/2022:10:37:33.274] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/1/0/1
302 108 - - ---- 6/6/0/0/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian HTTP/2.0"
Feb 17 10:37:33 209.112.9.82:65114 [17/Feb/2022:10:37:33.310] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/1/1
200 965 - - ---- 6/6/0/0/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian/ HTTP/2.0"
Feb 17 10:37:33 209.112.9.82:65114 [17/Feb/2022:10:37:33.590] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/6/6
304 110 - - ---- 6/6/1/1/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian/assets/custom.css HTTP/2.0"

When the request is sent to the default backend the host is missing:

Feb 17 10:30:43 209.112.9.82:64653 [17/Feb/2022:10:30:43.115] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 2/0/0/1/37
404 312 - - --NI 1/1/0/0/0 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/obsidian/ HTTP/2.0"
Feb 17 10:30:43 209.112.9.82:64653 [17/Feb/2022:10:30:43.153] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 200/0/0/0/199
200 1613 - - --VN 1/1/0/0/0 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/favicon.ico HTTP/2.0"

This is a though one to troubleshoot, because this is all hidden in SSL, so we can’t confirm easily what happens.

Is this happening in different browsers/clients, or is this a particular client?

I’d suggest to use a different haproxy release here. 2.5.1 has some bugs, but with the information available it is difficult to pin point to a specific bug.

I suggest you try 2.4.13 first of all.

The alternative is to setup SSL key logging tune.ssl.keylog and capture and decrypt the SSL connection and analyze the HTTP transaction.

I was running 2.5.2.
Following your suggestion I installed 2.4.13 on both HAProxy servers.
Chrome browser instance worked:

Feb 17 16:59:40 209.112.9.82:57773 [17/Feb/2022:16:59:40.293] main_ssl~ ssl_backend-obs/<NOSRV> 0/-1/-1/-1/0
302 90 - - LRNN 1/1/0/0/3 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb 17 16:59:40 209.112.9.82:57773 [17/Feb/2022:16:59:40.318] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/1/0/1
302 108 - - --NI 1/1/0/0/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian HTTP/2.0"
Feb 17 16:59:40 209.112.9.82:57773 [17/Feb/2022:16:59:40.443] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/2/2
200 965 - - --VN 1/1/0/0/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian/ HTTP/2.0"
Feb 17 16:59:40 209.112.9.82:57773 [17/Feb/2022:16:59:40.595] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/1/5/6
304 114 - - --VN 1/1/1/1/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian/assets/main.js HTTP/2.0"

Firefox browser instance failed:

Feb 17 17:02:06 209.112.9.82:57996 [17/Feb/2022:17:02:06.554] main_ssl~ ssl_backend-vr/<NOSRV> 2/-1/-1/-1/63
302 86 - - LRNN 1/1/0/0/3 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb 17 17:02:06 209.112.9.82:57996 [17/Feb/2022:17:02:06.617] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 83/0/0/5/94
200 38818 - - --VN 1/1/0/0/0 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/le5/ HTTP/2.0"

30 minutes later a refresh on Firefox browser renders site correctly.
Also an Opera browser that had displayed the default backend also worked correctly after a refresh.
I saw this behaviour with 2.5.2.
So what state change happens over time?
Continuing to test - hasn’t failed again yet.

I haven’t had any failures all morning.
The configuration wasn’t changed between 2.5.2 and 2.4.13.
So is this a 2.5 regression in SSL processing?
How do I report it to Willy et al?

You have been unable to reproduce the issue before for 3 days after making a small change, so I’d say that not hitting any failures for a single morning is not indicative of having found a proper workaround.

Regarding a bug report, we pretty much don’t know anything at all at this point. I’d suggest we make really sure that 2.4.13 works long-term, before making a bug reports. A bug report at this point won’t help anyway, considering the amount of information we have (basically none).

We don’t know that.

But let’s recap: you were using 2.5.1 which had the problem and then you upgrade to 2.5.2 which also had this problem, is that correct?

Yes, the problem occurred with 2.5.1 and 2.5.2.
A colleague just reported a failure with 2.4.13.

Feb 18 09:48:28 149.97.134.35:50483 [18/Feb/2022:09:48:27.989] main_ssl~ ssl_backend-vr/<NOSRV> 816/-1/-1/-1/815
302 86 - - LRNN 1/1/0/0/3 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"

Feb 18 09:48:28 149.97.134.35:50483 [18/Feb/2022:09:48:28.805] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 37/0/0/4/46 
200 38818 - - --NI 1/1/0/0/0 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/le5/ HTTP/2.0"

My logs go back to Feb 14.
My 2 HAProxy servers share the same haproxy.cfg configuration file.
I just scanned the logs of the first HAProxy server and find 0 instances of a misdirected request.
Only the 2nd proxy server’s log shows failures.

Something interesting.

On the first proxy server, every instance of empty hostname {} occurs where
the selected backend has no server:

Feb 18 08:45:38 167.99.133.28:36388 [18/Feb/2022:08:45:38.768] main_ssl~
ssl_backend-vr/<NOSRV> 0/-1/-1/-1/0
403 192 - - PRNN 1/1/0/0/3 0/0
{}
"GET / HTTP/1.1"

On the 2nd proxy server, there are many instances of empty hostname {} where
the selected backend is a real server, e.g.

Feb 18 15:55:35 163.114.224.3:59017 [18/Feb/2022:15:55:34.909] main_ssl~
ssl_backend-vr/i-0548140935b42ced2 0/0/1/430/431
200 123 - - --NI 1/1/0/0/0 0/0
{}
"GET /le5/hcs/all HTTP/1.1"

The first proxy server has no instances of misdirected requests.
They all appear in the logs of the 2nd proxy server.
To reiterate, both proxy servers share the same haproxy.cfg file.

Here a deny statements catches a request that you don’t want, as per your configuration, therefor haproxy returns a 403 Forbidden to the client. No backend server is selected because the HTTP request is denied anyway.

Nothing wrong with that.

If the release and the config is the same, maybe what is not the same are the requests from clients? We have been unable to confirm any haproxy misbehavior at this point.

I’m afraid we may need to capture the SSL traffic on proxy2 and decrypt it, to understand whether the host header is present in the request or not. Not sure how we would find out what actually happens here otherwise.

On February 20, I upgraded the O/S on my 2 HAProxy servers from RockyLinux 8.4 to 8.5. This also upgraded openssl to openssl-1.1.1k-5.el8_5.x86_64. It also deployed HAProxy 2.5.2.
There has not been a single case of misdirected URL since the upgrade.