I’ve been using HAProxy 1.7 to front Obsidian successfully for many years.
I’ve deployed 2 new HAProxy 2.51 servers in a development environment.
When it works correctly the logs look like this:
Feb 9 15:09:13 149.97.134.35:22159 [09/Feb/2022:15:08:42.499] main_ssl~ ssl_backend-obs/<NOSRV> 30601/-1/-1/-1/30600
302 90 - - LR-- 1/1/0/0/3 0/0 {} "GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb 9 15:09:13 149.97.134.35:22159 [09/Feb/2022:15:09:13.098] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 74/0/0/1/74
302 108 - - ---- 1/1/0/0/0 0/0 {} "GET https://tndci-obs.cloud.micropact.com/obsidian HTTP/2.0"
tndci-obs.cloud.micropact.com is sent via 302 to the ssl_backend-obs and “/obsidian” is appended to the path.
When it fails, the logs look like this:
Feb 9 13:26:24 209.112.9.82:65220 [09/Feb/2022:13:26:23.879] main_ssl~ ssl_backend-vr/<NOSRV> 1/-1/-1/-1/191
302 86 - - LRNN 3/3/0/0/3 0/0 {} "GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb 9 13:26:24 209.112.9.82:65220 [09/Feb/2022:13:26:24.071] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 56/0/0/5/67
200 38818 - - --VN 3/3/0/0/0 0/0 {} "GET https://tndci-obs.cloud.micropact.com/le5/ HTTP/2.0"
tndci-obs.cloud.micropact.com is sent to the default backend ssl_backend-vr and “/le5” is appended to the path.
Since this is HTTP 2.0 the URL is absolute and it clearly contains “tndci-obs”.
Frontend:
acl obs hdr_dom(host) tndci-obs.
use_backend ssl_backend-obs if obs
Backend:
backend ssl_backend-obs
balance roundrobin
option httpchk GET /obsidian/
http-check disable-on-404
acl down nbsrv() eq 0
acl path_root path /
http-response set-header Cache-Control no-cache,must-revalidate
http-response set-header Pragma no-cache
http-response set-header Expires 0
http-request allow if { src -f /etc/CONFIG/haproxy/whitelist.lst } || { ssl_c_used }
http-request deny
redirect location /obsidian if path_root !down
I assume the frontend occasionally fails to match.
Provide the full configuration please.
1 Like
# ---------------------------------------------------------------------
# See the full configuration options online:
# https://cbonte.github.io/haproxy-dconv/2.4/configuration.html
# ---------------------------------------------------------------------
# includes from /etc/CONFIG/haproxy/conf.d/
global
# To have these messages end up in /var/log/haproxy.log you will
# need to:
#
# 1) configure syslog to accept network log events. This is done
# by adding the '-r' option to the SYSLOGD_OPTIONS in
# /etc/sysconfig/syslog
#
# 2) configure local2 events to go to the /var/log/haproxy.log
# file. A line like the following can be added to
# /etc/sysconfig/syslog
#
# local2.* /var/log/haproxy.log
#
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
user haproxy
group haproxy
master-worker
nbthread 4
daemon
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-bind-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256
# ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECD
HE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:!AECDH
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256
stats socket /var/lib/haproxy/stats mode 600 level admin
stats timeout 2m
tune.ssl.default-dh-param 2048
userlist UsersFor_Java
user java insecure-password xxxxx
userlist UsersFor_Logi
user logi insecure-password xxxxxx
defaults
default_backend ssl_backend-vr
errorfile 408 /dev/null
log global
maxconn 5000
mode http
option abortonclose
option dontlognull
option forwardfor except 127.0.0.0/8
option httplog
option http-ignore-probes
option redispatch
retries 3
timeout check 10s
timeout client 1m
timeout connect 10s
timeout http-keep-alive 10s
timeout http-request 10s
timeout queue 1m
timeout server 5m
peers mypeers
# include hap_servers-haproxy declarations
peer ip-10-160-17-161 10.160.17.161:1024
peer ip-10-160-17-64 10.160.17.64:1024
# include frontend declarations
frontend main
bind 0.0.0.0:80
acl lvr hdr_dom(host) lvr.
acl ltndci-vr hdr_dom(host) ltndci-vr.
acl solr hdr_dom(host) ltndci-solr. ltndci-access.
acl search hdr_dom(host) ltndci-search.
acl apigw hdr_dom(host) ltndci-apigw.
acl ci hdr_dom(host) ci.
http-request redirect scheme https code 301 if !{ ssl_fc } !lvr !ltndci-vr !solr !search !apigw !ci
use_backend backend-solr if solr
# use_backend backend-search if search
# use_backend backend-apigw if apigw
use_backend backend-ci if ci
frontend main_ssl
# Bind SSL port with PFS-enabling cipher suite
# bind :443 ssl crt /etc/CONFIG/haproxy/certs/cert.pem alpn h2,http/1.1
bind :443 ssl crt /etc/CONFIG/haproxy/certs/cert.pem
# bind :8443 ssl crt /etc/CONFIG/haproxy/certs/cert.pem ca-file /apps/Config/certs/clients.cert.pem verify optional crt-ignore-err all
http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port 443 if { ssl_fc }
capture request header X-Forwarded-For len 15
# Rate limit
stick-table type ip size 100k expire 30s store http_req_rate(10s)
http-request track-sc0 src
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 150 } !{ src -f /etc/CONFIG/haproxy/cidr.lst }
# Distinguish between secure and insecure requests
acl secure dst_port eq 443
# Filter nasty input
option http-buffer-request
acl log4shell url,url_dec -m reg \${[^}]*\${
acl log4shell url,url_dec -m reg \${jndi:(?:ldaps?|iiop|dns|rmi)://
acl log4shell url,url_dec -i -m reg \${[\w${}\-:]*j[\w${}\-:]*n[\w${}\-:]*d[\w${}\-:]*i[\w${}\-:]*:.*}
acl log4shell req.hdrs -m reg \${[^}]*\${
acl log4shell req.hdrs -m reg \${jndi:(?:ldaps?|iiop|dns|rmi)://
acl log4shell req.hdrs -i -m reg \${[\w${}\-:]*j[\w${}\-:]*n[\w${}\-:]*d[\w${}\-:]*i[\w${}\-:]*:.*}
acl log4shell_form req.body,url_dec -m reg \${[^}]*\${
acl log4shell_form req.body,url_dec -m reg \${jndi:(?:ldaps?|iiop|dns|rmi)://
acl log4shell_form req.body,url_dec -i -m reg \${[\w${}\-:]*j[\w${}\-:]*n[\w${}\-:]*d[\w${}\-:]*i[\w${}\-:]*:.*}
http-request deny if log4shell
http-request deny if { req.fhdr(content-type) -m str application/x-www-form-urlencoded } log4shell_form
acl missing_cl hdr_cnt(Content-length) eq 0
acl METH_PUT method PUT
acl METH_GET method GET HEAD
acl METH_PATCH method PATCH
acl METH_DELETE method DELETE
http-request deny if HTTP_URL_STAR !METH_OPTIONS || METH_PUT missing_cl
http-request deny if METH_GET HTTP_CONTENT
http-request deny unless METH_GET or METH_POST or METH_OPTIONS or METH_PATCH or METH_DELETE or METH_PUT
# Mark all cookies as secure if sent over SSL
http-response replace-header Set-Cookie (.*)[Hh]ttp[Oo]nly;?(.*) \1\2 if secure
http-response replace-header Set-Cookie (.*)[Ss]ecure;?(.*) \1\2 if secure
http-response replace-header Set-Cookie (.*) \1;HttpOnly;Secure; if secure
# http-response replace-header Set-Cookie (.*) \1;HttpOnly;Secure;sameSite=strict; if secure
# Add the HSTS header with a 1 year max-age
http-response set-header Strict-Transport-Security max-age=31536000;includeSubDomains;preload if secure
# Add cache headers
# http-response set-header Cache-Control no-cache,must-revalidate
# http-response set-header Pragma no-cache
# http-response set-header Expires 0
# Add additional security headers
http-response set-header Referrer-Policy no-referrer-when-downgrade
http-response set-header X-Frame-Options SAMEORIGIN
http-response set-header Content-Security-Policy "frame-ancestors 'self';"
http-response set-header Access-Control-Allow-Origin *
http-response set-header X-XSS-Protection 1;report=https://micropact.report-uri.io/r/d/xss/enforce
http-response set-header X-Content-Type-Options nosniff
# http-response set-header Feature-Policy "vibrate self"
http-response set-header Expect-CT max-age=0;report=https://micropact.report-uri.io/r/d/ct/reportOnly
http-response set-header Expect-Staple report=https://micropact.report-uri.io/r/d/staple/reportOnly
http-response set-header Report-To {"group":"default","max_age":31536000,"endpoints":[{"url":"https://micropact.report-uri.com/a/d/g"}],"include_subdomains":true}
http-response set-header NEL {"report_to":"default","max_age":31536000,"include_subdomains":true}
http-response del-header Server
http-response del-header X-Powered-By
http-response del-header X-Runtime
# include front_ssl declarations
acl search hdr_dom(host) tndci-access.
acl apigw path_beg /apigw
acl solr path_beg /solr
use_backend backend-solr if solr
use_backend ssl_backend-apigw if apigw
use_backend ssl_backend-search if search
acl idm hdr_beg(host) -i tndci-idm.
use_backend ssl_backend-idm if idm
acl logi hdr_dom(host) logi.
use_backend ssl_backend-logi if logi
acl logitest hdr_dom(host) logitest.
use_backend ssl_backend-logitest if logitest
acl lrs hdr_dom(host) ltndci-rs.
use_backend ssl_backend-rs if lrs
acl lvr hdr_dom(host) tndci-lvr.
use_backend ssl_backend-lvr if lvr
acl obs hdr_dom(host) tndci-obs.
use_backend ssl_backend-obs if obs
acl rs hdr_dom(host) tndci-rs.
use_backend ssl_backend-rs if rs
acl vo hdr_dom(host) tndci-vo.
use_backend ssl_backend-vo if vo
acl vomaster hdr_dom(host) vomaster.
use_backend ssl_backend-vomaster if vomaster
acl vopreprod hdr_dom(host) tndci-vopreprod.
use_backend ssl_backend-vopreprod if vopreprod
acl votest hdr_dom(host) votest.
use_backend ssl_backend-votest if votest
acl vr hdr_dom(host) tndci-vr.
use_backend ssl_backend-vr if vr
acl vrmaster hdr_dom(host) vrmaster.
use_backend ssl_backend-vrmaster if vrmaster
acl vrpreprod hdr_dom(host) tndci-vrpreprod.
use_backend ssl_backend-vrpreprod if vrpreprod
acl vrtest hdr_dom(host) vrtest.
use_backend ssl_backend-vrtest if vrtest
# include backend declarations
backend backend-ci
balance roundrobin
option httpchk GET /login
http-check disable-on-404
http-request allow if { src -f /etc/CONFIG/haproxy/ci-whitelist.lst }
http-request deny
# acl path_root path /
# redirect location /rashome.aspx if path_root
# include servers-ci declarations
backend backend-solr
balance roundrobin
option httpchk GET /solr/versa/admin/ping
http-check disable-on-404
cookie SRV insert indirect nocache httponly secure
# include servers-solr declarations
server i-076e2e764968716eb-9034 10.160.17.160:9034 check cookie i-076e2e764968716eb-9034
# include back_ssl declarations
backend ssl_backend-apigw
balance roundrobin
option httpchk GET /
http-check disable-on-404
cookie SRV insert indirect nocache httponly secure
# include ssl_servers-apigw declarations
server i-076e2e764968716eb-9054 10.160.17.160:9054 check cookie i-076e2e764968716eb-9054
backend ssl_backend-idm
balance roundrobin
option httpchk GET /IronDataMobile/about.txt
http-check disable-on-404
cookie SRV insert indirect nocache httponly secure
acl path_form path_beg /form
acl path_root path /
redirect location /IronDataMobile/formBuilder/index-prod.html if path_form
redirect location /IronDataMobile/latest/index-cache.html if path_root
# include ssl_servers-idm declarations
server i-07837e30d48328986 10.160.17.51:9003 check cookie i-07837e30d48328986
backend ssl_backend-lrs
balance roundrobin
option httpchk GET /entellitrak/web-pub/udm/images/right-msdn.gif
http-check disable-on-404
cookie SRV insert indirect nocache httponly secure
acl path_root path /
redirect location /entellitrak if path_root
# include ssl_servers-rs declarations
server i-0f4cc998ff5c18cf4 10.160.17.42:9006 check cookie i-0f4cc998ff5c18cf4
backend ssl_backend-lvr
balance roundrobin
option httpchk GET /le5/about.txt
http-check disable-on-404
http-request allow if { src -f /etc/CONFIG/haproxy/whitelist.lst } || { ssl_c_used }
http-request deny
acl path_root path /
redirect location /le5/ if path_root
# include ssl_servers-vr declarations
server i-0548140935b42ced2 10.160.17.28:9001 check cookie i-0548140935b42ced2
backend ssl_backend-obs
balance roundrobin
option httpchk GET /obsidian/
http-check disable-on-404
acl down nbsrv() eq 0
acl path_root path /
http-response set-header Cache-Control no-cache,must-revalidate
http-response set-header Pragma no-cache
http-response set-header Expires 0
http-request allow if { src -f /etc/CONFIG/haproxy/whitelist.lst } || { ssl_c_used }
http-request deny
redirect location /obsidian if path_root !down
# include servers-obs declarations
server i-07507641ad1b1365c 10.160.17.17:9016 check cookie i-07507641ad1b1365c
backend ssl_backend-rs
balance roundrobin
option httpchk GET /entellitrak/web-pub/udm/images/right-msdn.gif
http-check disable-on-404
cookie SRV insert indirect nocache httponly secure
acl path_root path /
redirect location /entellitrak if path_root
# include ssl_servers-rs declarations
server i-0f4cc998ff5c18cf4 10.160.17.42:9006 check cookie i-0f4cc998ff5c18cf4
backend ssl_backend-search
balance roundrobin
option httpchk GET /locales/en/locale.json
http-check disable-on-404
cookie SRV insert indirect nocache httponly secure
# include ssl_servers-search declarations
server i-076e2e764968716eb-9064 10.160.17.160:9064 check cookie i-076e2e764968716eb-9064
backend ssl_backend-vo
balance roundrobin
option httpchk GET /datamart/about.txt
http-check disable-on-404
cookie SRV insert indirect nocache httponly secure
acl path_status path /status
acl AuthOkay_Java http_auth(UsersFor_Java)
acl path_admin path_beg /admin /staff
acl path_root path /
acl path_data path /datamart
http-request replace-path /datamart /datamart/languageChoice.do if path_data
http-request replace-path /(admin|staff)(.*) /datamart/\1/languageChoice.do\2 if path_admin
http-request auth realm Java if path_status !AuthOkay_Java
http-request allow if { src -f /etc/CONFIG/haproxy/vo-whitelist.lst } || { ssl_c_used }
http-request deny
redirect location /datamart/languageChoice.do if path_root
# include ssl_servers-vo declarations
server i-07837e30d48328986 10.160.17.51:9002 check cookie i-07837e30d48328986
backend ssl_backend-vr
balance roundrobin
option httpchk GET /le5/about.txt
http-check disable-on-404
cookie SRV insert indirect nocache httponly secure
acl path_status path /status
acl AuthOkay_Java http_auth(UsersFor_Java)
acl path_root path /
http-request auth realm Java if path_status !AuthOkay_Java
http-request allow if { src -f /etc/CONFIG/haproxy/vr-whitelist.lst } || { ssl_c_used }
http-request deny
redirect location /le5/ if path_root
# include ssl_servers-vr declarations
server i-0548140935b42ced2 10.160.17.28:9001 check cookie i-0548140935b42ced2
# include listen declarations
listen stats
bind 0.0.0.0:9999
mode http
balance
timeout client 5000
timeout connect 4000
timeout server 30000
stats hide-version
stats uri /haproxy_stats
stats auth admin: xxxxx
# include resolvers declarations
resolvers mydns
parse-resolv-conf
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold other 30s
hold refused 30s
hold nx 30s
hold timeout 30s
hold valid 10s
hold obsolete 30s
# include proxy declarations
Speedy
February 12, 2022, 4:43pm
4
have you tired making your match case-insensitive or using hdr_beg instead of hdr_dom since you are matching the subdomain?
I assume because the pattern always ended in a “.”, hdr_dom worked correctly for the last 7 years.
acl obs hdr_dom(host) tndci-obs.
use_backend ssl_backend-obs if obs
To:
acl obs hdr_beg(host) -i tndci-obs.
use_backend ssl_backend-obs if obs
After making this change, I wasn’t able to repeat the problem for 3 days but now it has reared its ugly head again!
Log extract from Feb 15 around 16:05 CST:
Feb 15 16:05:58 149.97.134.35:65389 [15/Feb/2022:16:05:58.741] main_ssl~ ssl_backend-obs/<NOSRV> 0/-1/-1/-1/0 302 90 - - LR-- 1/1/0/0/3 0/0 {} "GET / HTTP/1.1" -
Feb 15 16:05:58 149.97.134.35:65389 [15/Feb/2022:16:05:58.813] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/1/0/1 302 108 - - ---- 1/1/0/0/0 0/0 {} "GET /obsidian HTTP/1.1" -
Feb 15 16:05:58 149.97.134.35:65389 [15/Feb/2022:16:05:58.940] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/1/1 304 111 - - ---- 1/1/0/0/0 0/0 {} "GET /obsidian/ HTTP/1.1" -
Feb 15 16:05:59 149.97.134.35:65389 [15/Feb/2022:16:05:59.301] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/7/7 304 110 - - ---- 1/1/0/0/0 0/0 {} "GET /obsidian/assets/custom.css HTTP/1.1" -
Feb 15 16:05:59 149.97.134.35:65388 [15/Feb/2022:16:05:58.638] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 582/0/0/6/690 404 231 - - --NI 1/1/0/0/0 0/0 {} "GET https://tndci-obs.cloud.micropact.com/obsidian/assets/main.js HTTP/2.0"
The 1st line detects the correct backend ssl_backend-obs and does a 302 redirect to the correct server.
Speedy
February 16, 2022, 4:39pm
6
by best guess is that health check takes the backend offline, which is then causing haproxy to select the default backend
The Obsidian backend is UP and doesn’t flap:
[TNDCI-LB-64]# $CBIN/haproxy-state
# pxname,svname,status
main,FRONTEND,OPEN
main_ssl,FRONTEND,OPEN
ssl_backend-apigw,i-076e2e764968716eb-9054,UP
ssl_backend-idm,i-07837e30d48328986,UP
ssl_backend-lrs,i-0f4cc998ff5c18cf4,UP
ssl_backend-lsolr,i-076e2e764968716eb-9034,UP
ssl_backend-lvr,i-0548140935b42ced2,UP
ssl_backend-obs,i-07507641ad1b1365c,UP
ssl_backend-rs,i-0f4cc998ff5c18cf4,UP
ssl_backend-search,i-076e2e764968716eb-9064,UP
ssl_backend-solr,i-076e2e764968716eb-9034,UP
ssl_backend-vo,i-07837e30d48328986,UP
ssl_backend-vopreprod,i-08995fc9329e978e0,UP
ssl_backend-vr,i-0548140935b42ced2,UP
ssl_backend-vrpreprod,i-08995fc9329e978e0,UP
stats,FRONTEND,OPEN
Speedy
February 16, 2022, 8:32pm
8
your backend configs are different than what I’ve used in the past. I’ve never seen the use of nbsrv on the backend. Backends are more complicated than what I’ve ever used!
perhaps try taking out your monitor or moving it to the frontend. I’m out of ideas. Hopefully someone with more experience will be able to help!
You configuration is so complex, it’s very hard to understand what it’s supposed to do. With a configuration that complex you should probably avoid run bleeding edge haproxy releases.
I’d suggest running LTS release trains instead.
The host header is not constructed out of the absolute URI though. Host header is host header. So I’d suggest you start logging the host header together with the rest.
So in frontend main_ssl, let’s add Host header logging:
http-request capture req.hdr(Host) len 30
Then check your logs again and let’s see what the Host header actually contains.
I added the following:
http-request capture req.hdr(Host) len 100
Now when the request works correctly I see the host between the “{}”:
Feb 17 10:37:33 209.112.9.82:65114 [17/Feb/2022:10:37:33.232] main_ssl~ ssl_backend-obs/<NOSRV> 0/-1/-1/-1/0 302 90 - - LR-- 6/6/0/0/3 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb 17 10:37:33 209.112.9.82:65114 [17/Feb/2022:10:37:33.274] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/1/0/1
302 108 - - ---- 6/6/0/0/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian HTTP/2.0"
Feb 17 10:37:33 209.112.9.82:65114 [17/Feb/2022:10:37:33.310] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/1/1
200 965 - - ---- 6/6/0/0/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian/ HTTP/2.0"
Feb 17 10:37:33 209.112.9.82:65114 [17/Feb/2022:10:37:33.590] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/6/6
304 110 - - ---- 6/6/1/1/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian/assets/custom.css HTTP/2.0"
When the request is sent to the default backend the host is missing:
Feb 17 10:30:43 209.112.9.82:64653 [17/Feb/2022:10:30:43.115] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 2/0/0/1/37
404 312 - - --NI 1/1/0/0/0 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/obsidian/ HTTP/2.0"
Feb 17 10:30:43 209.112.9.82:64653 [17/Feb/2022:10:30:43.153] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 200/0/0/0/199
200 1613 - - --VN 1/1/0/0/0 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/favicon.ico HTTP/2.0"
This is a though one to troubleshoot, because this is all hidden in SSL, so we can’t confirm easily what happens.
Is this happening in different browsers/clients, or is this a particular client?
I’d suggest to use a different haproxy release here. 2.5.1 has some bugs, but with the information available it is difficult to pin point to a specific bug.
I suggest you try 2.4.13 first of all.
The alternative is to setup SSL key logging tune.ssl.keylog and capture and decrypt the SSL connection and analyze the HTTP transaction.
I was running 2.5.2.
Feb 17 16:59:40 209.112.9.82:57773 [17/Feb/2022:16:59:40.293] main_ssl~ ssl_backend-obs/<NOSRV> 0/-1/-1/-1/0
302 90 - - LRNN 1/1/0/0/3 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb 17 16:59:40 209.112.9.82:57773 [17/Feb/2022:16:59:40.318] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/1/0/1
302 108 - - --NI 1/1/0/0/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian HTTP/2.0"
Feb 17 16:59:40 209.112.9.82:57773 [17/Feb/2022:16:59:40.443] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/0/2/2
200 965 - - --VN 1/1/0/0/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian/ HTTP/2.0"
Feb 17 16:59:40 209.112.9.82:57773 [17/Feb/2022:16:59:40.595] main_ssl~ ssl_backend-obs/i-07507641ad1b1365c 0/0/1/5/6
304 114 - - --VN 1/1/1/1/0 0/0
{tndci-obs.cloud.micropact.com}
"GET https://tndci-obs.cloud.micropact.com/obsidian/assets/main.js HTTP/2.0"
Firefox browser instance failed:
Feb 17 17:02:06 209.112.9.82:57996 [17/Feb/2022:17:02:06.554] main_ssl~ ssl_backend-vr/<NOSRV> 2/-1/-1/-1/63
302 86 - - LRNN 1/1/0/0/3 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb 17 17:02:06 209.112.9.82:57996 [17/Feb/2022:17:02:06.617] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 83/0/0/5/94
200 38818 - - --VN 1/1/0/0/0 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/le5/ HTTP/2.0"
30 minutes later a refresh on Firefox browser renders site correctly.
I haven’t had any failures all morning.
You have been unable to reproduce the issue before for 3 days after making a small change, so I’d say that not hitting any failures for a single morning is not indicative of having found a proper workaround.
Regarding a bug report, we pretty much don’t know anything at all at this point. I’d suggest we make really sure that 2.4.13 works long-term, before making a bug reports. A bug report at this point won’t help anyway, considering the amount of information we have (basically none).
We don’t know that.
But let’s recap: you were using 2.5.1 which had the problem and then you upgrade to 2.5.2 which also had this problem, is that correct?
Yes, the problem occurred with 2.5.1 and 2.5.2.
Feb 18 09:48:28 149.97.134.35:50483 [18/Feb/2022:09:48:27.989] main_ssl~ ssl_backend-vr/<NOSRV> 816/-1/-1/-1/815
302 86 - - LRNN 1/1/0/0/3 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/ HTTP/2.0"
Feb 18 09:48:28 149.97.134.35:50483 [18/Feb/2022:09:48:28.805] main_ssl~ ssl_backend-vr/i-0548140935b42ced2 37/0/0/4/46
200 38818 - - --NI 1/1/0/0/0 0/0
{}
"GET https://tndci-obs.cloud.micropact.com/le5/ HTTP/2.0"
My logs go back to Feb 14.
Something interesting.
On the first proxy server, every instance of empty hostname {} occurs whereno server:
Feb 18 08:45:38 167.99.133.28:36388 [18/Feb/2022:08:45:38.768] main_ssl~
ssl_backend-vr/<NOSRV> 0/-1/-1/-1/0
403 192 - - PRNN 1/1/0/0/3 0/0
{}
"GET / HTTP/1.1"
On the 2nd proxy server, there are many instances of empty hostname {} where
Feb 18 15:55:35 163.114.224.3:59017 [18/Feb/2022:15:55:34.909] main_ssl~
ssl_backend-vr/i-0548140935b42ced2 0/0/1/430/431
200 123 - - --NI 1/1/0/0/0 0/0
{}
"GET /le5/hcs/all HTTP/1.1"
The first proxy server has no instances of misdirected requests.
Here a deny statements catches a request that you don’t want, as per your configuration, therefor haproxy returns a 403 Forbidden to the client. No backend server is selected because the HTTP request is denied anyway.
Nothing wrong with that.
If the release and the config is the same, maybe what is not the same are the requests from clients? We have been unable to confirm any haproxy misbehavior at this point.
I’m afraid we may need to capture the SSL traffic on proxy2 and decrypt it, to understand whether the host header is present in the request or not. Not sure how we would find out what actually happens here otherwise.
On February 20, I upgraded the O/S on my 2 HAProxy servers from RockyLinux 8.4 to 8.5. This also upgraded openssl to openssl-1.1.1k-5.el8_5.x86_64. It also deployed HAProxy 2.5.2.