Some requests aborted when connecting to http/2 backend

Help!
1

Hi,

Since a long time I’m using haproxy (as a package on pfsense, HAProxy version 2.4.9-f8dcd9f, released 2021/11/24) to handle incoming requests to my homelab environment. Recently I upgraded Tomcat to version 10 on one of by backends and also upgraded a few server running IIS from W2K12 to W2K19.

Since that moment I noticed that some requests were aborted. After doing some more investigation it looks like to problem is related to the combination of http/2 on the backend (which was introduced as a “default” after upgrade to Tomcat10 and W2K19) and using the transparant client ip mode in haproxy.

So I always used the following settings for one of my applations (reduced the config for readability to only the frontend and backend I’m currently testing with):

global
	maxconn			100
	log			srv-nas-01.home.mydomain.com	local0	info
	stats socket /tmp/haproxy.socket level admin  expose-fd listeners
	gid			80
	nbproc			1
	nbthread			1
	hard-stop-after		15m
	chroot				/tmp/haproxy_chroot
	daemon
	log-send-hostname		srv-fw-01-haproxy
	server-state-file /tmp/haproxy_server_state

listen HAProxyLocalStats
	bind 127.0.0.1:2200 name localstats
	mode http
	stats enable
	stats admin if TRUE
	stats show-legends
	stats uri /haproxy/haproxy_stats.php?haproxystats=1
	timeout client 5000
	timeout connect 5000
	timeout server 5000

frontend SharedFrontEnd-merged
	bind			192.168.1.1:443 name 192.168.1.1:443   
	bind			192.168.7.2:443 name 192.168.7.2:443   
	mode			tcp
	log			global
	timeout client		30000
	tcp-request inspect-delay	5s
	acl			host_login	req.ssl_sni -i login.mydomain.com
	tcp-request content accept if { req.ssl_hello_type 1 }
	use_backend OTDS_ipv4  if  host_login 
	
backend OTDS_ipv4
	mode			tcp
	id			10119
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	source ipv4@ usesrc clientip
	server			srv-otds-01_https 192.168.48.10:443 id 10121

When I load a page containing quite a lot of javascript files and other files, and caching is disabled so I force the browser to load all requests, let’s day 80% of the requests succeed.

image
image1721×749 75.5 KB

On a reload (with cache disabled) I see same behaviour, althought not exactly the same requests will fail, most times it are almost the same files. Opening these files individually is no problem. In the Apache log I see the requests are handled correctly.

If I remove the setting UpgradeProtocol from the Tomcat config, only 1 request out of about 300 requests for this page fails. Also not always the same.

image
image1016×141 12.5 KB

When I remove the line source ipv4@ usesrc clientip from the backend configuration it works without any problem. It then runs on http/2.0 according to the Tomcat access logs.

Initially my assumption was this only happens on my Tomcat server, but now I see common behavior on my IIS server as well and I guess since the update to W2K19 IIS is also working with http/2.

I’m a bit lost in what is/could be causing this issue. Of course I could just disable the userc clientip setting, but I like this as it gives me the source ips and it should work, isn’t it?

Any suggestions/ideas?