Socket error on adding ALPN

Hello,

My 1st topic here, so bear with me :slight_smile:

My current (working) setup is fully end2end encrypted.
Meaning my frontend does TLS offloading, and then reencypts all traffic to the backend.

Another thing we do is rewrite the host headers to whatever request url was sent to the server. All of this works great.

Now what happens is that to the frontend all traffic is either HTTP/2 or lower.
To the backend (IIS 10 on Server 2022) all traffic is Http 1.1 (or lower).

I wanted to see if HTTP/2 to the backend is a possibility and check what the impact would be on our webservers/ application.

So what I tried was adding alpn h2,h1 to a server line. However this immediately results in a Socket error for that server

I have verified that on the IIS webserver connecting to localhost, I do get HTTP/2 connections. What I’ve also tried was disabling the Healthchecks from HAProxy and force the site to be up. That also works, and then traffic is then coming in on IIS as HTTP/2.

So I figured it might have something to do with the healthcheck…

For reference this is my config (sanitized).

global
 maxconn 10000
 log /dev/log local0
 chroot /var/lib/haproxy
 stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
 stats timeout 30s
 user haproxy
 group haproxy
 daemon

 # Default SSL material locations
 ca-base /etc/ssl/certs
 crt-base /etc/ssl/private

 # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
 ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
 ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
 log global
 mode http
 option httpslog
 option dontlognull
    timeout connect 10s
    timeout client  300s
    timeout server  300s
 errorfile 400 /etc/haproxy/errors/400.http
 errorfile 403 /etc/haproxy/errors/403.http
 errorfile 408 /etc/haproxy/errors/408.http
 errorfile 500 /etc/haproxy/errors/500.http
 errorfile 502 /etc/haproxy/errors/502.http
 errorfile 503 /etc/haproxy/errors/503.http
 errorfile 504 /etc/haproxy/errors/504.http

frontend fe_main_tcp
 mode tcp
 option tcp-smart-accept
 option tcplog
 bind :80,:443
 tcp-request inspect-delay 10s
 tcp-request content track-sc0 capture.req.ver table st_http_stats
 default_backend be_maintcp

backend be_maintcp
 mode tcp
 option tcp-smart-connect
 server fe_acceptweb abns@haproxy send-proxy-v2

frontend fe_acceptweb
 mode http
 option tcp-smart-accept
 maxconn 1000
 option httpslog
 bind abns@haproxy ssl crt /etc/haproxy/certs ssl-min-ver TLSv1.2 alpn h2 accept-proxy

 option forwardfor
 acl allowed_method method GET HEAD POST PUT DELETE OPTIONS SSTP_DUPLEX_POST

 http-request redirect scheme https unless { ssl_fc }
 http-request set-header X-Forwarded-Port 443
 http-request set-header X-Forwarded-Proto https

 default_backend be_web

backend be_web
 option tcp-smart-connect
 option httpchk
 http-check send hdr Host webserver.local

 default-server ssl verify none check-sni webserver.local check sni str(webserver.local) on-marked-down shutdown-sessions

 http-request set-header X-Forwarded-Host %[req.hdr(Host)]
 http-request set-header Host webserver.local

 server WEB01 10.0.1.2:443 

So something is happening that I don’t understand, I figured adding alpn should be enough but that does not seem to work as I had expected. Since IIS seems to be able to serve HTTP2 I am leaning to an issue with HAProxy.
I used the Haproxy version 2.6 and 2.5.7 to check, but both have the same issue.

Hope someone is familiar with or has ran into a similar issue

Could you re-enable checks and start HAProxy in debug mode and run some clients against it and reply here with the output?

So when I change this line

server WEB01 10.0.1.2:443 

to
server WEB01 10.0.1.2:443 proto h2 alpn h2
(without proto h2 alpn h2 everything works)

And I run Haproxy in debug mode I get this

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result FAILED
Total: 3 (2 usable), will use epoll.

Available filters :
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
Using epoll() as the polling mechanism.
[WARNING] (72001) : Server be_web/WEB01 is DOWN, reason: Layer7 wrong status, code: 400, check duration: 9ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[ALERT] (72001) : backend ‘be_web’ has no server available!
00000002:fe_main_tcp.accept(0005)=000c from [sanitizedip:51997] ALPN=
00000003:fe_acceptweb.accept(0008)=000e from [sanitizedip:51997] ALPN=h2
00000003:fe_acceptweb.clireq[000e:ffffffff]: GET https://sanitized.url/ HTTP/2.0
00000003:fe_acceptweb.clihdr[000e:ffffffff]: host: sanitized.url
00000003:fe_acceptweb.clihdr[000e:ffffffff]: cache-control: max-age=0
00000003:fe_acceptweb.clihdr[000e:ffffffff]: sec-ch-ua: " Not A;Brand";v=“99”, “Chromium”;v=“102”, “Google Chrome”;v=“102”
00000003:fe_acceptweb.clihdr[000e:ffffffff]: sec-ch-ua-mobile: ?0
00000003:fe_acceptweb.clihdr[000e:ffffffff]: sec-ch-ua-platform: “Windows”
00000003:fe_acceptweb.clihdr[000e:ffffffff]: dnt: 1
00000003:fe_acceptweb.clihdr[000e:ffffffff]: upgrade-insecure-requests: 1
00000003:fe_acceptweb.clihdr[000e:ffffffff]: user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.115 Safari/537.36
00000003:fe_acceptweb.clihdr[000e:ffffffff]: accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,–sanitized headers–
00000003:be_web.clicls[000e:ffff]
00000003:be_web.closed[000e:ffff]

Have to sanitize some of the headers…
The used webclient won’t matter I guess, as the health check itself will just fail. And down the webserver.

I have an almost identical config running on another setup, using IIS 10 on Windows Server 2019. Same versions tested, and both work just fine.

The only difference in that configuration is that another url is used… (double checked)

So that would make me expect IIS is the culprit, however without HAProxy health checks it works and IIS logs tell me it is serving HTTP/2 content, also without HAProxy it also serves HTTP/2 content. Weird right?

If I comment these 2 lines and I change this line

server WEB01 10.0.1.2:443 ssl alpn h2

Everything just works, so I suppose it has to be an issue with HaProxy. Im guessing it’s some kind of bug… As I said I have an identical working solution. IIS reports outgoing h2 responses etc. So it can work…

So I figured it out,
It appears the httpchk option does not support HTTP/2
You need to use : check-alpn http/1.1

This would be a GREAT addition to the documentation people.
Perhaps consider an edit there. specifically to the option httpchk.

I am sure people here are doing their best to help people, and I do appreciate that. But as a new person here, I felt quite abandoned. Hope there will be some changes in that regard. (not talking about the newbies that just come here to have their problem fixed( one google search away)).