Haproxy Domain Based Routing with HTTP/2

Hi

I use Haproxy with SSL Termination in a LXC Container and it works great. Most Backends listen on pot 80 since i dont want to go through the hassle to manage a letsencrypt certificate on each container and personaly, i think there is no point in encrypting connections between containers.

I wanted to enable http2 on my configuration, but i cant get it to work while still being able to route the traffic to the particular backends. By far the most Tutorials you find online create one backend for http/1.1 and a second one for http2, but they never Route the Traffic to a Container with for example

acl example1_host hdr(host) -i example1.example.com

They just differentiate between http and http2

Of i add alpn h2,http/1.1 to the bind command the site wont load, if i add http/1.1,h2 it loads, but with http/1.1. It doesnt make a difference if i add send-proxy to the backends,

In my haproxy.log i see

[07/Aug/2019:14:58:13.861] web_frontend~ example1/example1 0/0/0/0/0 406 4755 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/2.0"

but Google Chrome shows me a ERR_SPDY_PROTOCOL_ERROR Error.

Can someone give me a tip how to make this happen ? Is this possible at all and if yes, is it possible to have the frontend speak http2 with a backend communicating with http/1.1 ?

Here is a example of my Configuration

frontend web_frontend

        mode http
        bind :::443 v4v6 tfo ssl crt /etc/haproxy/example.com.pem ecdhe secp384r1
        bind :::80 v4v6 tfo
        option http-use-htx
        option forwardfor header X-Forwarded-For

        http-response set-header Strict-Transport-Security max-age=31536000;\ includeSubdomains;\ preload
        http-response set-header X-Frame-Options sameorigin
        http-response set-header X-Content-Type-Options nosniff
        http-response set-header X-XSS-Protection: 1;mode=block
        http-response set-header Referrer-Policy no-referrer-when-downgrade
        http-response set-header Feature-Policy "geolocation 'none'; midi 'none'; camera 'none'; usb 'none'; magnetometer 'none';$
        http-request set-header X-Forwarded-Proto https

        redirect scheme https code 301 unless { ssl_fc }

        acl example1_host hdr(host) -i example1.example.com
        acl example2_host hdr(host) -i example2.example.com

        use_backend example1 if example1_host
        use_backend example2 if example2_host

backend example1

        mode http
        balance leastconn
        http-request set-header X-Client-IP %[src]
        http-request add-header X-Forwarded-Proto https
        option forwardfor
        option http-use-htx
        server example1server example1.lxd:1234

backend example2

        mode http
        balance leastconn
        http-request set-header X-Client-IP %[src]
        http-request add-header X-Forwarded-Proto https
        option forwardfor
        option http-use-htx
        server example2server example2.lxd:1234

Thanks in Adance
mthax

Please ignore those. Those are tutorials get some kind of H2 functionality before H2 was supported in haproxy.

Your backend server returns 406 Not Acceptable, that’s probably not normal. Find out why, perhaps it doesn’t like how the request looks like when H2 is in use (although there should not be a difference in theory, as haproxy should convert it all).

Does it work in Firefox?

Please provide the output of haproxy -vv and make sure you are using a recent stable release (currently 1.9.9 or 2.0.4).

The configuration seems fine, and haproxy should handle this all. You probably hit some bug, which is why I suggest you check if you are running latest stable releases.

Output of haproxy -vv

HA-Proxy version 2.0.3-1ppa1~bionic 2019/07/23 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-PJTjCX/haproxy-2.0.3=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.31 2018-02-12
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
        prometheus-exporter

Available filters :
        [SPOE] spoe
        [COMP] compression
        [CACHE] cache
        [TRACE] trace

One thing i noticed is, if i add a send-proxy to the server line of the backend, the Message Chrome shows changes to ERR_SPDY_SERVER_REFUSED_STREAM, but i dont know if this noteworthy at all since i just used it to test

[07/Aug/2019:17:28:18.921] web_frontend~ example1/example1 0/0/1/0/1 400 180 - - ---- 1/1/0/0/0 0/0 "GET / HTTP/2.0"

I installed Firefox and in Firefox it works.

[07/Aug/2019:17:39:41.344] web_frontend~ example1/example1 0/0/1/0/1 200 1409 - - ---- 1/1/0/0/0 0/0 "GET /style.css HTTP/2.0"

With Chrome it wont (tried the Incognito Mode too), clearing the Chrome Cache and restarting the container didnt help either…

No, you cannot just configure send-proxy to your backend, if it doesn’t support the proxy protocol. This will just make the request fail earlier, changing the error message, not solving anything at all. It would also brake the HTTP/1.1, as it is just plain wrong.

Like I said:

  • upgrade to haproxy 2.0.4 (I assume you installed this from Vincent Bernard’s PPA’s, 2.0.4 will probably hit that shortly)
  • check how the request between haproxy and your backend locks like (capture the traffic where the backend responds with a 2.0.4 header).
  • remove option http-use-htx everywhere (it’s already default anyway) and actually disable it, by putting into the default section no option http-use-htx - this will disable HTX and is worth a try (especially if your backend server does not support the lower-case headers)
  • confirm that different backend servers are affected (running different software stacks)

Yes, i installed it from his PPA, 2.0.4 isnt avaiable there atm.

Thanks for the information about http-use-htx, removed that.

I managed to solve it myself.

I changed

http-response set-header X-XSS-Protection: 1;mode=block

to

http-response set-header X-Xss-Protection "1; mode=block"

and suddenly Chrome loads.Its a bit strange that Chrome didnt load anything and that Firefox did load it normal.

I am sorry that i stole your time with such silly mistake.

Lg mthax

No problems, thanks for reporting back.