Massive surge in PR-- terminations with status code -1 after enabling frontend H2

Help!
1

Hi there!

We use Haproxy v2.2 (upgraded from 1.8) to load balance traffic to our web server stack of a few debian based vps servers, hosting a php website with apache2 (mpm_event) and php-fpm.

Ever since activating h2 on the haproxy config for the frontend part

frontend ws-443-in
        bind 172.16.xx.xxx:443 ssl  crt certfile-path alpn h2,http/1.1
        http-request add-header X-Forwarded-Proto https
        option http-server-close
        default_backend ws-443


backend ws-443
        balance roundrobin
        option httpchk
        option forwardfor
        timeout check 1000
        server ws1 ws1.domain.net:443 weight 2 check ssl verify none
        server ws2 ws2.domain.net:443 weight 2 check ssl verify none
        server ws3 ws3.domain.net:443 weight 2 check ssl verify none
        server ws4 ws4.domain.net:443 weight 2 check ssl verify none
        server ws5 ws5.domain.net:443 weight 2 check ssl verify none
        server ws6 ws6.domain.net:443 weight 2 check ssl verify none

the haproxy log gets spammed with bad requests like the following:

Jun 22 12:19:09 haproxy3 haproxy[20432]: 37.72.xx.xxx:53049 [22/Jun/2021:12:19:09.875] ws-443-in~ ws-443-in/<NOSRV> -1/-1/-1/-1/0 0 0 - - PR-- 365/297/0/0/0 0/0 "<BADREQ>"
Jun 22 12:19:13 haproxy3 haproxy[20432]: 92.137.xx.xxx:56220 [22/Jun/2021:12:19:13.661] ws-443-in~ ws-443-in/<NOSRV> -1/-1/-1/-1/0 0 0 - - PR-- 371/302/0/0/0 0/0 "<BADREQ>"
Jun 22 12:19:29 haproxy3 haproxy[20432]: 84.190.xx.xxx:56796 [22/Jun/2021:12:19:29.723] ws-443-in~ ws-443-in/<NOSRV> -1/-1/-1/-1/0 0 0 - - PR-- 381/306/0/0/0 0/0 "<BADREQ>"
Jun 22 12:19:35 haproxy3 haproxy[20432]: 178.51.xx.xxx:24415 [22/Jun/2021:12:19:35.957] ws-443-in~ ws-443-in/<NOSRV> -1/-1/-1/-1/0 0 0 - - PR-- 378/302/0/0/0 0/0 "<BADREQ>"
Jun 22 12:19:39 haproxy3 haproxy[20432]: 95.91.xx.xxx:61266 [22/Jun/2021:12:19:39.545] ws-443-in~ ws-443-in/<NOSRV> -1/-1/-1/-1/0 0 0 - - PR-- 371/292/0/0/0 0/0 "<BADREQ>"
Jun 22 12:19:41 haproxy3 haproxy[20432]: 217.136.xx.xxx:37964 [22/Jun/2021:12:19:41.533] ws-443-in~ ws-443-in/<NOSRV> -1/-1/-1/-1/0 0 0 - - PR-- 368/290/0/0/0 0/0 "<BADREQ>"

Meanwhile, using a browser, all requests seem to work normally.

After reading up on the web, we tried multiple options to understand these requests, but haven’t found a way to understand what these are or how they look.

The config has

option dontlognull

enabled, allowing invalid-requests and upping the http-request timeout to 30s had no effect either.

Connecting to the socket, show errors does not show any faulty requests, which seems in line with the fact that the connection is apparently never established looking at the timers in the log above.
We initially expected these to be browser preconnects, but option http-ignore-probes had no effect either.

As soon as h2 is deactivated on the haproxy frontend, the BADREQs are gone for good.

Any ideas what could cause this? We imagine this is just noise in the logs without actual impact, but cannot say for sure.

Full config:


global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/haproxy.sock mode 660 level admin
        stats socket ipv4@172.16.xx.xxx:7167 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        nbproc          1
        cpu-map         1 3
        maxconn         1000000

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
        ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
        ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
        ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
        tune.ssl.default-dh-param 2048

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  301s
        timeout server  301s
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http
        maxconn 1000000
	    option log-health-checks

listen stats
        bind 172.16.xx.xxx:9000
        mode http
        stats enable
        stats realm Haproxy\ Statistics
        stats uri /stats



frontend ws-80-in
	bind 172.16.xx.xxx:80
	http-request add-header X-Forwarded-Proto http
	default_backend ws-80

backend ws-80
        balance roundrobin
        option httpchk
        option forwardfor
        timeout check 1000
        server ws1 ws1.domain.net:80 weight 2 check 
        server ws2 ws2.domain.net:80 weight 2 check 
        server ws3 ws3.domain.net:80 weight 2 check 
        server ws4 ws4.domain.net:80 weight 2 check 
        server ws5 ws5.domain.net:80 weight 2 check 
        server ws6 ws6.domain.net:80 weight 2 check 

frontend ws-443-in
	bind 172.16.xx.xxx:443 ssl  crt /etc/ssl/private/wildcard.domain.com_bundle.pem alpn h2,http/1.1
	http-request add-header X-Forwarded-Proto https
	option http-server-close
	default_backend ws-443


backend ws-443
        balance roundrobin
        option httpchk
        option forwardfor
        timeout check 1000
        server ws1 ws1.domain.net:443 weight 2 check ssl verify none
        server ws2 ws2.domain.net:443 weight 2 check ssl verify none
        server ws3 ws3.domain.net:443 weight 2 check ssl verify none
        server ws4 ws4.domain.net:443 weight 2 check ssl verify none
        server ws5 ws5.domain.net:443 weight 2 check ssl verify none
        server ws6 ws6.domain.net:443 weight 2 check ssl verify none

backend no-match
	errorfile 503 /etc/haproxy/errors/fake_404.http

Thank you all in advance!
Best

2

Could there be http cleartext traffic on port 443?

Not sure if adding the following to your frontend ws-443-in would help to see more:

	http-request capture req.hdr(Host) len 30
	# Custom log format (extended by TLS and Cipher)
    log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r {%sslv:%sslc}"

Hope this helps?
Toni

3

Thank you for this, it definitely finally gave us some info that we were desperately looking for.
It seems like most of the requests are TLSv1.3:TLS_AES_256_GCM_SHA384 -which wasn’t part of the ssl-default-bind-options list. I added it, but the PR-- terminations sadly still come in.

Our Haproxy was build with OpenSSL 1.1.1d, which should support this.

4

Want to bump this, as the problem still exists - sadly, we have not been able to get any further information.