HTTP/2, Chrome 49 on Windows XP problems

[michal.paal]

Hi, we are currently migrating our servers behind new haproxy 1.8.12 from our older haproxy cluster with 1.5.4.
In this process, we have also enabled HTTP/2. We found out that our users that have exactly Chrome 49 on Windows XP SP3 cannot access our sites and they are getting ERR_CONNECTION_CLOSED.

After our investigation, we can replicate this problem, and it is occuring only in this specific version of Browser/OS, only when we enable HTTP/2.

Behaviour of this problem occurs only after some time, it is not present in every request.
Steps to reproduce are - login on our site, browse it for a bit, and suddenly whole site will became inaccessable.

We did debug it with Wireshark, and have decrypted traffic in dump, we can see that haproxy is sending GOAWAY with internal error flag.

config:

bind IP:443 ssl crt /etc/pki/tls/haproxy alpn h2,http/1.1
bind IP:443 ssl crt /etc/pki/tls/haproxy alpn h2,http/1.1

[lukastribus]

I assume you don’t want to hear that Chrome 49 is completely outdated.

This specific issue is caused by a bug in the blink engine of Chrome 49, which is causing H2 to fail when using long URLs, you can read more about it here:

HTTP/2 not compatible with filrefox on haproxy 1.8.3?

I’ve found the root of the rest problem: The max length of url has very strict limitation under the h2 mode on some old versions of the chrome blink engine. Even if the same url just working fine under the http/1.1 mode with the exactly same blink engine. For example: We have an url with 2713 bytes of length, and it is not working under h2 mode for the chrome 49. The solution is very simple: Just shrink the url length. For example, we have a function deal with the js file batch loading, here i…

[michal.paal]

Yes I know it, but we (newspaper) have many paying customers running this outdated browser, and sadly, in many times thay cannot (or dont know, or dont want) to update…

Anyway thank you for the link, it will be helpful

[michal.paal]

By the way I’ve checked and our URL’s are not very long, but I’ve found some correlation with cookies, when we delete few cookies it starts to work again. It seem that it has something to do with length of cookie headers also. Any suggestions about this?

[lukastribus]

There is no magic solution for this I’m afraid. I assume it has to do with very-long header lines. The total number of cookies should not have any impact though. edit: wrong: total cookies size also matters.

Try to limit the cookie length, I don’t have any other suggestions I’m afraid. Unsupported OS and unsupported browsers are not supportable.

If you can’t reduce cookie length in your website, and you have to enable H2, then there is nothing you can do, other than have your users disable H2 (I’m sure there is a command-line argument or in browser configuration flag for this).

[lukastribus]

I’m analyzing the issue more in-depth, maybe there is something we can do at haproxy level (however it is going to require some time).

edit: This is happening because haproxy currently lacks support for the H2 CONTINUATION frame. I reported this to Willy/the mailing list:

https://www.mail-archive.com/haproxy@formilux.org/msg31134.html

[michal.paal]

Thank you for investigation, I’ve also read your conversation on mailing list with Willy, at least all of us understand now why it is happening.

[lukastribus]

I have something else in mind as well. With the upcoming OpenSSL 1.1.1 (or boringssl), we may be able to configure ALPN based on the certificate type. So if we use both RSA and ECC certificates, we may enable h2 only for ECC certificates.

This would disable H2 for Chrome on XP and everybody else (in the ECC world) would be able to negotiate H2. The only case not covered by this would be Chrome on Windows Vista (which supports ECC certificates), but I don’t think there is a lot of people out there still using Vista.

I need to test it though, and OpenSSL 1.1.1 is still in beta.

edit: I’ve confirmed that this theory works: instead of declaring the certificate directly in the haproxy configuration, we refer to a crt-list, where we don’t enable h2 for the RSA certificate:

frontend httpsin
 bind :443 ssl crt-list /etc/haproxy/crtlist

/etc/haproxy/crtlist:

/etc/private/ssl/sitecert-rsa.pem [alpn http/1.1]
/etc/private/ssl/sitecert-ecc.pem [alpn h2,http/1.1]

However, this requires either Haproxy 1.8+ with OpenSSL 1.1.1 (currently in beta), or Haproxy 1.9dev (currently in development) with current boringssl.

I think the best thing for production would be to wait to enable H2 until OpenSSL 1.1.1 is stable, and then use the approach above to avoid breaking Windows XP.

Remember that Chrome 49 on Vista still fails. For both XP and Vista users, Firefox 52.9.0 ESR would be a huge upgrade, fixing all those problems, even though it is also EOL since last month (but that’s way better then Chrome 49 which was EOLed a few years ago).

[lukastribus]

CONTINUATION is supported since v1.9.2 and it fixes this issue.