HTTP/2: huge transfer-speed degradation

happy · January 26, 2018, 1:56am

Most of HTTPS requests with HTTP/2 are slower than normal.
And there are some requests that are slower in 3-10 times. An extra time appears on “read request” stage.

Example:

HTTP/1.1 https://prnt.sc/i5p0hs
HTTP/2 https://prnt.sc/i5p0vr

I’ve tested with GTMetrix (some pages have 3-5 longer average full-load time), DareBoost and local test with Hey

For local test I’ve used single 14MB file and 10 threads.

Average request-time degradation 20-30%
Slowest requests are 1.5-3 times worse

It is with default tune-h2 settings.

Changing to the maximum header-table-size and 512MB initial-window-size makes it a bit better, but it is still much worse than HTTP/1.1.

HA-Proxy version 1.8.3-a91f55-27 (USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1 USE_NS=1)

happy · January 29, 2018, 1:47pm

Does anyone see normal speed for HTTP/2?

eva2000 · February 8, 2018, 6:58am

tried testing with webpagetest.org in various web browsers ?

happy · February 9, 2018, 4:11pm

Dareboost and GTMetrix uses different browsers and Hey-script is one more way. So, it is 100% problem not from client.

Do you have any own HTTP2-tests? I want to know at least - is it Haproxy native problem, or something related to my settings or OS.

happy · February 9, 2018, 4:33pm

First versions of HTTP/2 support in some popular web-servers also were slower than normal, so it might be similar case for Haproxy
However, it is really huge difference

eva2000 · February 9, 2018, 7:04pm

Okay did some Haproxy 1.8.3 tests and yes the HTTP/2 slowness compared to HTTP/1.1 might be related to images specifically. If you test non-image requests, HTTP/2 is faster while image requests HTTP/2 are slower.

haproxy -vv
HA-Proxy version 1.8.3-205f675 2017/12/30
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = native
  CC      = gcc
  CFLAGS  = -march=native -m64 -march=x86-64 -O2 -g
  OPTIONS = USE_LINUX_SPLICE=1 USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_THREAD=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0g  2 Nov 2017
Running on OpenSSL version : OpenSSL 1.1.0g  2 Nov 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.41 2017-07-05
Running on PCRE version : 8.41 2017-07-05
PCRE library supports JIT : yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
        [SPOE] spoe
        [COMP] compression
        [TRACE] trace

webpagetest filmstrips comparing

Centmin Mod Nginx 1.13.8 with proxy_cache and without proxy_cache
Haproxy 1.8.3 HTTP/2 HTTPS
Haproxy 1.8.3 HTTP/1.1 HTTPS

Visually Complete

Document Complete

Fully Loaded

nghttp2 h2load HTTP/1.1 and HTTP/2 HTTPS tests

port 447 = haproxy HTTP/1.1 HTTPS
port 444 = haproxy HTTP/2 HTTPS

for static index.html pages

port 447 = haproxy HTTP/1.1 HTTPS

/usr/local/bin/h2load -p http/1.1 -t2 -c50 -m100 -n10000 -H 'Accept-Encoding: gzip' https://baremetal.domain.com:447/
starting benchmark...
spawning thread #0: 25 total client(s). 5000 total requests
spawning thread #1: 25 total client(s). 5000 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: http/1.1
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 357.96ms, 27935.93 req/s, 102.77MB/s
requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 36.79MB (38573750) total, 4.63MB (4853750) headers (space savings 0.00%), 31.33MB (32850000) data
                     min         max         mean         sd        +/- sd
time for request:      518us    214.16ms    122.48ms     54.81ms    68.33%
time for connect:    11.35ms     21.57ms     16.64ms      2.87ms    58.00%
time to 1st byte:    14.56ms     29.50ms     18.67ms      3.24ms    72.00%
req/s           :     559.41      655.22      592.42       24.24    76.00%

port 444 = haproxy HTTP/2 HTTPS

/usr/local/bin/h2load -t2 -c50 -m100 -n10000 -H 'Accept-Encoding: gzip' https://baremetal.domain.com:444/            
starting benchmark...
spawning thread #0: 25 total client(s). 5000 total requests
spawning thread #1: 25 total client(s). 5000 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 220.87ms, 45276.53 req/s, 92.80MB/s
requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 20.50MB (21490962) total, 4.09MB (4283750) headers (space savings 8.93%), 16.15MB (16930000) data
                     min         max         mean         sd        +/- sd
time for request:    11.02ms    165.97ms     68.03ms     25.02ms    71.41%
time for connect:    10.10ms     28.82ms     20.56ms      4.77ms    68.00%
time to 1st byte:    23.29ms    111.87ms     67.16ms     24.58ms    62.00%
req/s           :     909.67     1653.41     1122.10      181.65    74.00%

for png image

port 447 = haproxy HTTP/1.1 HTTPS

/usr/local/bin/h2load -p http/1.1 -t2 -c50 -m100 -n10000 -H 'Accept-Encoding: gzip' https://baremetal.domain.com:447/images/slide2.png
starting benchmark...
spawning thread #0: 25 total client(s). 5000 total requests
spawning thread #1: 25 total client(s). 5000 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: http/1.1
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 2.27s, 4397.64 req/s, 1.04GB/s
requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 2.36GB (2534863750) total, 4.24MB (4443750) headers (space savings 0.00%), 2.36GB (2529710000) data
                     min         max         mean         sd        +/- sd
time for request:     2.18ms       2.10s    804.63ms    380.48ms    75.73%
time for connect:     5.58ms     22.30ms     15.11ms      3.93ms    70.00%
time to 1st byte:    14.57ms     28.97ms     20.40ms      4.10ms    50.00%
req/s           :      87.98      104.10       96.25        3.88    76.00%

port 444 = haproxy HTTP/2 HTTPS

/usr/local/bin/h2load -t2 -c50 -m100 -n10000 -H 'Accept-Encoding: gzip' https://baremetal.domain.com:444/images/slide2.png
starting benchmark...
spawning thread #0: 25 total client(s). 5000 total requests
spawning thread #1: 25 total client(s). 5000 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-RSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 3.08s, 3244.12 req/s, 784.88MB/s
requests: 10000 total, 10000 started, 10000 done, 10000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 10000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 2.36GB (2536928095) total, 3.66MB (3833750) headers (space savings 16.00%), 2.36GB (2529710000) data
                     min         max         mean         sd        +/- sd
time for request:    35.43ms       2.47s       1.11s    349.90ms    70.71%
time for connect:     5.42ms     29.75ms     16.00ms      6.16ms    72.00%
time to 1st byte:    23.08ms    199.11ms     89.85ms     34.39ms    68.00%
req/s           :      64.90      104.36       81.36        8.74    74.00%

For tests used my w3layout flatmate theme test site which you can find at GitHub - centminmod/testpages

Just noticed for Haproxy HTTP/2 h2load tests comparing png image vs static index.html, that header space savings for HPACK compression has a higher compression ratio for png image than static index.html at 16% saved vs 8.93% saved ? Why would png images compress more ?

Haproxy HTTP/1.1. HTTPS index.html

curl -I https://baremetal.domain.com:447/
HTTP/1.1 200 OK
Date: Fri, 09 Feb 2018 18:09:17 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 4074
Last-Modified: Sun, 04 Feb 2018 12:30:13 GMT
Vary: Accept-Encoding
ETag: "5a76fcd5-fea"
Server: nginx centminmod
X-Powered-By: centminmod
Expires: Sun, 11 Mar 2018 18:09:17 GMT
Cache-Control: max-age=2592000
Access-Control-Allow-Origin: *
Cache-Control: public, must-revalidate, proxy-revalidate, immutable, stale-while-revalidate=86400, stale-if-error=604800
Accept-Ranges: bytes
Set-Cookie: SERVERID=1_server5; path=/

Haproxy HTTP/2. HTTPS index.html

curl -I https://baremetal.domain.com:444/
HTTP/2 200 
date: Fri, 09 Feb 2018 18:09:21 GMT
content-type: text/html; charset=utf-8
content-length: 4074
last-modified: Sun, 04 Feb 2018 12:30:13 GMT
vary: Accept-Encoding
etag: "5a76fcd5-fea"
server: nginx centminmod
x-powered-by: centminmod
expires: Sun, 11 Mar 2018 18:09:21 GMT
cache-control: max-age=2592000
access-control-allow-origin: *
cache-control: public, must-revalidate, proxy-revalidate, immutable, stale-while-revalidate=86400, stale-if-error=604800
accept-language: bytes
set-cookie: SERVERID=1_server6; path=/

Haproxy HTTP/1.1. HTTPS png image

curl -I https://baremetal.domain.com:447/images/slide2.png
HTTP/1.1 200 OK
Date: Fri, 09 Feb 2018 19:10:42 GMT
Content-Type: image/png
Content-Length: 252971
Last-Modified: Fri, 09 Feb 2018 18:24:25 GMT
ETag: "5a7de759-3dc2b"
Server: nginx centminmod
X-Powered-By: centminmod
Expires: Sun, 11 Mar 2018 19:10:42 GMT
Cache-Control: max-age=2592000
Access-Control-Allow-Origin: *
Cache-Control: public, must-revalidate, proxy-revalidate, immutable, stale-while-revalidate=86400, stale-if-error=604800
Accept-Ranges: bytes
Set-Cookie: SERVERID=1_server10; path=/

Haproxy HTTP/2. HTTPS png image

curl -I https://baremetal.domain.com:444/images/slide2.png
HTTP/2 200 
date: Fri, 09 Feb 2018 19:10:58 GMT
content-type: image/png
content-length: 252971
last-modified: Fri, 09 Feb 2018 18:24:25 GMT
etag: "5a7de759-3dc2b"
server: nginx centminmod
x-powered-by: centminmod
expires: Sun, 11 Mar 2018 19:10:58 GMT
cache-control: max-age=2592000
access-control-allow-origin: *
cache-control: public, must-revalidate, proxy-revalidate, immutable, stale-while-revalidate=86400, stale-if-error=604800
accept-language: bytes
set-cookie: SERVERID=1_server11; path=/

happy · February 10, 2018, 10:55am

Thank you for such detailed tests

Only one thing to add. It is not about image and non-image. In your case for “index.html” - there is about 2 times less overall traffic (even for data-part). It should be the reason why H2 was so much faster.

I’ve seen degradation for html, js, css (gzipped and not), images and video.
Bigger file-size -> degradation is more significant and more often (many requests for small files are almost normal).

Now, I’ve retried tests with 1.8.4 and all the same.

lukastribus · February 10, 2018, 12:08pm

This is may be unrelated, but compared to HTTP/1.1 H2 cannot reuse backend connections.

So to exclude this limitation as a possible factor in different performance figures, use “option http-server-close” instead of the default keep-alive mode, so H1 and H2 are comparable:
https://www.mail-archive.com/haproxy@formilux.org/msg28861.html

happy · February 10, 2018, 12:51pm

Unfortunately, that is really unrelated. It can make difference with a huge amount of requests and\or remote backends

This is 14mb video file from local backend:

HTTP/2

Total: 65.5290 secs
Slowest: 2.5039 secs
Fastest: 0.6438 secs
Average: 2.1648 secs
Requests/sec: 4.5781
Total data: 4333206300 bytes
Size/request: 14444021 bytes

HTTP/1.1

Total: 40.7544 secs
Slowest: 1.6909 secs
Fastest: 1.1414 secs
Average: 1.3568 secs
Requests/sec: 7.3612
Total data: 4333206300 bytes
Size/request: 14444021 bytes

eva2000 · February 10, 2018, 4:33pm

ah yes that maybe the case

didn’t know that… unfortuately i have to spin down my server as vultr promo credits run out so will have to retest later on

happy · April 21, 2018, 5:19pm

Had hopes on last commit, but the problem still exists in 1.8.8

Topic		Replies	Views
HTTP2 not speed up Help!	0	447	May 13, 2020
More oddities, looking for guidance Help!	0	634	June 7, 2021
Haproxy slow upload speed Help!	11	1680	February 4, 2022
Overall slow file transfers Help!	2	129	February 19, 2024
Very Slow Performance Help!	3	5928	May 10, 2017

HTTP/2: huge transfer-speed degradation

Related Topics