Haproxy 1.8.x url_param issue in http2

Help!
1

Hi, When I chose url_param balance method I got an “Error in the HTTP2 framing layer”, but in http1.1 it works. Whereas when I change the balancing mode, e.g. roundrobin instead url_param then it’s okay so probably it’s an issue of usage url_param in http2. I checked several versions of 1.8.3 -> 1.8.8 and it is the same.

# haproxy -vv

HA-Proxy version 1.8.8-1.el7 2018/04/19
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-unused-label
  OPTIONS = USE_ZLIB=1 USE_DL=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace

Sample config:

# /etc/haproxy/haproxy.cfg 
global
  chroot  /var/lib/haproxy
  daemon  
  group  haproxy
  log  127.0.0.1 local2 debug
  maxconn  210000
  nbproc  2
  pidfile  /var/run/haproxy.pid
  spread-checks  2
  ssl-default-bind-ciphers  ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!DHE:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
  ssl-default-bind-options  ssl-min-ver TLSv1.0 no-tls-tickets
  stats  socket /var/lib/haproxy/stats uid 0 gid 0 mode 0440 process 1
  stats  bind-process all
  tune.bufsize  16384
  tune.h2.max-concurrent-streams  100
  tune.maxrewrite  1024
  tune.ssl.cachesize  100000
  tune.ssl.default-dh-param  2048
  tune.ssl.lifetime  600
  ulimit-n  500000
  user  haproxy

defaults
  log  global
  maxconn  120000
  mode  http
  option  redispatch
  option  http-server-close
  option  dontlognull
  retries  3
  stats  enable
  timeout  http-request 302s
  timeout  queue 60s
  timeout  connect 5s
  timeout  client 302s
  timeout  server 302s
  timeout  check 1s

frontend https
  bind 0.0.0.0:443 ssl crt /etc/ssl/server.pem alpn h2,http/1.1
  bind 0.0.0.0:80 
  mode http
  bind-process 1
  default_backend http2_backend
  maxconn 120000
  option httplog
  option http-server-close
  option forwardfor except 127.0.0.0/8
  timeout client 30s
  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }
  reqadd X-Forwarded-Proto:\ https
  reqadd Https:\ on

backend http2_backend
  mode http
  balance url_param bh
  bind-process 1
  hash-type consistent
  option redispatch
  option http-server-close
  server http1.1 127.0.0.1:81 check

listen stats
  bind 10.254.0.10:888 process 1
  mode http
  stats uri /
  stats enable
  stats show-node
  stats refresh 20s
  stats show-legends

Requests in url_param:

# curl -v -I -k --http2 -XGET https://10.254.0.10/index.html
*   Trying 10.254.0.10...
* TCP_NODELAY set
* Connected to 10.254.0.10 (10.254.0.10) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  start date: May  9 21:03:10 2018 GMT
*  expire date: May  9 21:03:10 2019 GMT
*  issuer: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7ffe1e000000)
> GET /index.html HTTP/2
> Host: 10.254.0.10
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (16) Error in the HTTP2 framing layer

# curl -v -I -k --http1.1 -XGET https://10.254.0.10/index.html
*   Trying 10.254.0.10...
* TCP_NODELAY set
* Connected to 10.254.0.10 (10.254.0.10) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  start date: May  9 21:03:10 2018 GMT
*  expire date: May  9 21:03:10 2019 GMT
*  issuer: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /index.html HTTP/1.1
> Host: 10.254.0.10
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 09 May 2018 22:28:55 GMT
Date: Wed, 09 May 2018 22:28:55 GMT
< Server: Apache/2.4.6 (CentOS)
Server: Apache/2.4.6 (CentOS)
< Last-Modified: Wed, 09 May 2018 21:57:36 GMT
Last-Modified: Wed, 09 May 2018 21:57:36 GMT
< ETag: "1ca-56bccfda5fc1c"
ETag: "1ca-56bccfda5fc1c"
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Length: 458
Content-Length: 458
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8

< 
* Excess found in a non pipelined read: excess = 458 url = /index.html (zero-length body)
* Connection #0 to host 10.254.0.10 left intact

Dump from h2c:

-> SETTINGS(0)
    - ACK
    SETTINGS_INITIAL_WINDOW_SIZE: 1073741824
    SETTINGS_ENABLE_PUSH: 0
    SETTINGS_MAX_CONCURRENT_STREAMS: 100

<- SETTINGS(0)
    - ACK
    SETTINGS_MAX_CONCURRENT_STREAMS: 100

-> WINDOW_UPDATE(0)
    Window size increment: 1073676289

<- SETTINGS(0)
    + ACK
    {empty}

-> HEADERS(1)
    + END_STREAM
    + END_HEADERS
    :method: GET
    :path: /index.html?bh=1233
    :scheme: https
    :authority: 10.254.0.10:443
    user-agent: curl/7.54.0
    accept: */*

-> SETTINGS(0)
    + ACK
    {empty}

<- RST_STREAM(1)
    Error code: NO_ERROR

<- GOAWAY(0)
    Last stream id: 1
    Error code: NO_ERROR

Error while reading next frame: EOF
Closing connection.

After changing to roundrobin:

# curl -v -I -k --http2 -XGET https://10.254.0.10/index.html
*   Trying 10.254.0.10...
* TCP_NODELAY set
* Connected to 10.254.0.10 (10.254.0.10) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  start date: May  9 21:03:10 2018 GMT
*  expire date: May  9 21:03:10 2019 GMT
*  issuer: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f845e80d800)
> GET /index.html HTTP/2
> Host: 10.254.0.10
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
HTTP/2 200 
< date: Wed, 09 May 2018 22:49:11 GMT
date: Wed, 09 May 2018 22:49:11 GMT
< server: Apache/2.4.6 (CentOS)
server: Apache/2.4.6 (CentOS)
< last-modified: Wed, 09 May 2018 21:57:36 GMT
last-modified: Wed, 09 May 2018 21:57:36 GMT
< etag: "1ca-56bccfda5fc1c"
etag: "1ca-56bccfda5fc1c"
< accept-language: bytes
accept-language: bytes
< content-length: 458
content-length: 458
< content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8

< 
* Excess found in a non pipelined read: excess = 458 url = /index.html (zero-length body)
* Connection #0 to host 10.254.0.10 left intact

# curl -v -I -k --http1.1 -XGET https://10.254.0.10/index.html
*   Trying 10.254.0.10...
* TCP_NODELAY set
* Connected to 10.254.0.10 (10.254.0.10) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  start date: May  9 21:03:10 2018 GMT
*  expire date: May  9 21:03:10 2019 GMT
*  issuer: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> GET /index.html HTTP/1.1
> Host: 10.254.0.10
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 09 May 2018 22:49:08 GMT
Date: Wed, 09 May 2018 22:49:08 GMT
< Server: Apache/2.4.6 (CentOS)
Server: Apache/2.4.6 (CentOS)
< Last-Modified: Wed, 09 May 2018 21:57:36 GMT
Last-Modified: Wed, 09 May 2018 21:57:36 GMT
< ETag: "1ca-56bccfda5fc1c"
ETag: "1ca-56bccfda5fc1c"
< Accept-Ranges: bytes
Accept-Ranges: bytes
< Content-Length: 458
Content-Length: 458
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8

< 
* Excess found in a non pipelined read: excess = 458 url = /index.html (zero-length body)
* Connection #0 to host 10.254.0.10 left intact

Dump from h2c:

-> SETTINGS(0)
    - ACK
    SETTINGS_MAX_CONCURRENT_STREAMS: 100
    SETTINGS_INITIAL_WINDOW_SIZE: 1073741824
    SETTINGS_ENABLE_PUSH: 0

-> WINDOW_UPDATE(0)
    Window size increment: 1073676289

-> HEADERS(1)
    + END_STREAM
    + END_HEADERS
    :method: GET
    :path: /index.html?bh=1233
    :scheme: https
    :authority: 10.254.0.10:443
    user-agent: curl/7.54.0
    accept: */*

<- SETTINGS(0)
    - ACK
    SETTINGS_MAX_CONCURRENT_STREAMS: 100

-> SETTINGS(0)
    + ACK
    {empty}

<- SETTINGS(0)
    + ACK
    {empty}

<- HEADERS(1)
    - END_STREAM
    + END_HEADERS
    :status: 200
    date: Thu, 10 May 2018 15:18:03 GMT
    server: Apache/2.4.6 (CentOS)
    last-modified: Wed, 09 May 2018 21:57:36 GMT
    etag: "1ca-56bccfda5fc1c"
    accept-language: bytes
    content-length: 458
    content-type: text/html; charset=UTF-8

<- DATA(1)
    + END_STREAM
    {458 bytes}

Error while reading next frame: EOF
Closing connection.

And some logs in http1.1/http2:

http2_backend/http1.1 0/0/0/1/1 200 720 - - ---- 1/1/0/0/0 0/0 "GET /index.html?bh=1 HTTP/1.1"
May 10 01:49:57 localhost haproxy[24227]: 10.254.0.1:64065 [10/May/2018:01:49:57.403] https~ http2_backend/<NOSRV> 0/-1/-1/-1/0 -1 0 - - CH-- 1/1/0/0/3 0/0 "GET /index.html?bh=1 HTTP/1.1"
May 10 01:49:57 localhost haproxy[24227]: Connect from 10.254.0.1:64066 to 10.254.0.10:888 (stats/HTTP)
May 10 01:50:03 localhost haproxy[24227]: 10.254.0.1:64087 [10/May/2018:01:50:03.132] https~ http2_backend/<NOSRV> 0/-1/-1/-1/0 -1 0 - - CH-- 2/1/0/0/3 0/0 "GET /index.html?bh=1 HTTP/1.1"
May 10 01:50:25 localhost haproxy[24227]: 10.254.0.1:64190 [10/May/2018:01:50:25.344] https~ http2_backend/<NOSRV> 0/-1/-1/-1/0 -1 0 - - CH-- 2/1/0/0/3 0/0 "GET /index.html?bh=12 HTTP/1.1"

Has anyone ever used url_param balance method in http2 traffic?

Thanks
Pawel

2

This is gonna require some help from @willy

3

Thanks for the ping. It’s very strange. At the moment I have absolutely zero idea what could cause this (unless url_param is broken and affects all HTTP versions of course), but I’ll have a look.

4

@willy could this have a similar use-case than the problems we saw with h2 in httpclose mode?

edit: I mean root cause, not use-case :slight_smile:

1 Like
5

As usual my e-mail based response was dropped, I’ll have to check my settings.

I was saying that it could indeed be related, given that the first one doesn’t make real sense in theory and that I failed to reproduce it. Probably the two share a same root cause. I’ll try to reproduce.

6

And sadly, here I cannot reproduce the issue at all, it works as expected :frowning: It starts to smell a lot like this other httpclose issue which I couldn’t reproduce at all either.

@pkaroluk, could you please try to progressively remove a lot of statements from your config to find the smallest one that exhibits the error ? It will help eliminate some possible candidates.

7

For the record, that report would be here:
https://discourse.haproxy.org/t/h2-option-httpclose-error-in-the-http2-framing-layer/

We also have ML threads:
https://www.mail-archive.com/haproxy@formilux.org/msg28400.html
https://www.mail-archive.com/haproxy@formilux.org/msg29072.html

8

Thanks Lukas for the URLs. But as you can see I never could reproduce any of them. That’s really embarrassing. The limited number of reports tends to indicate it affects users only under a specific condition which has not yet been identified. I already tried sizes, delays etc without success. We could have some hopes with this one if Pavel manages to strip down the config to the smallest reproducer
and it sparks a light in my head :-/

9

Yes, I know, I just thought it would be a good idea to list them all.

My gut feeling is that this is somehow related to how the backend server behaves. Like how it closes the connection in certain situations, or other connection related things.

But that just a shot in the dark.

10

It’s a good idea.url_param requires to rewind the stream, maybe it doesn’t survive a connection retry only in the case where the request is made over H2, and we’ll figure that the httpclose fails only on retries as well for example. I’ll study that idea just in case.

11

Hi, I tied httpclose and http-server-close in few configurations but the result was the same.

@willy as you request, sample config:

#/etc/haproxy/haproxy.cfg 
global
  chroot  /var/lib/haproxy
  daemon  
  group  haproxy
  log  127.0.0.1:512 local2 debug
  nbproc  1
  pidfile  /var/run/haproxy.pid
  spread-checks  1
  stats  socket /var/lib/haproxy/stats uid 0 gid 0 mode 0440 process 1
  stats  bind-process all
  tune.ssl.default-dh-param  2048
  ulimit-n  500000
  user  haproxy

defaults
  hash-type  consistent
  log  global
  mode  http
  #option  http-server-close
  #option  httpclose
  retries  3
  stats  enable
  timeout  connect 5s
  timeout  client 302s
  timeout  server 302s
  timeout  check 1s

frontend https
  bind 0.0.0.0:443 ssl crt /etc/ssl/server.pem alpn h2,http/1.1
  mode http
  bind-process 1
  default_backend http2_backend

backend http2_backend
  mode http
  balance url_param bh
  #balance roundrobin
  bind-process 1
  server http1.1 127.0.0.1:81 check

listen stats
  bind 10.254.0.10:888 process 1
  mode http
  stats uri /
  stats enable
  stats show-node
  stats refresh 20s
  stats show-legends

and the request:

# curl -v -I -k --http2 https://10.254.0.10:443/index.html?bh=123
*   Trying 10.254.0.10...
* TCP_NODELAY set
* Connected to 10.254.0.10 (10.254.0.10) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  start date: May  9 21:03:10 2018 GMT
*  expire date: May  9 21:03:10 2019 GMT
*  issuer: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa6f4807400)
> HEAD /index.html?bh=123 HTTP/2
> Host: 10.254.0.10
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (16) Error in the HTTP2 framing layer

As I mentioned earlier, when I use roundrobin instead url_param, everything is correct. I also suspected that there might be a dependence on the first request and the use of hash table, but I checked different variants with no effect.

@lukastribus I also can’t reproduce the httpclose issue (provided I use roundrobin or leastconn)

Thanks

12

Thanks Pawel, but given that I cannot reproduce the issue with this config, it would help me if you can comment out almost all lines in your config to figure the smallest set of lines which are absolutely required for the problem to happen. For example you may detect that it doesn’t fail when there is no timeout or no user/group setting, etc. If in the end we only have a frontend, a bind line, a backend, a server, a “balance url_param” line and 2 or 3 lines, it will either indicate that these lines play an important role in the problem, and that others don’t play such a role.

13

It’s absolutely minimal valid config:

global
  daemon
  nbproc  1
  pidfile  /var/run/haproxy.pid
  stats  socket /var/lib/haproxy/stats uid 0 gid 0 mode 0440 process 1
  tune.ssl.default-dh-param  2048

defaults
  timeout  connect 302s
  timeout  client 302s
  timeout  server 302s

frontend https
  bind 0.0.0.0:443 ssl crt /etc/ssl/server.pem alpn h2,http/1.1
  mode http
  default_backend http2_backend

backend http2_backend
  mode http
  balance url_param bh
  #balance roundrobin
  server http1.1 127.0.0.1:81 check

but the same result

14

many thanks! This should help.

Willy

15

Pawel,

Christopher managed to figure where the problem was and to reproduce it. It’s not directly related to the httpclose one though it certainly has some common roots given that it’s related to the fact that H2 closes the requests after sending them. The fix was just merged, you can pick it here, it works for us :

http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=1c10e5b1b95142bb3ac385be1e60d8b180b2e99e

Thanks for your report!

1 Like
16

Well done! I can confirm that after applying the patch everything works as it should! Thanks @willy and @lukastribus I really appreciate your prompt patch !!

# curl -v -I -k --http2 https://10.254.0.10/index.html?bh=123
*   Trying 10.254.0.\10...
* TCP_NODELAY set
* Connected to 10.254.0.10 (10.254.0.10) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  start date: May  9 21:03:10 2018 GMT
*  expire date: May  9 21:03:10 2019 GMT
*  issuer: C=US; ST=CA; L=LA; O=ACME Inc.; OU=IT; CN=example.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f96da006400)
> HEAD /index.html?bh=123 HTTP/2
> Host: 10.254.0.10
> User-Agent: curl/7.54.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
HTTP/2 200 
< date: Wed, 16 May 2018 13:31:58 GMT
date: Wed, 16 May 2018 13:31:58 GMT
< server: Apache/2.4.6 (CentOS)
server: Apache/2.4.6 (CentOS)
< last-modified: Wed, 09 May 2018 21:57:36 GMT
last-modified: Wed, 09 May 2018 21:57:36 GMT
< etag: "1ca-56bccfda5fc1c"
etag: "1ca-56bccfda5fc1c"
< accept-language: bytes
accept-language: bytes
< content-length: 458
content-length: 458
< content-type: text/html; charset=UTF-8
content-type: text/html; charset=UTF-8

< 
* Connection #0 to host 10.254.0.10 left intact
17

thank you for confirming!