HAProxy with ALPN support

Help!
1

I have a GO http server where I want to use http/2 with ALPN. I am using https://tools.keycdn.com/http2-test to test for this.

I am using the following code to configure TLS from my GO code.

tlsConfig := &tls.Config{
	PreferServerCipherSuites: true,
	MinVersion:               tls.VersionTLS12,
	SessionTicketsDisabled:   true,
	CipherSuites: []uint16{
		tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
		tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
		// tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, // Go 1.8 only
		// tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,   // Go 1.8 only
		tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
		tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
	},
	ClientAuth: tls.VerifyClientCertIfGiven,
}

If I terminate TLS at my GO server, I see that HTTP/2 with ALPN is working.

Now, I want to use HAProxy to terminate TLS and load balance across multiple replicas of my GO server. I am using HAProxy 1.7.2 in a Ubuntu 16.04 Docker image with the following cipher suite:

# haproxy -vv
HA-Proxy version 1.7.2 2017/01/13
Copyright 2000-2017 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built without Lua support
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
	[COMP] compression
	[TRACE] trace
	[SPOE] spoe

–

global
    daemon
    stats socket /tmp/haproxy
    server-state-file global
    server-state-base /var/state/haproxy/
    maxconn 4000
    # log using a syslog socket
    log /dev/log local0 info
    log /dev/log local0 notice
    
    tune.ssl.default-dh-param 2048
    ssl-default-bind-ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK
    
defaults
    log global

    option http-server-close

    # Disable logging of null connections (haproxy connections like checks).
    # This avoids excessive logs from haproxy internals.
    option dontlognull

    # Maximum time to wait for a connection attempt to a server to succeed.
    timeout connect         50000

    # Maximum inactivity time on the client side.
    # Applies when the client is expected to acknowledge or send data.
    timeout client          50000

    # Inactivity timeout on the client side for half-closed connections.
    # Applies when the client is expected to acknowledge or send data
    # while one direction is already shut down.
    timeout client-fin      50000

    # Maximum inactivity time on the server side.
    timeout server          50000

    # timeout to use with WebSocket and CONNECT
    timeout tunnel          50000

    # default traffic mode is http
    # mode is overwritten in case of tcp services
    mode http

frontend tcp-frontend-key-3443
    bind *:3443
    mode tcp
    default_backend tcp-service-27o4zg

backend tcp-service-27o4zg
    mode tcp
    server server-10.244.5.39 10.244.5.39:50077

This is what I see from curl:

$ curl -v https://api.appscode.info:3443/_appscode/api/health/json
*   Trying 104.198.249.218...
* Connected to api.appscode.info (104.198.249.218) port 3443 (#0)
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 697 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: appscode.info (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=appscode.info
* 	 start date: Tue, 09 May 2017 03:41:00 GMT
* 	 expire date: Mon, 07 Aug 2017 03:41:00 GMT
* 	 issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
* 	 compression: NULL
* ALPN, server did not agree to a protocol
> GET /_appscode/api/health/json HTTP/1.1
> Host: api.appscode.info:3443
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK

My hunch is that somehow a blacklisted cipher is getting selected and causing ALPN to not work. How do I get HTTP/2 with ALPN to work with HAproxy?

2

I don’t see any SSL or ALPN configuration in haproxy - no “ssl” keyword on the bind line, no “alpn” configuration, not even a certificate is configured on haproxy.

I assume you shared the wrong configuration with us?

3

This is actually the relevant part of our HAproxy config. In our setup, we use port 443 for http/1.1 protocol with TLS terminated at HAProxy. ALPN is working for this.

For proper HTTP/2 connection, we have found that we have to terminate TLS at our Go server. GO only supports ‘h2’ and does not support h2c. So, if we terminate TLS at Haproxy, our http/2 clients do not work. FYI, this server and client is using Google’s GRPC protocol.

The problem I had was that in my Go code, I only used h2 as Next protocol. So, when I used curl, it could not find a server protocol. Now, I use both h2 and http/1.1 as Next protocol in my Go code. So, curl is using ALPN. I have also built curl with http2 support. I see that curl is using ALPN in http/2 mode, too.

Sorry for the false alert.