HAProxy use a backend that is only accessible on HTTPS

I’m trying to use a static site (S3 + Cloudfront) as a backend in my HAProxy configuration. The static service is configured to redirect HTTP requests to HTTPS. So —

# Gives a #301
curl <site>.cloudfrount.net

and

# Gives a 200
curl https://<site>.cloudfront.net

However, if I enter this as a backend in HAProxy —

backend my_server
    http-response set-header Strict-Transport-Security max-age=31536000
    server my_server <id>.cloudfront.net ssl verify none

I get a bunch of IP address of my_server changed from to logs continuously, and whenever I hit a route which evaluates to use the cloudfront backend, I receive a messageFailed to connect() no free ports`

I suspect this is because HAProxy is falling into a redirect loop. Is this because HAProxy is not making a https call to the server? How can I ensure it does?

First of all you need to specify the port, otherwise haproxy will reuse the same frontend destination port that it has, which not necessarily is the correct one (443).

So it should be:

server my_server <id>.cloudfront.net:443 ssl verify none

Also please share the ouput of haproxy -vv and more of the configuration so that we get a better picture. Especially whatever those frontend and backend contain and inherit from the default configuration.

haproxy --vv output —

2020-07-18 14:53:00,527 INFO spawned: 'haproxy' with pid 18
listening on /usr/share/haproxy/log, starting.
HA-Proxy version 1.8.9-83616ec 2018/05/18
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-format-truncation -Wno-null-dereference -Wno-unused-label
  OPTIONS = USE_ZLIB=yes USE_OPENSSL=yes USE_LUA=yes USE_PCRE=yes USE_PCRE_JIT=yes

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2p  14 Aug 2018
Running on OpenSSL version : OpenSSL 1.0.2p  14 Aug 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.41 2017-07-05
Running on PCRE version : 8.41 2017-07-05
PCRE library supports JIT : yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace

@lukastribus The problem with adding a port :443 is that HAProxy isn’t able to connect to the server anymore — it receives a 400 Bad Request from Cloudfront.

curl ' <id>.cloudfront.net'  # Gives 301 (redirect to https)
curl ' <id>.cloudfront.net:443' # Gives 400 Bad Request
curl 'https://<id>.cloudfront.net:443' # Gives 200

So it appears that HAProxy doesn’t add the protocol https while making the request. My complete backend block is —

  global
    ca-base /etc/ssl/certs
    chroot /usr/share/haproxy
    log /log local0 debug
    maxconn 5000
    pidfile /var/run/haproxy.pid

  resolvers my_dns
    <dns>

  defaults main
    compression algo gzip
    maxconn 3000
    mode http
    option abortonclose
    option redispatch
    option dontlognull
    option forwardfor
    option httpclose
    option httplog
    retries 3

  frontend web
    <some-acls>

    bind :9090

    http-request lua.log_request
    http-response lua.log_response

    maxconn 1000
    mode http
    monitor-uri /healthcheck

    use_backend my_backend if <condition>

    backend my_backend
      http-response set-header Strict-Transport-Security max-age=31536000
      http-response set-header Host <id>.cloudfront.net
      option httpchk GET /healthcheck HTTP/1.1\\r\\nHost:\\ <id>.cloudfront.net
      server my_server <id>.cloudfront.net:443 sni str(<id>.cloudfront.net) resolvers my_resolver check init-addr libc,none verify none weight 100

I’m saying add :443 to the backend server configuration in haproxy, because otherwise haproxy will connect to port 9090 on the cloundfront server, since that is the frontend port.

This has nothing todo with curl and you should NOT use the same syntax for curl, because it’s nothing like haproxy.

  • do you still see this error NOW ?
  • you do not appear to have a my_resolver section in your configuration, instead it seems to be called my_dns?
  • you are missing the ssl keyword on your server line, so haproxy doesn’t know it needs to encrypt the connection
  • you are configuring health checks, but that very likely fails, can you elaborate why you need to health-check a cloudfront endpoint anyway?
  • those headers you are setting don’t make a lot of sense, the Host header would belong to the http-request, not the response, and the STS header header doesn’t make any sense at all in a HTTP environment

Please try with a simplified backend configuration

backend my_backend
  http-request set-header Host <id>.cloudfront.net
  server my_server <id>.cloudfront.net:443 sni str(<id>.cloudfront.net) ssl verify none

I’ve redacted some less relavent bits (dns resolvers) so they may be inconsistent in my example — they work fine with other non cloudfront addresses, so I don’t expect the problem to lie there.

I’ve disabled all health checks now and included ssl


  backend my_backend
    # Remove the /test prefix in the path
    http-request set-path %[path,regsub(^/test/,/)]
    http-request set-header Host <id>.cloudfront.net
    server my_server <id>.cloudfront.net:443 sni str(<id>.cloudfront.net) resolvers my_dns  ssl verify none weight 100

And I still get the free port error —

local0.err: Jul 18 17:49:06 haproxy[18]: Connect() failed for backend my_backend: no free ports.

And end up receiving a 503 Service Unavailable response.

Note: This error disappears if I use a non-CloudFront address.

Kernel either returns EGAIN or EADDRNOTAVAIL.

We need to find out whether the connection setup is bogus (for example DNS has issues), or if the connection setup is actually correct and your kernel rejects this connection (for example due to selinux and other security restrictions).

I suggest you run haproxy through strace -tt and capture the connection setup that ultimately leads to this error message.

So what solved it for me finally was adding resolve-prefer ipv4 to my server line.
Turns out that the frequent —

my_server changed its IP from <ipv6> to <ipv6>

log messages were causing the PORTs to clog up, and this led to the no free ports message.
This seems to be very common with folks trying to use cloudfront as a server for HAProxy, and this issue tracks it —