I’m trying to use a static site (S3 + Cloudfront) as a backend in my HAProxy configuration. The static service is configured to redirect HTTP requests to HTTPS. So —
# Gives a #301
curl <site>.cloudfrount.net
and
# Gives a 200
curl https://<site>.cloudfront.net
However, if I enter this as a backend in HAProxy —
backend my_server
http-response set-header Strict-Transport-Security max-age=31536000
server my_server <id>.cloudfront.net ssl verify none
I get a bunch of IP address of my_server changed from to logs continuously, and whenever I hit a route which evaluates to use the cloudfront backend, I receive a messageFailed to connect() no free ports`
I suspect this is because HAProxy is falling into a redirect loop. Is this because HAProxy is not making a https call to the server? How can I ensure it does?
First of all you need to specify the port, otherwise haproxy will reuse the same frontend destination port that it has, which not necessarily is the correct one (443).
So it should be:
server my_server <id>.cloudfront.net:443 ssl verify none
Also please share the ouput of haproxy -vv and more of the configuration so that we get a better picture. Especially whatever those frontend and backend contain and inherit from the default configuration.
haproxy --vv output —
2020-07-18 14:53:00,527 INFO spawned: 'haproxy' with pid 18
listening on /usr/share/haproxy/log, starting.
HA-Proxy version 1.8.9-83616ec 2018/05/18
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow -Wno-format-truncation -Wno-null-dereference -Wno-unused-label
OPTIONS = USE_ZLIB=yes USE_OPENSSL=yes USE_LUA=yes USE_PCRE=yes USE_PCRE_JIT=yes
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2p 14 Aug 2018
Running on OpenSSL version : OpenSSL 1.0.2p 14 Aug 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.41 2017-07-05
Running on PCRE version : 8.41 2017-07-05
PCRE library supports JIT : yes
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
@lukastribus The problem with adding a port :443 is that HAProxy isn’t able to connect to the server anymore — it receives a 400 Bad Request from Cloudfront.
curl ' <id>.cloudfront.net' # Gives 301 (redirect to https)
curl ' <id>.cloudfront.net:443' # Gives 400 Bad Request
curl 'https://<id>.cloudfront.net:443' # Gives 200
So it appears that HAProxy doesn’t add the protocol https while making the request. My complete backend block is —
global
ca-base /etc/ssl/certs
chroot /usr/share/haproxy
log /log local0 debug
maxconn 5000
pidfile /var/run/haproxy.pid
resolvers my_dns
<dns>
defaults main
compression algo gzip
maxconn 3000
mode http
option abortonclose
option redispatch
option dontlognull
option forwardfor
option httpclose
option httplog
retries 3
frontend web
<some-acls>
bind :9090
http-request lua.log_request
http-response lua.log_response
maxconn 1000
mode http
monitor-uri /healthcheck
use_backend my_backend if <condition>
backend my_backend
http-response set-header Strict-Transport-Security max-age=31536000
http-response set-header Host <id>.cloudfront.net
option httpchk GET /healthcheck HTTP/1.1\\r\\nHost:\\ <id>.cloudfront.net
server my_server <id>.cloudfront.net:443 sni str(<id>.cloudfront.net) resolvers my_resolver check init-addr libc,none verify none weight 100
I’m saying add :443 to the backend server configuration in haproxy, because otherwise haproxy will connect to port 9090 on the cloundfront server, since that is the frontend port.
This has nothing todo with curl and you should NOT use the same syntax for curl, because it’s nothing like haproxy.
- do you still see this error NOW ?
- you do not appear to have a
my_resolver section in your configuration, instead it seems to be called my_dns?
- you are missing the
ssl keyword on your server line, so haproxy doesn’t know it needs to encrypt the connection
- you are configuring health checks, but that very likely fails, can you elaborate why you need to health-check a cloudfront endpoint anyway?
- those headers you are setting don’t make a lot of sense, the Host header would belong to the http-request, not the response, and the STS header header doesn’t make any sense at all in a HTTP environment
Please try with a simplified backend configuration
backend my_backend
http-request set-header Host <id>.cloudfront.net
server my_server <id>.cloudfront.net:443 sni str(<id>.cloudfront.net) ssl verify none
I’ve redacted some less relavent bits (dns resolvers) so they may be inconsistent in my example — they work fine with other non cloudfront addresses, so I don’t expect the problem to lie there.
I’ve disabled all health checks now and included ssl —
backend my_backend
# Remove the /test prefix in the path
http-request set-path %[path,regsub(^/test/,/)]
http-request set-header Host <id>.cloudfront.net
server my_server <id>.cloudfront.net:443 sni str(<id>.cloudfront.net) resolvers my_dns ssl verify none weight 100
And I still get the free port error —
local0.err: Jul 18 17:49:06 haproxy[18]: Connect() failed for backend my_backend: no free ports.
And end up receiving a 503 Service Unavailable response.
Note: This error disappears if I use a non-CloudFront address.
Kernel either returns EGAIN or EADDRNOTAVAIL.
We need to find out whether the connection setup is bogus (for example DNS has issues), or if the connection setup is actually correct and your kernel rejects this connection (for example due to selinux and other security restrictions).
I suggest you run haproxy through strace -tt and capture the connection setup that ultimately leads to this error message.
So what solved it for me finally was adding resolve-prefer ipv4 to my server line.
Turns out that the frequent —
my_server changed its IP from <ipv6> to <ipv6>
log messages were causing the PORTs to clog up, and this led to the no free ports message.
This seems to be very common with folks trying to use cloudfront as a server for HAProxy, and this issue tracks it —