multiple “interceptor” servers (in http mode) that accept the initial connection and send it (using send-proxy-v2-ssl) to multiple:
“routers” that know about the various backends to which we need to route the requests
Everything is working great, except when CloudFlare is involved. In the frontend on “router” we’re using:
http-request set-src hdr(x-forwarded-for) if is_cloudflare_src
and we find:
the connections are properly coming from various Cloudflare IPs within their published ranges (188.114.110.101, 172.68.133.229, 162.158.255.95) – this is OK
between “interceptor” and “router”, a single connection is fronted by a PROXY header with the actual outside layer 3/4 connection information – this is OK
inside that stream are multiple requests – this is OK :
POST /message-bus/c009f71cb32a656f70b46a6db8c6ad42/poll?dlp=t HTTP/1.1
Host: community.customer.com
X-Forwarded-For: 192.0.2.18
CF-Connecting-IP: 192.0.2.18
POST /message-bus/97361f6ea08fc8c7a1eda4d34435907b/poll?dlp=t HTTP/1.1
Host: community.customer.com
X-Forwarded-For: 198.51.100.37
CF-Connecting-IP: 198.51.100.37
GET /admin/users/195.json?_=1556350420519 HTTP/1.1
Host: community.customer.com
X-Forwarded-For: 203.0.113.39
CF-Connecting-IP: 203.0.113.39
what’s being reported by “router” is NOT OK as it has the source IP as the X-Forwarded-For IP from the first request:
I think that with our setup haproxy should be inspecting the X-Forwarded-For of each request, but it appears to be using the first IP seen for every single subsequent request.
Where’s the problem? Is this haproxy’s error or do we need to tell it to do something different?
This is a total guess but I wonder if the http-request set-src overwrites the connection source instead of the request source and then on the next request on that connection it doesn’t match is_cloudflare_src
This cannot possibly work. The PROXY protocol you are using between the interceptor and the router is per connection. So when Cloudflare multiplexes transaction from multiple clients, so will the interceptor, and only the first IP will show up at your router.
You need to work with the - per request - connection headers, instead of the proxy protocol.
That’s fine - at the router I’m asking “does this connection come from cloudflare”? Once that’s known I can use the IP listed in X-Forwarded-For for all requests in that connection. Which is working for the first request:
http-request set-src hdr(x-forwarded-for) if is_cloudflare_src
but not on any subsequent requests in the connection.
From reading the source I think I’m right - looks like tcp_action_req_set_src replaces the connection source address as there’s no separate “request source address”. And the next-time through the code path, the source address is the IP from X-F-F instead of cloudflare, so X-F-F on the next request in the connection doesn’t get used.
Looks like I can handle my situation by using setting a flag on the session (connection?) as follows:
acl cdn_src var(sess.cdn) -m found
acl cdn_src src -f trusted_ip_file
http-request set-var(sess.cdn) always_true if { http_first_req } cdn_src
http-request set-src hdr(x-forwarded-for) if cdn_src
This way haproxy can remember that we trust the X-F-F header when processing pipelined requests with our setup. Deployed to production with success!
@lukastribus Do you think it’s worth including a note in the documentation for http-request set-src (and presumably others?) that it affects the connection, not the individual request?
n.b. I used always_true there because I couldn’t figure out a way to just use a string (“1”) in the set-var since it needs a fetch request.
HA-Proxy version 1.8.17 2019/01/08
Copyright 2000-2019 Willy Tarreau <willy@haproxy.org>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=yes USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g 1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
Our production configuration is too huge but I reproduced the problem and fix locally on 1.9.6 with a minimal configuration:
For the 4 requests above I see in the log the source IPs:
192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.3
The first 3 being correct, the last one for the lack of a X-Forwarded-For header, has the previous set-src in there (we may expect the real socket IP instead).
So I’m unable to reproduce the issue in the first place. Please share the entire configuration of your lab repro, including the entire default and global section.
That’s not what you reported here initially, which is that the first IP of the X-F-F header is set for all subsequent requests of the same connection - this I was unable to reproduce. This would lead to issues with Cloudflare (as multiplexed requests come from different clients), which is what you reported here.
What I reproduced does not explain the issue with Cloudflare, because cloudflare would always set the X-F-F header.
OK, fair enough, a different manifestation of the same root cause.
Here’s a complete reproduction (the ONLY thing changed is the IP address reported by the reflector at ip4.supermathie.net and the telnet source address is 172.18.0.1):
haproxy.cfg:
global
defaults
log global
log /dev/log len 65535 daemon
option httplog
retries 3
timeout server 60s
timeout client 60s
timeout connect 5s
frontend fe_default
mode http
bind *:80
http-request set-src hdr(x-forwarded-for) if { src 172.18.0.1 }
default_backend be_ip
backend be_ip
mode http
server ip4 ip4.supermathie.net:80
requests:
○ → telnet 172.18.0.2 80
Trying 172.18.0.2...
Connected to 172.18.0.2.
Escape character is '^]'.
GET / HTTP/1.1
Host: ip4.supermathie.net
X-Forwarded-For: 192.168.0.1
HTTP/1.1 200 OK
Server: nginx/1.10.3
Date: Thu, 02 May 2019 02:09:25 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
e
192.0.2.1
0
GET / HTTP/1.1
Host: ip4.supermathie.net
X-Forwarded-For: 192.168.0.2
HTTP/1.1 200 OK
Server: nginx/1.10.3
Date: Thu, 02 May 2019 02:09:35 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
e
192.0.2.1
0
GET / HTTP/1.1
Host: ip4.supermathie.net
X-Forwarded-For: 192.168.0.3
HTTP/1.1 200 OK
Server: nginx/1.10.3
Date: Thu, 02 May 2019 02:09:45 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
e
192.0.2.1
0
HA-Proxy version 1.9.6 2019/03/29 - https://haproxy.org/
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.0j 20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTX side=FE|BE
h2 : mode=HTTP side=FE
<default> : mode=HTX side=FE|BE
<default> : mode=TCP|HTTP side=FE|BE
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace