Cloudflare: haproxy is using the wrong IP for HTTP requests after the first

We are using haproxy 1.8.17 in a two-stage setup:

  • multiple “interceptor” servers (in http mode) that accept the initial connection and send it (using send-proxy-v2-ssl) to multiple:
  • “routers” that know about the various backends to which we need to route the requests

Everything is working great, except when CloudFlare is involved. In the frontend on “router” we’re using:

http-request set-src hdr(x-forwarded-for) if is_cloudflare_src

and we find:

  • the connections are properly coming from various Cloudflare IPs within their published ranges (188.114.110.101, 172.68.133.229, 162.158.255.95) – this is OK

  • between “interceptor” and “router”, a single connection is fronted by a PROXY header with the actual outside layer 3/4 connection information – this is OK

  • inside that stream are multiple requests – this is OK :

    POST /message-bus/c009f71cb32a656f70b46a6db8c6ad42/poll?dlp=t HTTP/1.1
    Host: community.customer.com
    X-Forwarded-For: 192.0.2.18
    CF-Connecting-IP: 192.0.2.18
    
    POST /message-bus/97361f6ea08fc8c7a1eda4d34435907b/poll?dlp=t HTTP/1.1
    Host: community.customer.com
    X-Forwarded-For: 198.51.100.37
    CF-Connecting-IP: 198.51.100.37
    
    GET /admin/users/195.json?_=1556350420519 HTTP/1.1
    Host: community.customer.com
    X-Forwarded-For: 203.0.113.39
    CF-Connecting-IP: 203.0.113.39
    
  • what’s being reported by “router” is NOT OK as it has the source IP as the X-Forwarded-For IP from the first request:

192.0.2.18:30658 [30/Apr/2019:18:54:41.486] app community/server03 0/0/1/61/62 200 1904 - - ---- 718/718/29/12/0 0/0 "GET /admin/users/195.json?_=1556350420519 HTTP/1.1"

I think that with our setup haproxy should be inspecting the X-Forwarded-For of each request, but it appears to be using the first IP seen for every single subsequent request.

Where’s the problem? Is this haproxy’s error or do we need to tell it to do something different?


This is a total guess but I wonder if the http-request set-src overwrites the connection source instead of the request source and then on the next request on that connection it doesn’t match is_cloudflare_src

This cannot possibly work. The PROXY protocol you are using between the interceptor and the router is per connection. So when Cloudflare multiplexes transaction from multiple clients, so will the interceptor, and only the first IP will show up at your router.

You need to work with the - per request - connection headers, instead of the proxy protocol.

That’s fine - at the router I’m asking “does this connection come from cloudflare”? Once that’s known I can use the IP listed in X-Forwarded-For for all requests in that connection. Which is working for the first request:

http-request set-src hdr(x-forwarded-for) if is_cloudflare_src

but not on any subsequent requests in the connection.

From reading the source I think I’m right - looks like tcp_action_req_set_src replaces the connection source address as there’s no separate “request source address”. And the next-time through the code path, the source address is the IP from X-F-F instead of cloudflare, so X-F-F on the next request in the connection doesn’t get used.

… I’m not sure what you mean by this.

Looks like I can handle my situation by using setting a flag on the session (connection?) as follows:

      acl cdn_src var(sess.cdn) -m found
      acl cdn_src src -f trusted_ip_file

      http-request set-var(sess.cdn) always_true if { http_first_req } cdn_src
      http-request set-src hdr(x-forwarded-for) if cdn_src

This way haproxy can remember that we trust the X-F-F header when processing pipelined requests with our setup. Deployed to production with success!

@lukastribus Do you think it’s worth including a note in the documentation for http-request set-src (and presumably others?) that it affects the connection, not the individual request?


n.b. I used always_true there because I couldn’t figure out a way to just use a string (“1”) in the set-var since it needs a fetch request.

First we will have to figure out if the actual behavior here is really expected, or if this is a bug.

Can you provide the output of haproxy -vv and more complete configuration?

1 Like

haproxy -vv:

HA-Proxy version 1.8.17 2019/01/08
Copyright 2000-2019 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=yes USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
	[SPOE] spoe
	[COMP] compression
	[TRACE] trace

Our production configuration is too huge but I reproduced the problem and fix locally on 1.9.6 with a minimal configuration:

frontend fe_default
  mode http
  bind *:80
  http-request set-src hdr(x-forwarded-for) if { var(sess.cdn) -m found } { hdr(CDN) YES }
  default_backend be_ip
frontend fe_default
  mode http
  bind *:80
  http-request set-var(sess.cdn) always_true if { http_first_req } { hdr(CDN) YES }
  http-request set-src hdr(x-forwarded-for) if { var(sess.cdn) -m found }
  default_backend be_ip

and then sending requests such as:

GET / HTTP/1.1
Host: localhost
CDN: YES
X-Forwarded-For: 192.168.0.1

GET / HTTP/1.1
Host: localhost
X-Forwarded-For: 192.168.0.2

GET / HTTP/1.1
Host: localhost
X-Forwarded-For: 192.168.0.3

GET / HTTP/1.1
Host: localhost

then watching the logs.

In Haproxy 1.8.17, with a frontend configuration like this:

frontend myfrontend
        bind :80
        http-request set-src hdr(x-forwarded-for)
        default_backend be_ip

For the 4 requests above I see in the log the source IPs:

192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.3

The first 3 being correct, the last one for the lack of a X-Forwarded-For header, has the previous set-src in there (we may expect the real socket IP instead).

So I’m unable to reproduce the issue in the first place. Please share the entire configuration of your lab repro, including the entire default and global section.

But, you did reproduce it. You should have seen the real socket IP in the log for the fourth request.

I’ll share a more complete set up when I get back to my desk.

That’s not what you reported here initially, which is that the first IP of the X-F-F header is set for all subsequent requests of the same connection - this I was unable to reproduce. This would lead to issues with Cloudflare (as multiplexed requests come from different clients), which is what you reported here.

What I reproduced does not explain the issue with Cloudflare, because cloudflare would always set the X-F-F header.

OK, fair enough, a different manifestation of the same root cause.

Here’s a complete reproduction (the ONLY thing changed is the IP address reported by the reflector at ip4.supermathie.net and the telnet source address is 172.18.0.1):

haproxy.cfg:

global

defaults
  log global
  log /dev/log len 65535 daemon
  option httplog
  retries 3
  timeout server 60s
  timeout client 60s
  timeout connect 5s

frontend fe_default
  mode http
  bind *:80
  http-request set-src hdr(x-forwarded-for) if { src 172.18.0.1 }
  default_backend be_ip

backend be_ip
  mode http
  server ip4 ip4.supermathie.net:80

requests:

○ → telnet 172.18.0.2 80
Trying 172.18.0.2...
Connected to 172.18.0.2.
Escape character is '^]'.
GET / HTTP/1.1
Host: ip4.supermathie.net
X-Forwarded-For: 192.168.0.1

HTTP/1.1 200 OK
Server: nginx/1.10.3
Date: Thu, 02 May 2019 02:09:25 GMT
Content-Type: text/plain
Transfer-Encoding: chunked

e
192.0.2.1

0

GET / HTTP/1.1
Host: ip4.supermathie.net
X-Forwarded-For: 192.168.0.2

HTTP/1.1 200 OK
Server: nginx/1.10.3
Date: Thu, 02 May 2019 02:09:35 GMT
Content-Type: text/plain
Transfer-Encoding: chunked

e
192.0.2.1

0

GET / HTTP/1.1
Host: ip4.supermathie.net
X-Forwarded-For: 192.168.0.3

HTTP/1.1 200 OK
Server: nginx/1.10.3
Date: Thu, 02 May 2019 02:09:45 GMT
Content-Type: text/plain
Transfer-Encoding: chunked

e
192.0.2.1

0

log:

<30>May  2 02:09:25 haproxy[6]: 192.168.0.1:37584 [02/May/2019:02:09:12.976] fe_default be_ip/ip4 12771/0/23/34/12828 200 180 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
<30>May  2 02:09:35 haproxy[6]: 192.168.0.1:37584 [02/May/2019:02:09:32.606] fe_default be_ip/ip4 3349/0/0/20/3369 200 180 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"
<30>May  2 02:09:45 haproxy[6]: 192.168.0.1:37584 [02/May/2019:02:09:40.478] fe_default be_ip/ip4 4493/0/0/37/4530 200 180 - - ---- 1/1/0/1/0 0/0 "GET / HTTP/1.1"

this was run against:

HA-Proxy version 1.9.6 2019/03/29 - https://haproxy.org/
Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_ZLIB=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0j  20 Nov 2018
Running on OpenSSL version : OpenSSL 1.1.0j  20 Nov 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with multi-threading support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE
              h2 : mode=HTTP       side=FE
       <default> : mode=HTX        side=FE|BE
       <default> : mode=TCP|HTTP   side=FE|BE

Available filters :
	[SPOE] spoe
	[COMP] compression
	[CACHE] cache
	[TRACE] trace

Ok I got it. I went through your posts a little too fast is why it took some time to register…

We need to understand if this can be fixed, or if we need to explain this in the docs.

It’s not a regression - from the initial commit 2fbcafc9ce 4 years ago until latest HEAD the behavior is the same.

I have filed a bug here so this can be discussed with the developers.

Thanks for the report.

1 Like

please do tag me as @supermathie on the github issue as well, I’ll keep :eyes: on it.

Thanks!

Tagged you there, not sure if that’s enough. Anyway you can just hit subscribe on the right.

1 Like