Capturing HTTP header (X-Forwarded-For with wrong IP: 10.0.0.2)?

Help!
1

Yet another X-Forwarded-For issues (e.g. Need to see client IP in HAProxy logs or X-Forwarded-For not working - #6 by moscardo)… I have read through all of them including the docs, but I am lost.

I am using haproxytech/haproxy-debian:2.0 and having trouble with setting the X-Forwarded-For attribute correctly via HAProxy. I always get 10.0.0.2 from wherever a user connects.

My current (relevant part of) the HAProxy is full of attempts to log everything:

frontend default
    bind *:80
    bind *:443 ssl crt /etc/ssl/private/ # strict-sni

    # default_backend no-match

    http-request set-header X-Forwarded-For %[src]
    http-request redirect scheme https code 301 unless { ssl_fc }
    http-request capture req.hdr(X-Forwarded-For) len 64

    # this is experimental, to see if ELOG needs it
    #http-request set-header X-Forwarded-Proto https if { ssl_fc }
    #http-request set-header X-Forwarded-Proto http if !{ ssl_fc }

    reqadd X-Forwarded-Proto:\ https
    option http-server-close

    option  httplog
    #option  http-server-close
    option  http-keep-alive
    no option logasap   # disable early logging of HTTP requests so that total transfer time is logged

    option forwardfor
    option httpclose

    capture request header Referer len 512
    capture request header Content-Length len 512
    capture request header User-Agent len 64
    capture request header X-Forwarded-For      len 500
    capture request header Host                 len 500
    capture request header X-Request-UID        len 500


    #Generate the X-Haproxy-Unique-ID and log it to make it easy to track requests
    log-format %ci:%cp\ [id=%ID]\ [%t]\ %f\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ {%hrl}\ {%hsl}\ %{+Q}r
    unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
    unique-id-header X-Haproxy-Unique-ID

You can spot the 10.0.0.2 everywhere, which is probably in the X-Forwarded-For (edit: see TCP-dump in the next post, which confirms it)

haproxy-service_haproxy.1.6e5nnop9@s022    | <150>Apr  3 12:22:30 haproxy[1251]: 10.0.0.2:17395 [id=0A000004:47F3_0A000027:01CC_643AC506_03D9:04E3] [03/Apr/2023:12:22:30.509] default be_some.backend/some-service-6 30/0/1/14/45 304 475 - - ---- 7/6/1/1/0 0/0 {10.0.0.2 https://some.url/and/some/path - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605. - some.url -} {} "GET /some/path HTTP/1.1"
haproxy-service_haproxy.1.6e5nnop9@s022    | <150>Apr  3 12:22:30 haproxy[1251]: 10.0.0.2:17394 [id=0A000004:47F2_0A000027:01CC_643AC506_03DA:04E3] [03/Apr/2023:12:22:30.507] default be_some.backend/some-service-6 41/0/1/7/49 304 475 - - ---- 6/5/0/0/0 0/0 {10.0.0.2 https://some.url/and/some/path - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605. - some.url -} {} "GET /some/path HTTP/1.1"
haproxy-service_haproxy.1.6e5nnop9@s022    | <150>Apr  3 12:22:30 haproxy[1251]: 10.0.0.2:17396 [id=0A000004:47F4_0A000027:01CC_643AC506_03DB:04E3] [03/Apr/2023:12:22:30.728] default be_some.backend/some-service-6 24/0/1/13/38 200 955 - - ---- 6/5/0/0/0 0/0 {10.0.0.2 https://some.url/and/some/path - Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605. - some.url -} {} "GET /api/some/path HTTP/1.1"
2

OK, I got a bit further. HAProxy adds an via X-Forwarded-For:

(obtained with ngrep '^GET .* HTTP/1.[01]' on one of the services, removed a bunch of sensitive data

T 10.0.9.107:59440 -> 10.0.9.104:80 [AP] #172
  GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1..host: ... 
... ..sec-fetch-dest: empty..sec-fetch-mode: cors..sec-fetch-site: 
same-origin..dnt: 1..sec-gpc: 1..x-forwarded-for: 10.0.0.2..
x-forwarded-proto: https..x-haproxy-unique-id: .....x-forwarded-for: 
10.0.0.2..connection: close....

Now: 10.0.9.107 is the IP of the HAProxy instance, 10.0.9.104 the IP of the Service (running Apache), both IPs are in the same Docker network, all fine. However, HAProxy, for whatever reason puts 10.0.0.2 into X-Forwarded-For and this is clearly wrong, since that IP is part of Docker’s own network bridge 10.0.0.x, and it’s own IP address on that bridge is 10.0.0.2 (10.0.0.1 is the gateway).

So something is wrong but I don’t know what… any ideas?

HAProxy itself runs in a Docker container where all the frontends of the underlying services are connected via separate networks (the backends are isolated each).

3

From my experience, this is expected. In Bridge mode, Docker cannot see IP addresses from beyond the bridge. That level of isolation is one of the reasons Docker is desired, but it introduces this problem. Your problem is not HAProxy but Docker one. You can bypass this in docker by using network=host in your Docker run command or adding the equivalent into your docker-compose file.
Sources:

2 Likes
4

Ah yes, you’re right. I search for the misconfiguration after the bridge network but you are totally right. I’ll report back…