# Send-proxy not modifying some traffic with proxy ip/port details
**URL:** https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336
**Category:** Help!
**Created:** [December 19, 2018, 5:56am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336 "2018-12-19T05:56:45Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@roobesh](https://discourse.haproxy.org/u/roobesh)
#### Post date: [December 19, 2018, 5:56am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/1 "2018-12-19T05:56:45Z")
We are running below version of code
# haproxy -vv
HA-Proxy version 1.8.14-1ppa1~trusty 2018/09/23
Copyright 2000-2018 Willy Tarreau [willy@haproxy.org](mailto:willy@haproxy.org)
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D\_FORTIFY\_SOURCE=2
OPTIONS = USE\_GETADDRINFO=1 USE\_ZLIB=1 USE\_REGPARM=1 USE\_OPENSSL=1 USE\_LUA=1 USE\_PCRE=1 USE\_PCRE\_JIT=1 USE\_NS=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP\_TRANSPARENT IPV6\_TRANSPARENT IP\_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.31 2012-07-06
Running on PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (libpcre build without JIT?)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
- We have send-proxy configured and running fine all these days. Noticed when we started logging, some traffic which passed from haproxy to backend server noticed that send-proxy is not actually modifying the proxy ip/port, instead retaining same client ip/port like below,
PROXY TCP4 192.220.26.39 192.220.26.39 45066 45066
PROXY TCP4 192.220.26.39 192.220.26.39 45075 45075
We tried to sniff the packet the moment it leaves haproxy and above is what we see. This is only happening only sometimes for some traffic and rest of the traffic properly looks good.
Could you please see this is a known bug in the version we running or something known issue ?
Thanks for your help.
-Roobesh G M
---
### Author:  [@lukastribus](https://discourse.haproxy.org/u/lukastribus)
#### Post date: [December 19, 2018, 8:56am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/2 "2018-12-19T08:56:03Z")
There is no known issue explaining what you see.
Can you share the configuration please?
---
### Author:  [@roobesh](https://discourse.haproxy.org/u/roobesh)
#### Post date: [December 19, 2018, 9:35am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/3 "2018-12-19T09:35:28Z")
Thanks for your response. Here you go,
global
# Docker image doesn’t run rsyslogd
# so send logs to docker host
# log-send-hostname @LOG\_HOSTNAME@
log-tag haproxy-tst
chroot /var/lib/haproxy
maxconn 500000
user haproxy
group haproxy
log /dev/log len 8192 local1
log /dev/log len 8192 local2 err
stats timeout 2m
tune.maxrewrite 1024
# we have seen requests and responses including large cookies/MCP headers that failed
tune.bufsize 18432
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
lua-load /etc/haproxy/bdcast.lua
nbproc 8
stats socket ipv4@127.0.0.1:9035 level admin process 1
stats socket ipv4@172.17.0.2:9035 level admin process 1
stats socket ipv4@10.251.1.1:9035 level admin process 1
stats socket ipv4@172.19.0.4:9035 level admin process 1
stats socket ipv4@127.0.0.1:9036 level admin process 2
stats socket ipv4@172.17.0.2:9036 level admin process 2
stats socket ipv4@10.251.1.1:9036 level admin process 2
stats socket ipv4@172.19.0.4:9036 level admin process 2
stats socket ipv4@127.0.0.1:9037 level admin process 3
stats socket ipv4@172.17.0.2:9037 level admin process 3
stats socket ipv4@10.251.1.1:9037 level admin process 3
stats socket ipv4@172.19.0.4:9037 level admin process 3
stats socket ipv4@127.0.0.1:9038 level admin process 4
stats socket ipv4@172.17.0.2:9038 level admin process 4
stats socket ipv4@10.251.1.1:9038 level admin process 4
stats socket ipv4@172.19.0.4:9038 level admin process 4
stats socket ipv4@127.0.0.1:9039 level admin process 5
stats socket ipv4@172.17.0.2:9039 level admin process 5
stats socket ipv4@10.251.1.1:9039 level admin process 5
stats socket ipv4@172.19.0.4:9039 level admin process 5
stats socket ipv4@127.0.0.1:9040 level admin process 6
stats socket ipv4@172.17.0.2:9040 level admin process 6
stats socket ipv4@10.251.1.1:9040 level admin process 6
stats socket ipv4@172.19.0.4:9040 level admin process 6
stats socket ipv4@127.0.0.1:9041 level admin process 7
stats socket ipv4@172.17.0.2:9041 level admin process 7
stats socket ipv4@10.251.1.1:9041 level admin process 7
stats socket ipv4@172.19.0.4:9041 level admin process 7
stats socket ipv4@127.0.0.1:9042 level admin process 8
stats socket ipv4@172.17.0.2:9042 level admin process 8
stats socket ipv4@10.251.1.1:9042 level admin process 8
stats socket ipv4@172.19.0.4:9042 level admin process 8
stats socket /var/run/haproxy-1.sock mode 600 level admin process 1
stats socket /var/run/haproxy-2.sock mode 600 level admin process 2
stats socket /var/run/haproxy-3.sock mode 600 level admin process 3
stats socket /var/run/haproxy-4.sock mode 600 level admin process 4
stats socket /var/run/haproxy-5.sock mode 600 level admin process 5
stats socket /var/run/haproxy-6.sock mode 600 level admin process 6
stats socket /var/run/haproxy-7.sock mode 600 level admin process 7
stats socket /var/run/haproxy-8.sock mode 600 level admin process 8
listen stats
bind 127.0.0.1:1936 process 1
bind 172.17.0.2:1936 process 1
bind 10.10.1.1:1936 process 1
bind 172.19.0.4:1936 process 1
bind 127.0.0.1:1937 process 2
bind 172.17.0.2:1937 process 2
bind 10.10.1.1:1937 process 2
bind 172.19.0.4:1937 process 2
bind 127.0.0.1:1938 process 3
bind 172.17.0.2:1938 process 3
bind 10.10.1.1:1938 process 3
bind 172.19.0.4:1938 process 3
bind 127.0.0.1:1939 process 4
bind 172.17.0.2:1939 process 4
bind 10.10.1.1:1939 process 4
bind 172.19.0.4:1939 process 4
bind 127.0.0.1:1940 process 5
bind 172.17.0.2:1940 process 5
bind 10.10.1.1:1940 process 5
bind 172.19.0.4:1940 process 5
bind 127.0.0.1:1941 process 6
bind 172.17.0.2:1941 process 6
bind 10.10.1.1:1941 process 6
bind 172.19.0.4:1941 process 6
bind 127.0.0.1:1942 process 7
bind 172.17.0.2:1942 process 7
bind 10.10.1.1:1942 process 7
bind 172.19.0.4:1942 process 7
bind 127.0.0.1:1943 process 8
bind 172.17.0.2:1943 process 8
bind 10.10.1.1:1943 process 8
bind 172.19.0.4:1943 process 8
mode http
stats enable
stats uri /
timeout connect 10s
timeout client 1m
timeout server 1m
acl service\_down nbsrv(lb-backend-tst) lt 1
monitor-uri /alive
monitor fail if service\_down
defaults
log global
# log as JSON, we have an ELK stack
log-format ‘{“type”:“haproxy-tst”,“client\_ip”:"%ci",“client\_port”:%cp,“timestamp”:%Ts,“accept\_date”:"%tr",“frontend\_name”:"%ft",“backend\_name”:"%b",“server\_name”:"%s",“TR”:%TR,“Tw”:%Tw,“Tc”:%Tc,“Tr”:%Tr,“Ta”:%Ta,“http\_status\_code”:%ST,“bytes\_read”:%B,“bytes\_uploaded”:%U,“termination\_state”:"%tsc",“actconn”:%ac,“feconn”:%fc,“beconn”:%bc,“srvconn”:%sc,“retries”:%rc,“srv\_queue”:%sq,“backend\_queue”:%bq,“http\_method”:"%HM",“http\_uri”:"%{+E}HU",“http\_version”:"%HV"}’
#option httplog
option log-separate-errors
maxconn 100000
http-check disable-on-404
monitor-uri /alive
# HTTP mode is enabled per default for backends; also accept invalid reponses
mode http
option accept-invalid-http-response
option accept-invalid-http-request
timeout connect 25s
timeout client 3h
timeout server 10m
timeout http-request 25s
# Use all backup servers in case all operational servers are down
option allbackups
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
backend lb-backend-tst
# restrict overall connections to the sum of maxconns of all active backend servers
fullconn 250000
acl blocked\_host src -f /etc/haproxy/lists/ip\_blocklist.lst
http-request deny if blocked\_host
# block unresolvable known hosts
acl unresolvable\_host req.hdr(host) -f /etc/haproxy/lists/unresolvable\_hosts.lst
http-request deny if unresolvable\_host
# delete untrusted incoming XFF header
reqidel ^X-Forwarded-For:.\*
#Call lua script to broadcast
acl is\_tst\_rt\_url hdr(Host) -i [xx.yy.com](http://xx.yy.com)
acl is\_tst\_rt\_url urlp(rtparams) -m found
http-request lua.broadcast\_txn if is\_tst\_rt\_url is\_secure\_rt\_url
# stickyness configuration for load balancing and session validity (Host header, source IP)
http-request set-header X-LB %[req.hdr(Host),lower]_%[src]_%[src\_port]
balance hdr(X-LB)
# do compression on outgoing traffic
compression algo gzip
compression type text/css text/html text/javascript application/javascript text/plain text/xml application/json
hash-type consistent
# configure thresholds for filter nodes
default-server inter 3s fastinter 1s downinter 500ms fall 3 rise 2 slowstart 10s
server node00 10.10.2.3:8080 check maxconn 25000 send-proxy
server node01 10.10.2.4:8080 check maxconn 25000 send-proxy
server node02 10.10.2.5:8080 check maxconn 25000 send-proxy
server node03 10.10.2.6:8080 check maxconn 25000 send-proxy
server node04 10.10.2.7:8080 check maxconn 25000 send-proxy
server node05 10.10.2.8:8080 check maxconn 25000 send-proxy
server node06 10.10.2.9:8080 check maxconn 25000 send-proxy
server node07 10.10.2.10:8080 check maxconn 25000 send-proxy
frontend region1
bind 192.168.123.110:80
monitor fail if { nbsrv(lb-backend-tst) lt 1 }
stick-table type string size 1k expire 1m store http\_req\_rate(1s)
acl is\_tst\_it\_url hdr(Host) -i [xx.yy.com](http://xx.yy.com)
http-request track-sc0 hdr(Host) if is\_tst\_it\_url
http-request deny if { sc0\_http\_req\_rate gt 1 } is\_tst\_it\_url
tcp-request connection accept if { src -f /etc/haproxy/lists/whitelist.lst }
default\_backend lb-backend-tst
Points:
1. Issue is not always happening , occassionally we facing this issue from some IPs.
2. example: x.x.x.x:30000 client IP reaching y.y.y.y:80, using send-proxy always that is been seen by backend as’PROXY TCP4 x.x.x.x y.y.y.y 30000 80’ in the format of ‘PROXY INET:PROTO4 clientip proxyip client-sourceport proxy-port’.
But some requests are just showing as ‘PROXY TCP4 x.x.x.x x.x.x.x 30000 30000’
No Idea why ?
-Roobesh G M
---
### Author:  [@roobesh](https://discourse.haproxy.org/u/roobesh)
#### Post date: [December 20, 2018, 6:44am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/4 "2018-12-20T06:44:27Z")
> [@lukastribus](#):
>
> There is no known issue explaining what you see.
Any thoughts how to troubleshoot why occasionally send-proxy is not actually sending right headers ?
PROXY TCP4
instead wrongly some times showing as
PROXY TCP4
---
### Author:  [@mzielin](https://discourse.haproxy.org/u/mzielin)
#### Post date: [December 20, 2018, 2:23pm UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/5 "2018-12-20T14:23:04Z")
Apparently the situation is the same as in [https://stackoverflow.com/questions/11417187/getsockopt-so-original-dst-occasionally-returns-client-address](https://stackoverflow.com/questions/11417187/getsockopt-so-original-dst-occasionally-returns-client-address)
commenting out getsockopt(… SO\_ORIGINAL\_DST) and only use getsockname() fixed the issue so far.
Under high traffic the PROXY line contained approx once out of 10000 times the source IP as destination instead the destination IP.
---
### Author:  [@lukastribus](https://discourse.haproxy.org/u/lukastribus)
#### Post date: [December 20, 2018, 9:25pm UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/6 "2018-12-20T21:25:27Z")
@mzielin Thanks, that’s interesting, so it seams that SO\_ORIGINAL\_DST may have a reliability issue. This would be a bug in the kernel, rather than haproxy.
What kernel are you guys running exactly?
@roobesh can you try recompiling haproxy with `USE_TPROXY=` (nothing after the equal sign) ? This disables the mentioned code path (along with TPROXY support).
---
### Author:  [@roobesh](https://discourse.haproxy.org/u/roobesh)
#### Post date: [December 21, 2018, 4:02am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/7 "2018-12-21T04:02:55Z")
> [@lukastribus](#):
>
> SO\_ORIGINAL\_DST
@lukastribus We are running Ubuntu 18.04 with kernel 4.15.0-39-generic
You mean disabling all the below options and recompile or just adjust USE\_TPROXY=1 as USE\_TPROXY=
./Makefile:# USE\_TPROXY : enable transparent proxy. Automatic.
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile: USE\_TPROXY = implicit
./Makefile:USE\_TPROXY = 1
Also looks like SO\_ORIGINAL\_DST reliability issue is retained from older version even to the latest what we running looks like (going through other forums).
@Martin Thanks much for narrow down on the cause.
-RGM
---
### Author:  [@lukastribus](https://discourse.haproxy.org/u/lukastribus)
#### Post date: [December 21, 2018, 7:50am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/8 "2018-12-21T07:50:25Z")
@roobesh No, don’t modify any files, just add `USE_TPROXY=` to your make command, for example:
```
make clean; make TARGET=linux2628 CPU=native USE_GETADDRINFO=1 \
USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_TPROXY=
```
---
### Author:  [@roobesh](https://discourse.haproxy.org/u/roobesh)
#### Post date: [December 26, 2018, 5:42am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/9 "2018-12-26T05:42:05Z")
@lukastribus Thanks for your response. We will compile a fresh binary with the options provided by you above.
Can you clarify, getsockopt(… SO\_ORIGINAL\_DST) is only used by send-proxy feature or any other feature in haproxy is using this often ? Basically trying to understand what are the cases where destination-ip needed to be looked into by haproxy other than send-proxy feature. As an alternative what @Martin tried by fixing using this getsockname() solved the issue, but would like understand can there be any other broken area due to this fix ?
---
### Author:  [@roobesh](https://discourse.haproxy.org/u/roobesh)
#### Post date: [December 26, 2018, 6:11am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/10 "2018-12-26T06:11:50Z")
@lukastribus Can we create this as a bug with haproxy and update the source-code to not use getsockopt(… SO\_ORIGINAL\_DST) and find an alternative for this ? getsockname() fixed the issue.
---
### Author:  [@lukastribus](https://discourse.haproxy.org/u/lukastribus)
#### Post date: [December 26, 2018, 9:20am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/11 "2018-12-26T09:20:12Z")
Let’s continue the discussion in the mailing list, where you just posted:
[https://www.mail-archive.com/haproxy@formilux.org/msg32198.html](https://www.mail-archive.com/haproxy@formilux.org/msg32198.html)
Otherwise we fragment the information in different forums.
---
### Author:  [@roobesh](https://discourse.haproxy.org/u/roobesh)
#### Post date: [December 26, 2018, 10:58am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/12 "2018-12-26T10:58:54Z")
Sure, which ever is convenient for you. Basically i didnt realized that you are part of that mailing list and thought to file a bug with haproxy and take this further. Sorry for the noise around here and over email.
---
### Author:  [@lukastribus](https://discourse.haproxy.org/u/lukastribus)
#### Post date: [December 26, 2018, 11:32am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/13 "2018-12-26T11:32:23Z")
No problem, the discussion needed to move there anyway, for further discussions with the developers, etc.
---
### Author:  [@lukastribus](https://discourse.haproxy.org/u/lukastribus)
#### Post date: [July 8, 2019, 8:51am UTC](https://discourse.haproxy.org/t/send-proxy-not-modifying-some-traffic-with-proxy-ip-port-details/3336/14 "2019-07-08T08:51:45Z")
This topic wasn’t updated, for the record, this is a conntrack issue that can be worked around by disabling `nf_conntrack_tcp_loose`:
[https://www.spinics.net/lists/netdev/msg546371.html](https://www.spinics.net/lists/netdev/msg546371.html)