Send-proxy not modifying some traffic with proxy ip/port details

roobesh · December 19, 2018, 5:56am

We are running below version of code

haproxy -vv

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.31 2012-07-06
Running on PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (libpcre build without JIT?)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

We have send-proxy configured and running fine all these days. Noticed when we started logging, some traffic which passed from haproxy to backend server noticed that send-proxy is not actually modifying the proxy ip/port, instead retaining same client ip/port like below,

PROXY TCP4 192.220.26.39 192.220.26.39 45066 45066
PROXY TCP4 192.220.26.39 192.220.26.39 45075 45075

We tried to sniff the packet the moment it leaves haproxy and above is what we see. This is only happening only sometimes for some traffic and rest of the traffic properly looks good.

Could you please see this is a known bug in the version we running or something known issue ?

Thanks for your help.

-Roobesh G M

lukastribus · December 19, 2018, 8:56am

There is no known issue explaining what you see.

Can you share the configuration please?

roobesh · December 19, 2018, 9:35am

Thanks for your response. Here you go,

global
# Docker image doesn’t run rsyslogd
# so send logs to docker host
# log-send-hostname @LOG_HOSTNAME@
log-tag haproxy-tst
chroot /var/lib/haproxy
maxconn 500000
user haproxy
group haproxy
log /dev/log len 8192 local1
log /dev/log len 8192 local2 err
stats timeout 2m
tune.maxrewrite 1024
# we have seen requests and responses including large cookies/MCP headers that failed
tune.bufsize 18432
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
lua-load /etc/haproxy/bdcast.lua
nbproc 8
stats socket ipv4@127.0.0.1:9035 level admin process 1
stats socket ipv4@172.17.0.2:9035 level admin process 1
stats socket ipv4@10.251.1.1:9035 level admin process 1
stats socket ipv4@172.19.0.4:9035 level admin process 1
stats socket ipv4@127.0.0.1:9036 level admin process 2
stats socket ipv4@172.17.0.2:9036 level admin process 2
stats socket ipv4@10.251.1.1:9036 level admin process 2
stats socket ipv4@172.19.0.4:9036 level admin process 2
stats socket ipv4@127.0.0.1:9037 level admin process 3
stats socket ipv4@172.17.0.2:9037 level admin process 3
stats socket ipv4@10.251.1.1:9037 level admin process 3
stats socket ipv4@172.19.0.4:9037 level admin process 3
stats socket ipv4@127.0.0.1:9038 level admin process 4
stats socket ipv4@172.17.0.2:9038 level admin process 4
stats socket ipv4@10.251.1.1:9038 level admin process 4
stats socket ipv4@172.19.0.4:9038 level admin process 4
stats socket ipv4@127.0.0.1:9039 level admin process 5
stats socket ipv4@172.17.0.2:9039 level admin process 5
stats socket ipv4@10.251.1.1:9039 level admin process 5
stats socket ipv4@172.19.0.4:9039 level admin process 5
stats socket ipv4@127.0.0.1:9040 level admin process 6
stats socket ipv4@172.17.0.2:9040 level admin process 6
stats socket ipv4@10.251.1.1:9040 level admin process 6
stats socket ipv4@172.19.0.4:9040 level admin process 6
stats socket ipv4@127.0.0.1:9041 level admin process 7
stats socket ipv4@172.17.0.2:9041 level admin process 7
stats socket ipv4@10.251.1.1:9041 level admin process 7
stats socket ipv4@172.19.0.4:9041 level admin process 7
stats socket ipv4@127.0.0.1:9042 level admin process 8
stats socket ipv4@172.17.0.2:9042 level admin process 8
stats socket ipv4@10.251.1.1:9042 level admin process 8
stats socket ipv4@172.19.0.4:9042 level admin process 8
stats socket /var/run/haproxy-1.sock mode 600 level admin process 1
stats socket /var/run/haproxy-2.sock mode 600 level admin process 2
stats socket /var/run/haproxy-3.sock mode 600 level admin process 3
stats socket /var/run/haproxy-4.sock mode 600 level admin process 4
stats socket /var/run/haproxy-5.sock mode 600 level admin process 5
stats socket /var/run/haproxy-6.sock mode 600 level admin process 6
stats socket /var/run/haproxy-7.sock mode 600 level admin process 7
stats socket /var/run/haproxy-8.sock mode 600 level admin process 8

listen stats
bind 127.0.0.1:1936 process 1
bind 172.17.0.2:1936 process 1
bind 10.10.1.1:1936 process 1
bind 172.19.0.4:1936 process 1
bind 127.0.0.1:1937 process 2
bind 172.17.0.2:1937 process 2
bind 10.10.1.1:1937 process 2
bind 172.19.0.4:1937 process 2
bind 127.0.0.1:1938 process 3
bind 172.17.0.2:1938 process 3
bind 10.10.1.1:1938 process 3
bind 172.19.0.4:1938 process 3
bind 127.0.0.1:1939 process 4
bind 172.17.0.2:1939 process 4
bind 10.10.1.1:1939 process 4
bind 172.19.0.4:1939 process 4
bind 127.0.0.1:1940 process 5
bind 172.17.0.2:1940 process 5
bind 10.10.1.1:1940 process 5
bind 172.19.0.4:1940 process 5
bind 127.0.0.1:1941 process 6
bind 172.17.0.2:1941 process 6
bind 10.10.1.1:1941 process 6
bind 172.19.0.4:1941 process 6
bind 127.0.0.1:1942 process 7
bind 172.17.0.2:1942 process 7
bind 10.10.1.1:1942 process 7
bind 172.19.0.4:1942 process 7
bind 127.0.0.1:1943 process 8
bind 172.17.0.2:1943 process 8
bind 10.10.1.1:1943 process 8
bind 172.19.0.4:1943 process 8
mode http
stats enable
stats uri /
timeout connect 10s
timeout client 1m
timeout server 1m
acl service_down nbsrv(lb-backend-tst) lt 1
monitor-uri /alive
monitor fail if service_down

defaults
log global
# log as JSON, we have an ELK stack
log-format ‘{“type”:“haproxy-tst”,“client_ip”:"%ci",“client_port”:%cp,“timestamp”:%Ts,“accept_date”:"%tr",“frontend_name”:"%ft",“backend_name”:"%b",“server_name”:"%s",“TR”:%TR,“Tw”:%Tw,“Tc”:%Tc,“Tr”:%Tr,“Ta”:%Ta,“http_status_code”:%ST,“bytes_read”:%B,“bytes_uploaded”:%U,“termination_state”:"%tsc",“actconn”:%ac,“feconn”:%fc,“beconn”:%bc,“srvconn”:%sc,“retries”:%rc,“srv_queue”:%sq,“backend_queue”:%bq,“http_method”:"%HM",“http_uri”:"%{+E}HU",“http_version”:"%HV"}’
#option httplog
option log-separate-errors
maxconn 100000
http-check disable-on-404
monitor-uri /alive
# HTTP mode is enabled per default for backends; also accept invalid reponses
mode http
option accept-invalid-http-response
option accept-invalid-http-request
timeout connect 25s
timeout client 3h
timeout server 10m
timeout http-request 25s
# Use all backup servers in case all operational servers are down
option allbackups
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

backend lb-backend-tst
# restrict overall connections to the sum of maxconns of all active backend servers
fullconn 250000
acl blocked_host src -f /etc/haproxy/lists/ip_blocklist.lst
http-request deny if blocked_host
# block unresolvable known hosts
acl unresolvable_host req.hdr(host) -f /etc/haproxy/lists/unresolvable_hosts.lst
http-request deny if unresolvable_host
# delete untrusted incoming XFF header
reqidel ^X-Forwarded-For:.*
#Call lua script to broadcast
acl is_tst_rt_url hdr(Host) -i xx.yy.com
acl is_tst_rt_url urlp(rtparams) -m found
http-request lua.broadcast_txn if is_tst_rt_url is_secure_rt_url
# stickyness configuration for load balancing and session validity (Host header, source IP)
http-request set-header X-LB %[req.hdr(Host),lower]%[src]%[src_port]
balance hdr(X-LB)
# do compression on outgoing traffic
compression algo gzip
compression type text/css text/html text/javascript application/javascript text/plain text/xml application/json
hash-type consistent
# configure thresholds for filter nodes
default-server inter 3s fastinter 1s downinter 500ms fall 3 rise 2 slowstart 10s
server node00 10.10.2.3:8080 check maxconn 25000 send-proxy
server node01 10.10.2.4:8080 check maxconn 25000 send-proxy
server node02 10.10.2.5:8080 check maxconn 25000 send-proxy
server node03 10.10.2.6:8080 check maxconn 25000 send-proxy
server node04 10.10.2.7:8080 check maxconn 25000 send-proxy
server node05 10.10.2.8:8080 check maxconn 25000 send-proxy
server node06 10.10.2.9:8080 check maxconn 25000 send-proxy
server node07 10.10.2.10:8080 check maxconn 25000 send-proxy

frontend region1
bind 192.168.123.110:80
monitor fail if { nbsrv(lb-backend-tst) lt 1 }
stick-table type string size 1k expire 1m store http_req_rate(1s)
acl is_tst_it_url hdr(Host) -i xx.yy.com
http-request track-sc0 hdr(Host) if is_tst_it_url
http-request deny if { sc0_http_req_rate gt 1 } is_tst_it_url
tcp-request connection accept if { src -f /etc/haproxy/lists/whitelist.lst }
default_backend lb-backend-tst

Points:

Issue is not always happening , occassionally we facing this issue from some IPs.
example: x.x.x.x:30000 client IP reaching y.y.y.y:80, using send-proxy always that is been seen by backend as’PROXY TCP4 x.x.x.x y.y.y.y 30000 80’ in the format of ‘PROXY INET:PROTO4 clientip proxyip client-sourceport proxy-port’.
But some requests are just showing as ‘PROXY TCP4 x.x.x.x x.x.x.x 30000 30000’

No Idea why ?

-Roobesh G M

roobesh · December 20, 2018, 6:44am

Any thoughts how to troubleshoot why occasionally send-proxy is not actually sending right headers ?

PROXY TCP4

instead wrongly some times showing as
PROXY TCP4

mzielin · December 20, 2018, 2:23pm

Apparently the situation is the same as in https://stackoverflow.com/questions/11417187/getsockopt-so-original-dst-occasionally-returns-client-address

commenting out getsockopt(… SO_ORIGINAL_DST) and only use getsockname() fixed the issue so far.
Under high traffic the PROXY line contained approx once out of 10000 times the source IP as destination instead the destination IP.

lukastribus · December 20, 2018, 9:25pm

@mzielin Thanks, that’s interesting, so it seams that SO_ORIGINAL_DST may have a reliability issue. This would be a bug in the kernel, rather than haproxy.

What kernel are you guys running exactly?

@roobesh can you try recompiling haproxy with USE_TPROXY= (nothing after the equal sign) ? This disables the mentioned code path (along with TPROXY support).

roobesh · December 21, 2018, 4:02am

@lukastribus We are running Ubuntu 18.04 with kernel 4.15.0-39-generic
You mean disabling all the below options and recompile or just adjust USE_TPROXY=1 as USE_TPROXY=

./Makefile:# USE_TPROXY : enable transparent proxy. Automatic.
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile: USE_TPROXY = implicit
./Makefile:USE_TPROXY = 1

Also looks like SO_ORIGINAL_DST reliability issue is retained from older version even to the latest what we running looks like (going through other forums).

@Martin Thanks much for narrow down on the cause.

-RGM

lukastribus · December 21, 2018, 7:50am

@roobesh No, don’t modify any files, just add USE_TPROXY= to your make command, for example:

make clean; make TARGET=linux2628 CPU=native USE_GETADDRINFO=1 \
USE_PCRE=1 USE_OPENSSL=1 USE_ZLIB=1 USE_TPROXY=

roobesh · December 26, 2018, 5:42am

@lukastribus Thanks for your response. We will compile a fresh binary with the options provided by you above.

Can you clarify, getsockopt(… SO_ORIGINAL_DST) is only used by send-proxy feature or any other feature in haproxy is using this often ? Basically trying to understand what are the cases where destination-ip needed to be looked into by haproxy other than send-proxy feature. As an alternative what @Martin tried by fixing using this getsockname() solved the issue, but would like understand can there be any other broken area due to this fix ?

roobesh · December 26, 2018, 6:11am

@lukastribus Can we create this as a bug with haproxy and update the source-code to not use getsockopt(… SO_ORIGINAL_DST) and find an alternative for this ? getsockname() fixed the issue.

lukastribus · December 26, 2018, 9:20am

Let’s continue the discussion in the mailing list, where you just posted:

https://www.mail-archive.com/haproxy@formilux.org/msg32198.html

Otherwise we fragment the information in different forums.

roobesh · December 26, 2018, 10:58am

Sure, which ever is convenient for you. Basically i didnt realized that you are part of that mailing list and thought to file a bug with haproxy and take this further. Sorry for the noise around here and over email.

lukastribus · December 26, 2018, 11:32am

No problem, the discussion needed to move there anyway, for further discussions with the developers, etc.

lukastribus · July 8, 2019, 8:51am

This topic wasn’t updated, for the record, this is a conntrack issue that can be worked around by disabling nf_conntrack_tcp_loose:

https://www.spinics.net/lists/netdev/msg546371.html

Topic		Replies	Views
Cross-Origin Request Blocked: Help!	0	688	May 22, 2020
Haproxy 1.8 vs 2.7.3 Help!	11	871	February 28, 2023
Trouble with getting TLS 1.3 working on Ubuntu 16.04 Help!	2	1756	February 24, 2020
Haproxy 1.7.x 1.8.x 1.9.x tcp:receive crash Help!	1	573	August 2, 2018
Haproxy 1.8.9 CPU usage increased significantly Help!	7	1577	June 5, 2018

Send-proxy not modifying some traffic with proxy ip/port details

haproxy -vv

Related topics