IP routing with haproxy tcp for ftp

ebourgui · September 8, 2016, 12:27pm

Hi guys,

My configuration:

server1
ha proxy 1.5.14 - centos 7
public ip eth0 : 1.2.3.4
public ip eth0:0 : 5.6.7.8 (ovh ip failover which can point to another haproxy server when failover occured)
||
server3
vsftp server 3.0.2 - centos7
public ip eth0: 9.10.11.12

Only public addresses. server1 and server3 have backups in a pacemaker cluster (active / passive) : server2 (haproxy backup) and server4 (ftp backup)

When I connect in passive mode to server3 : no problem.
When I connect in passive mode to server1 (on 5.6.7.8):

ftp *****.com
Connected to *****.com (5.6.7.8).
220 Welcome to FTP service.
Name (******.com:root): toto
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
227 Entering Passive Mode (9,10,11,12,198,124).
ftp: connect: Connexion terminée par expiration du délai d'attente
=> in english : timeout

I take a look at /var/log/messages on the client and see
Sep 8 12:26:47 ***** kernel: [11686946.541337] Firewall: *TCP_OUT Blocked* IN= OUT=eth2 SRC=******* DST=9.10.11.12 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=45793 DF PROTO=TCP SPT=52258 DPT=50812 WINDOW=29200 RES=0x00 SYN URGP=0 UID=0 GID=0

The client try to connect directly to server3 and not to server1. I think that the packets are blocked because packets should be iptables RELATED to server1 and not to server3 to pass the firewall.
I don’t want to change iptables output rules because it concerns a lot of clients.
Is there a way to respond to server1 and not to server3 (or another way to work properly)?

server1 : haproxy.cfg
…
listen ftplb 5.6.7.8:21, 5.6.7.8:50000-50999
mode tcp
option tcplog
server server3 9.10.11.12 check port 21
server server4 13.14.15.16 check port 21

server1 et server3 : iptables
-A INPUT -p tcp -i eth0 -m multiport --dport 21,50000:50999 -m conntrack --ctstate NEW,ESTABLISHED,RELATED -j ACCEPT

server3 vsftpd.conf

anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=022
dirmessage_enable=YES
xferlog_enable=YES
xferlog_std_format=YES
chroot_local_user=YES
listen=YES
pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES
allow_writeable_chroot=YES
ftpd_banner=Welcome to FTP service.
pasv_enable=YES
pasv_promiscuous=YES
pasv_min_port=50000
pasv_max_port=50999
dual_log_enable=YES
log_ftp_protocol=YES
port_enable=YES
port_promiscuous=YES
connect_from_port_20=NO
seccomp_sandbox=NO

Any idea?

Regards,
Eric

lukastribus · September 9, 2016, 7:24am

The proper solution is to configure a dedicated range of ports on server3 and open that range on iptables.

If thats not what you want, for whatever reason, then you need to look at a proxy server that intercepts FTP message and redirects data traffic as well. Haproxy does not do that (as it is not a FTP proxy, but a TCP proxy in this case).

Quick sidenote: iptables would also fail to open passive ports when you would use FTP with SSL. Once you enable SSL, iptables FTP helper are no longer able to intercept the passive ports, meaning it would be closed. Relying on those FTP helpers in iptables is therefor almost always a bad idea.

ebourgui · September 9, 2016, 8:49am

Thank you for your response.

Already done.

Do you know ftp proxies? ftp-proxy seems to be outdated…

I won’t use FTP with SSL. I use SFTP with haproxy and It works well.

Eric

lukastribus · September 9, 2016, 9:40am

No, I don’t. You could also forward the passive port range from haproxy to your backend FTP server. That would probably be the easiest way.

You would force the FTP server to announce the IP of haproxy in the control channel, and forward the pasv ports as well through haproxy.

This is what’s explained here and is actually the simplest approach:

ebourgui · September 9, 2016, 12:23pm

I’ve ever seen this link and it helped me. But if I add in /etc/vsftp/vsftp.conf pasv_address=5.6.7.8:

client firewall doesn’t block output packets : OK
but I get
ftp failover fqdn
Connected to failover fqdn (5.6.7.8).
220 Welcome to FTP service.
Name (failover fqdn:root): toto
331 Please specify the password.
Password:
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
227 Entering Passive Mode (5,6,7,8,196,94).
ftp: connect: Connexion refusée
=> connection refused

lukastribus · September 9, 2016, 1:17pm

Did you configure haproxy correctly? How does the configuration look like?

ebourgui · September 9, 2016, 1:39pm

global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     1000
    #user        haproxy
    #group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    tune.ssl.default-dh-param 2048

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option                  http-server-close
    option                  forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s

    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s

frontend stats
        bind 5.6.7.8:8080
        stats enable
        stats uri /******
        stats realm Haproxy\ Statistics
        stats auth ****:******

(....)

listen ftplb2 5.6.7.8:21, 5.6.7.8:50000-50999
    mode tcp
    option tcplog
    server server3 9.10.11.12 check port 21 source interface eth0:0
    server server4 13.14.15.16 check port 21 source interface eth0:0

where eth0:0=5.6.7.8

lukastribus · September 9, 2016, 2:21pm

Does haproxy actually listen on the port range?

You would be better up configuring a proper frontend/backend section, instead of using this obsolete approach with the listen section. And don’t use “source interface”, specify the source IP.

frontend ftplb2
 bind 5.6.7.8:21
 bind 5.6.7.8:50000-50999
 mode tcp
 option tcplog
 default_backend ftpservers

backend ftpservers
 mode tcp
 server server3 9.10.11.12 check port 21 source 5.6.7.8
 server server4 13.14.15.16 check port 21 source 5.6.7.8

ebourgui · September 9, 2016, 2:56pm

Yes!!! It works in passive mode. THANK YOU!!!

Just a problem in active mode:
200 PORT command successful. Consider using PASV.
421 Service not available, remote server has closed connection

It’s week end time for me. I will retry on monday.
Thank you again.

ebourgui · September 13, 2016, 7:42am

From client (which accept all output traffic in firewall):

I can connect in active mode directly to ftp server (server3)
I can’t connect in active mode to ftp server by haproxy

200 PORT command successful. Consider using PASV.
421 Service not available, remote server has closed connection

In this situation no packets are blocked by firewall on server3 and server1.

lukastribus · September 13, 2016, 12:21pm

For active mode to work you will have to NAT your active traffic through the haproxy box, otherwise the client will see the active connection attempt from a different IP (the IP of server3 instead of the IP of the haproxy box) and will close the connection.

ebourgui · September 14, 2016, 7:31am

Thanks for your help.

I add this rule on ftp server
-A POSTROUTING -p tcp -o eth0 -j SNAT --to 5.6.7.8
It doesn’t work. tcpdump says:
08:53:34.562777 IP 5.6.7.8.35191 > client's private ip.50301: Flags [S], seq 113635828, win 29200, options [mss 1460,sackOK,TS val 1309390568 ecr 0,nop,wscale 7], length 0
Without the rule it was
08:54:19.567917 IP ftp server ip.41179 > client's private ip.51138: Flags [S], seq 1040902272, win 29200, options [mss 1460,sackOK,TS val 1309435573 ecr 0,nop,wscale 7], length 0

So the iptables rule is OK : the packet is from 5.6.7.8 (haproxy). But ftp server replies to the client’s private ip (and not the client’s public ip). But why?

lukastribus · September 14, 2016, 2:10pm

Because the client is behind NAT, incorrectly configured and the NAT gateway doesn’t have an “ftp helper”.

You have to use a properly configured FTP client.

edit: I don’t think the iptables configuration suffices though; you would have to tunnel to the haproxy box and NAT it there, as your FTP doesn’t have see traffic desalinated to 5.6.7.8. You will also have to restrict the tunnel+NAT rule to only the active traffic. This goes beyond what we can support here.

karsten · September 12, 2018, 9:40am

When you are using PASV the server will tell the client which IP and port to connect to. - so you’ll have to set up the server to tell the client to connect to your Proxy-server. - look for something like ‘firewall support’
Remember also that you must configure individual portranges for PASV on the servers, and setup the rules for that in haproxy.

So setup one rule for port 21 that does roundrobin (or leastconn) - remember to set ‘timeout tunnel’, as the server will shut down the session if this conn disappears.
Then setup one rule for each of the servers to listen for the PASV portrange eg:
listen ftp
bind *:21
mode tcp
option tcplog
balance leastconn
timeout tunnel 300s
server ftp1 x.y.x.i check port 21
server ftp2 x.y.z.ii check port 21

listen ftp_pasv1
bind *:63535-64534
mode tcp
option tcplog
server ftp1 x.y.z.i check port 21

listen ftp_pasv2
bind *:64535-65535
mode tcp
option tcplog
server ftp2 x.y.z.ii check port 21

Topic		Replies	Views
Ftp transfert in active mode Help!	1	3627	March 24, 2017
Ftp connexion trough haproxy Help!	1	505	April 3, 2019
TCP mode passthrough - Client ip Help!	0	310	March 11, 2024
Haproxy with Tomcat issue Help!	9	4644	February 9, 2018
Crushftp Behind HaProxy Help!	5	954	January 21, 2021

IP routing with haproxy tcp for ftp

Related Topics