HAproxy as reverse-proxy with 2 interfaces

Help!
1

Hello,

I use HAProxy 1.8.8 as reverse-proxy in product with Linux 4.14 from scratch.
HAProxy brings the SSL dimension to make HTTPS, FTPS (Implicit) and SMTPS (Implicit) based on HTTP, FTP and SMTP.
On this product there are 2 IP interfaces:

  • Lan eth0 IP : 192.68.1.150 GW : 192.68.1.21
  • Wan ppp0 (modem 3G/4G) IP : 10.160.241.16 GW: 192.200.1.21
    The system integrates a web server, and the ability to send mails.

As a server, everything works perfectly, in Lan and Wan.
As a customer, everything works perfectly in Lan.
The problem occurs in client mode by the Wan, for example for the SMTP.

here are my routing rules:

iproute list table modem3G

default via 192.200.1.21 dev ppp0
10.160.241.16 dev ppp0 src 10.160.241.16

iproute list table main

default via 192.68.1.20 dev eth0
192.68.1.0/24 dev eth0 src 192.68.1.150
192.200.1.21 dev ppp0 src 10.160.241.16

Here is the configuration passed to HAProxy
frontend frt3078899856
mode tcp
bind 127.0.0.1:2525
default_backend bck3078899856

backend bck3078899856
mode tcp
server srv3078899856 smtp.orange.fr:465 ssl verify none

The SMTP connection uses the address 127.0.0.1:2525 as the recipient. (HAProxy)
Without HAProxy, the connection works.

After many hours of research and testing, here is my question:
- How can HAProxy know to use the Wan gateway ?

Last point: If I add the following command, it works.
#route add default ppp0
But I don’t think that is the solution, because it will hide the Lan gateway.

Thank you for your help

2

I find the description of the problem quite convoluted and hard to understand, so I’ll try to ask a few helper questions:

  • you are in the situation where you have in fact two default (and active) routes? do you want to employ source-based routing?
  • your frontend is listening on localhost, therefore without additional iptables rules, that frontend shouldn’t be accessible from either WAN or LAN; could you also provide the entire iptables rules, especially the NAT and if needed mangle tables?
  • what do you mean client mode by the wan? could you provide a test case with socat (or nc)? what error does it encounter? (perhaps a tcpdump of that stream?)
3

Hello,
Thank you for your reply. Sorry for the confusion because my english is not good.

Yes, there is two default and active routes: LAN and WAN.

do you want to employ source-based routing?

I don’t understand the question

your frontend is listening on localhost, therefore without additional iptables rules, that frontend shouldn’t be accessible from either WAN or LAN;

It’s normal because he’s listening to inside the system

sorry, client is the French word for customer.
the customer mode is in opposition to the server mode.

a schema is better than a long speech

Plan%20HAProxy
Plan%20HAProxy747×435 6.04 KB

the first problem is the resolved DNS but if I use an IP address the communication does not work.
It would be necessary to be able to specify the interface used by the backend.

LAN: IP 192.68.1.150
LAN: GW 192.68.1.20
WAN: IP 10.65.3.93
GW 192.200.1.21

Entire iptables rules:
#iproute list table main
default via 192.68.1.20 dev eth0
10.8.8.0/28 dev usb0 src 10.8.8.8
192.68.1.0/24 dev eth0 src 192.68.1.150
192.200.1.21 dev ppp0 src 10.65.3.93
#iproute list table modem3G
default via 192.200.1.21 dev ppp0
10.65.3.93 dev ppp0 src 10.65.3.93
#iprule list
0: from all lookup local
32764: from all to 10.65.3.93 lookup modem3G
32765: from 10.65.3.93 lookup modem3G
32766: from all lookup main
32767: from all lookup default

Thank you for your help.

4

You seem to already be using source based routing (i.e. policy routing) because you have configured the various ip route entries and ip rules. At a first glance they seem OK, meaning that if someone connects to your HAProxy via WAN or via LAN it should work properly, as long as from the WAN they always use the WAN IP, namely 10.65.3.93, and respectively when connecting from the LAN to always use the LAN IP, i.e. 192.168.1.150.

If on the other hand a client from the LAN uses your WAN IP, then this won’t work. (As the reply packets take an asymmetric route, which most likely won’t work due to NAT, etc.)

BTW, ip route and ip rule, are not the same as iptables. (iptables is the firewall which is usually also used for NAT; also iptables could be used to “jump” one connection to a different table, in a similar fashion with ip rule.)

Now, if you want to be able to connect from within the LAN with the WAN IP of the server, you’ll have to use iptables mangle chain to assign that connection to a particular ip route table. (The setup can become quite complex and brittle, therefore I suggest on using the proper IP on each side.)

If something is listening on 127.0.0.1 (or localhost) it can be connected to only by services running on the same server. Therefore all this source based routing doesn’t actually apply.

Most likely you want to access this from outside the server. Therefore without some iptables nat rules it won’t be accessible.
[/quote]

From this schema it seems that you want to use HAProxy as an “outgoing” SMTP “proxy”. (I.e. your backend seem to point towards the internet, meanwhile the frontend points towards your LAN.) If this is the case, then definitively the frontend shouldn’t be listening on the localhost.
[/quote]

Please provide the full configuration of the not working HAProxy, and also an socat based example of what you are trying to achieve, and what error it gives. (Also the socat should be run from the actual client machine, i.e. from a different LAN client if this is the case.)

5

Hello,

actually it works very well.

as you say, I don’t want the 2 networks to communicate.
They must remain separated.

What is important, is that the whole is in the same embedded machine.

you are right. My software connects to the frontend and the backend goes to the internet but can not.
If I add the rule “route add default ppp0” it works. But then it is the LAN customer connection that are no longer working.

here is the HAProxy configuration:
global
log /dev/log local0 debug
log 127.0.0.1 local0
tune.ssl.default-dh-param 1030
daemon
maxconn 50

defaults
log global
option tcplog
timeout connect 120s
timeout client 120s
timeout server 120s

frontend frt3070081168
mode tcp
bind 127.0.0.1:2525
default_backend bck3070081168

backend bck3070081168
mode tcp
server srv3070081168 smtp.orange.fr:465 ssl verify none

I don’t know socat. I will see to install it on my embedded machine and send you log.

6

Hello ciprian.craciun,

I installed socat on the embedded machine and it works.
It is a very complete and complex tool.
With the diagram above, can you help me to the socat commands important for you.

Thank you for your help

7

Hello,
as my problem is still not resolved, I come back to the original question:
- How can HAProxy know to use the Wan gateway (ex: ppp0) ?

one solution would be to be able to specify the working device in the backend

example:
frontend frt3070081168
mode tcp
bind 127.0.0.1:2525
default_backend bck3070081168

backend bck3070081168
mode tcp
server srv3070081168 smtp.orange.fr:465 ssl verify none on ppp0

Thank you for your help

8

You need to understand that Haproxy does not make L3 routing decisions. Your kernel does.

If you want haproxy #1 to connect to the Internet, you need to have a default-route pointing to that internet connection. And you cannnot have another default router pointing somewhere else, not in the same routing table.

Create multiple routing tables (with namespaces), configure dynamic routing decisions with source or mark based routing, or the simplest of them all: actually only configured routes towards your LAN that you need, so that it doesn’t conflict with the default-route.

9

Thank you for your reply, Lukas.
Ok, Haproxy does not make L3 routing decisions. But, I use HAProxy in reverse-proxy.
I use 2 routing tables with iproute2. (see above Entire iptables rules )

In the same case, if I do not use HAProxy it works fine. So I think my routing rules are good and that HAProxy does not use the right device.

With HAProxy, if I add the rule “ route add default ppp0 ” it works, but is not the solution.

Thank you for your help

10

You configured 2 tables, but nowhere in the configuration did you specify how haproxy is supposed to behave.

Like I said above, either use namespaces, source or mark based routing or use a single table with a single default route and properly configured reverse routes.

11

I understand that I did not do everything correctly, and that it does not concern HAProxy, but can you give me an example because I do not know how to do it.

thank you

12

You can start by picking one of the proposals I made.

13

After research to understand the different solutions, I think that the solutions “namespace” and “source or mark based routing” do not agree with my problem.

It remains “single table with a single default route and properly configured reverse routes” for which I do not find much information. Especially for the party “configured reverse routes”

14

I don’t know what you are saying.

You use a single routing table, with a default-route pointing to the ppp0 interface. To reach the other networks in your LAN, you make specific routes for it.