HAProxy behind NAT

Help!
1

Hello everyone, i am new to HAProxy and struggling for more than 3 days to make it works but unfortunately nothing achieved.

So i short words trying to achieve this kind of logic:

Dedicated Server (Proxmox VE+ 1 Public IP) → (NAT) OPNsense + HAProxy → Other VMs connected to OPNsense LAN interface.

The configuration of Proxmox Server is as the following:

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback

auto enp0s31f6
iface enp0s31f6 inet static
        address 94.130.x.x/26
        gateway 94.130..x.x

auto vmbr0
iface vmbr0 inet static
        address 10.10.10.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

        post-up echo 1 > /proc/sys/net/ipv4/ip_forward
        post-up   iptables -t nat -A POSTROUTING -s '10.10.10.0/24' -o enp0s31f6 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s '10.10.10.0/24' -o enp0s31f6 -j MASQUERADE
        post-up   iptables -t raw -I PREROUTING -i fwbr+ -j CT --zone 1
        post-down iptables -t raw -D PREROUTING -i fwbr+ -j CT --zone 1

auto vmbr1
iface vmbr1 inet static
        address 172.16.0.1/24
        bridge-ports none
        bridge-stp off
        bridge-fd 0

Ok, so created new VM(OPNsense), install and configure it as following:

WAN → vtnet0 (bridge to vmbr0 at Proxmox Server)
LAN → vtnet1 (brigde to vmbr1 at Proxmox Server)

WAN configured with 10.10.10.10/24
LAN configured with 172.16.0.1/24 DHCP(yes) Range: 172.16.0.2-172.16.0.254

Now the servers part:

  • VM 1

VM(Ubuntu Server) with OpenLiteSpeed Web Server running (example.com) and Postfix/Dovecot for email purposes and connected to vmbr1 (LAN of OPNsense connected to Proxmox vtnet1)
The Ubuntu server get the IP successfully via OPNsense as following → IP 172.16.0.2 , Gateway 172.16.0.1.

  • VM2

VM(Ubuntu Server) with OpenLiteSpeed Web Server running (anotherexample.com) and Postfix/Dovecot for email purposes and connected to vmbr1 (LAN of OPNsense connected to Proxmox vtnet1)
The Ubuntu server get the IP successfully via OPNsense as following → IP 172.16.0.3 , Gateway 172.16.0.1.

Both of the VMs connected through OPNsense LAN and able to communicate with public internet successfuly.

OK now the hard part :slight_smile:

CloudFlare DNS for example.com:

A Record example.com pointing to Public IP of Proxmox Server -> 94.130.x.x

Created some iptables rules to communicate from Public IP to local OPNsense and HAProxy:

For OPNsense:
iptables -t nat -A PREROUTING -p tcp --dport 10443 -j DNAT --to-destination 10.10.10.10:10443

HAProxy configuration:

#
# Automatically generated configuration.
# Do not edit this file manually.
#

global
    uid                         80
    gid                         80
    chroot                      /var/haproxy
    daemon
    stats                       socket /var/run/haproxy.socket group proxy mode 775 level admin
    nbthread                    1
    hard-stop-after             60s
    no strict-limits
    tune.ssl.default-dh-param   2048
    spread-checks               2
    tune.bufsize                16384
    tune.lua.maxmem             0
    log                         /var/run/log local0 info
    lua-prepend-path            /tmp/haproxy/lua/?.lua

defaults
    log     global
    option redispatch -1
    timeout client 30s
    timeout connect 30s
    timeout server 30s
    retries 3
    default-server init-addr last,libc

# autogenerated entries for ACLs


# autogenerated entries for config in backends/frontends

# autogenerated entries for stats

# Frontend: Public_Facing_Pool ()
frontend Public_Facing_Pool
    bind *:443 name *:443  proto h2
    bind *:80 name *:80  proto h2
    mode http
    option http-keep-alive
    maxconn 500

    # logging options
    # ACL: Web-Server
    acl acl_65baf2832edf80.37086579 hdr_beg(host) -i example.com

    # ACL: Web-Server1
    acl acl_66baf2832edf80.37086579 hdr_beg(host) -i anotherexample.com

    # ACTION: Web-Server
    use_backend Web-Server if acl_65baf2832edf80.37086579

    # ACTION: Web-Server1
    use_backend Web-Server1 if acl_66baf2832edf80.37086579

# Backend: Web-Server ()
backend Web-Server
    # health checking is DISABLED
    mode http
    balance roundrobin

    http-reuse safe
    server Web-Server 172.16.0.2:443

# Backend: Web-Server1 ()
backend Web-Server
    # health checking is DISABLED
    mode http
    balance roundrobin

    http-reuse safe
    server Web-Server 172.16.0.3:443

# Backend: acme_challenge_backend (Added by ACME Client plugin)
backend acme_challenge_backend
    # health checking is DISABLED
    mode http
    balance source
    # stickiness
    stick-table type ip size 50k expire 30m  
    stick on src
    http-reuse safe
    server acme_challenge_host 127.0.0.1:43580 



# statistics are DISABLED

Trying to open in browser example.com it fails to open.

Please anybody can help to achieve that since it is very important for me and I don’t know anymore what to do, coming around to this more than 3 days for hours and hours. I don’t know if something wrong with it or lack of my knowledge.

2

It’s a very complex…
But I think I see a issue right away.
You are routing traffic in iptables with 10443, source and destination.
And using 443 in your HaProxy configs.

But I haven’t understand how your public traffic will pass through Proxmox and reach OPNSense having two VMs using the same ports / public IP…

Greetings, Pedro

3

The rule for port 10443 is to pass wan traffic to OPNsense management web gui.
I have another rule with port 443 which pass the traffic on the same port on HAproxy.

Via HAproxy