However backends can not reply back correct and send the data to client.
With tcpdump on backend i see only incoming packets from client but it does not attempt to send back the data.
I do not see any info in the article how to configure the backends, it only explains the haproxy config.
Somewhere is mentioned i must configure backends default gateway to be the HAproxy box but how to do this with example?
I found a solution and would like to share it with people looking in similar setup.
On backend servers i created second routing table so it does not break connection to the internet on default gateway.
Routing tables could be added in file /etc/iproute2/rt_tables by appending a line like 1 rt2
Next in the rt2 table i added static route so all traffic coming from my backend server with IP 192.168.1.11 goes through network interface ens19 ip route add 192.168.1.0/24 dev ens19 src 192.168.1.11 table rt2
Then added the IP of the loadbalancer 192.168.1.20 as default gateway only for table rt2 ip route add default via 192.168.1.20 dev ens19 table rt2
Finally i added 2 rules for using rt2 - when traffic originates from 192.168.1.11 (the backend server IP) and when traffic comes to 192.168.1.11 ip rule add from 192.168.1.11 table rt2 ip rule add to 192.168.1.11 table rt2
This works perfect and there are many guides how to make the changes permanent.
To make it clear, without modifying the routes on the backend servers things are not working when the clients and the servers are in the same subnet because the client starts a connection to the IP of the proxy but gets reply from the IP of the backend server thus the client simply droppes those packets as invalid. That’s why it is important to send the reply traffic back via the proxy server which in this case is a NAT gateway for the backend servers to the local subnet.
I do not think the above article could solve my case.
Client and backends are not on same subnet.
Clients are coming from public internet, not from private 192.168.x.x
Also the article in your comment does not solve connection to the internet from backend.
I understand the SNAT modified packets must be returned back to haproxy but there is no example or i was unable to find how this common task could be resolved. I asked for sample configs in the opening post and no one replied.
After some tests and reading about routing i found the solution and posted it in case other people need it.
I see, that was kinda a missing piece of information.
Assuming 192.168.1.20 is your LB, you change the default GW on the backends like:
ip route set default via 192.168.1.20 dev ens19
which will achieve the same as you did with a separate routing table. Except that in this case all traffic from the backends will always go via the LB including stuff like downloading packages when you do update/upgrade etc. which might not be optimal. At the end of the day your solution is better in that aspect.