HAProxy forwarding HTTPS OK, TCP not OK. Please help!

Brizoo · May 29, 2019, 8:28am

Hi community,

I’m trying to build an HAProxy setup to make available some LAN Servers from external. The majority is HTTP/HTTPS ports to forward but I also have some TCP ports to forward

I have this basic setup in place and working:
HAProxy server is in my DMZ, I have a firewall between WAN <-> DMZ and DMZ <-> LAN. Servers are in LAN.

The goal here is to go further and enhance my configuration as well as fix bugs I have. Currently all HTTPS redirects are working fine, but not TCP. I also have a problem to identify my sources correctly. For example, if I have two different source to redirect on the two different backend but on same port (this is the case with 8443, I need to add a new backend on 8443), I’m not able to make the difference between them and redirect them on the good backend.

Servers and ports that need to be accessed from outside:

ITAM1 TCP: 8027 HTTPS: 8383 8022 8020 8021 8443 8444 8031
ITAM2 HTTPS: 8443
ITSM TCP: 9000 HTTPS: 443
AV TCP: 8013 8014
OTHER HTTPS: 8085 9443

My configuration file, looks like this so far. This is my first use at HAProxy and the conf I have done is really basic. Please be indulgent!

global
    maxconn 4096
    user haproxy
    group haproxy
    daemon
    # Default SSL material locations
    #ca-base /etc/ssl/certs
    # tune & ssl params to force diffie-hellman defaults, disallow most tls/poodle attacks, and restrict binders to secure ciphers
    #tune.ssl.default-dh-param 4096
    #ssl-default-bind-options no-sslv3 no-tls-tickets
    #ssl-default-bind-ciphers EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH:EECDH+AES

defaults
    mode tcp
    log 127.0.0.1 local0 notice
    timeout connect  5000
    timeout client  50000
    timeout server  50000

frontend tcpServers
    bind *:8013
    bind *:8014
    bind *:8027
    bind *:8383
    bind *:8022
    bind *:8020
    bind *:8021
    bind *:8443
    bind *:8444
    bind *:8031
    bind *:443
    bind *:9000
    bind *:8085
    bind *:9443
    #bind *:443 ssl crt /etc/haproxy/certs/ no-sslv3

    #ACL by Port
    acl tcp_8013 dst_port 8013
    acl tcp_8014 dst_port 8014
    acl tcp_8027 dst_port 8027
    acl https_8383 dst_port 8383
    acl http_8022 dst_port 8022
    acl https_8020 dst_port 8020
    acl https_8021 dst_port 8021
    acl https_8443 dst_port 8443
    acl https_8444 dst_port 8444
    acl https_8031 dst_port 8031
    acl https_443 dst_port 443
    acl tcp_9000 dst_port 9000
    acl http_8085 dst_port 8085
    acl https_9443 dst_port 9443

    tcp-request inspect-delay 5s

# Start SSL Passthrough Backend #
backend fortiemstcp8013
    server fortiems X.X.X.X:8013

backend fortiemstcp8014
    server fortiems X.X.X.X:8014

backend itamtcp8027
    server itam X.X.X.X:8027

backend itamhttps8383
    server itam X.X.X.X:8383

backend itamhttp8022
    server itam X.X.X.X:8022

backend itamhttps8020
    server itam X.X.X.X:8020

backend itamhttps8021
    server itam X.X.X.X:8021

backend itamhttps8443
    server itam X.X.X.X:8443

backend itamhttps8444
    server itam X.X.X.X:8444

backend itamhttps8031
    server itam X.X.X.X:8031

backend supportcenterhttps443
    server supportcenter X.X.X.X:443

backend supportcentertcp9000
    server supportcenter X.X.X.X:9000

backend assetshttp8085
    server assets X.X.X.X:8085

backend assetshttps9443
    server assets X.X.X.X:9443

lukastribus · May 29, 2019, 10:10pm

I don’t know what you mean to achieve here. Can you elaborate?

In other words, on port 8443 you both need a) SSL termination and b) TCP mode (no SSL termination).

Is that a correct understanding of the issue?

Brizoo · May 31, 2019, 5:44am

I have some backends that are web pages in HTTPS and for them the redirection from haproxy to the backend is working fine, I can navigate on the web page from outside.

But some other backends are TCP services such as telemetry for Antivirus and the port is 8013. This redirection from Haproxy to this backend is not working for example, and if I telnet myantivirus.domain.com 8013 from outside I have a timeout.

If I telnet one web page using https 443 for example telnet mywebpage.domain.com 443
Responding ok.

So it looks like HAproxy is never redirecting the connections to the backends when dealing with TCP pure connections others than HTTPS.

Also if I have two services on a same port, for example this is the case with 8443, I don’t know how to configure the acl to identify both sources properly and redirect them on the proper backends.

I don’t need no SSL termination, the backends are handling SSL themselves just fine and where already configured so I just prefer to let them like this.
On the TCP part yes all are encrypted services and must handle their SSL themselves as well. So no SSL termination on Haproxy at all is needed.

lukastribus · June 3, 2019, 3:28pm

The word “redirect” means something else.

Redirect means that haproxy will not forward the request to a backend server, and instead create a local, HTTP redirect response with something like 302 Moved temporarily status. The browser will then disconnect from haproxy and connect to the indicated localtion.

It is something that is only possible when you terminate HTTP (and SSL, if this is HTTPS). None of which you are doing here. There is no such thing as a redirect in TCP mode.

Use the word forwarding, if you mean that. This is important, because by using the wrong terminology everybody will get confused here.

There is no action associated with the ACL tcp_8013 in your configuration. Haproxy will either forward it to a default_backend if any (your configuration does not contain such an action), or drop the connection.

Certainly not with the configuration you posted, because port 443 is clearly commented out. Please provide the actual configuration you are using and testing with.

How would you like to Haproxy to know whether the request belong to service A or service B? SNI? Source IP ranges? You will first have to figure that out, before you can actually think of howto configure haproxy.

As far as I can tell, with the configuration provided in the first post, nothing will work. You did not point to a default_backend, and neither did you configure a use_backend directive, so whatever connection comes in, haproxy will not know where to forward it.

Brizoo · June 3, 2019, 4:14pm

Yes I mean, that. Thank you for the explanation. I mean forward. I want to forward all, not redirect.

Here is my actual conf file, as I’ve worked a bit more onto it. I’ve forget the part where I redirect to backends, weirdly it hasn’t been copied…

global
    maxconn 4096
    log /dev/log local0
    user haproxy
    group haproxy
    daemon

defaults
    timeout connect 10s
    timeout client 30s
    timeout server 30s
    log global
    mode tcp

frontend tcpServers
    bind *:8013
    bind *:8014
    bind *:8027
    bind *:8383
    bind *:8022
    bind *:8020
    bind *:8021
    bind *:8443
    bind *:8444
    bind *:8031
    bind *:443
    bind *:9000
    bind *:8085
    bind *:9443

#ACL by Port
    acl tcp_8013 dst_port 8013
    acl tcp_8014 dst_port 8014
    acl tcp_8027 dst_port 8027
    acl https_8383 dst_port 8383
    acl http_8022 dst_port 8022
    acl https_8020 dst_port 8020
    acl https_8021 dst_port 8021
   acl https_8443 dst_port 8443
    acl https_8444 dst_port 8444
    acl https_8031 dst_port 8031
    acl https_443 dst_port 443
    acl tcp_9000 dst_port 9000
    acl http_8085 dst_port 8085
    acl https_9443 dst_port 9443

#ACL by domain name
    acl host_supportcenter hdr(host) -i supportcenter.domainname.com
    acl host_itam hdr_sub(host) -i itam
    acl host_jss hdr_sub(host) -i voljss

    use_backend fortiemstcp8013 if tcp_8013
    use_backend fortiemstcp8014 if tcp_8014
    use_backend itamtcp8027 if tcp_8027
    use_backend itamhttps8383 if https_8383
    use_backend itamhttp8022 if http_8022
    use_backend itamhttps8020 if https_8020
    use_backend itamhttps8021 if https_8021
    #use_backend itamhttps8443 if https_8443
    use_backend voljsshttps8443 if host_jss
    use_backend itamhttps8444 if https_8444
    use_backend itamhttps8031 if https_8031
    use_backend supportcenterhttps443 if https_443
    use_backend supportcentertcp9000 if tcp_9000
    use_backend assetshttp8085 if http_8085
    use_backend assetshttps9443 if https_9443

    tcp-request inspect-delay 5s

#Backends implementation
backend fortiemstcp8013
    server fortiems X.X.X.X:8013

backend fortiemstcp8014
    server fortiems X.X.X.X:8014

backend itamtcp8027
    server itam X.X.X.X:8027

backend itamhttps8383
    server itam X.X.X.X:8383

backend itamhttp8022
    server itam X.X.X.X:8022

backend itamhttps8020
    server itam X.X.X.X:8020

backend itamhttps8021
    server itam X.X.X.X:8021

backend itamhttps8443
    server itam X.X.X.X:8443

backend itamhttps8444
    server itam X.X.X.X:8444

backend itamhttps8031
server itam X.X.X.X:8031

backend supportcenterhttps443
    server supportcenter X.X.X.X:443

backend supportcentertcp9000
    server supportcenter X.X.X.X:9000

backend assetshttp8085
    server assets X.X.X.X:8085

backend assetshttps9443
    server assets X.X.X.X:9443

backend voljsshttps8443
    server voljss X.X.X.X:8443

I have figured the way I would like to achieve this, and this is by the subdomain name. Each service has a different subdomain name. svc1.domain.com, svc2.domain.com, etc. But they can have same port.
So I’ve started to try this out, creating an acl section for targeting the subdomain and as far as my tests goes, I can’t make any acl match…

Sorry for forgetting to copy a portion of the configuration, so your help was impossible. Now I think everyhting is there.
Thank you by advance for your time.

lukastribus · June 3, 2019, 5:48pm

There is no such thing as a hostname in the TCP packet, or in the socket API.

What hdr(host) does is it extracts the value of the HTTP header Host from a HTTP request. None of this applies to your configuration, as you

don’t use HTTP mode
don’t necessarily use HTTP as layer 7 protocol and even more importantly
forward mostly SSL encrypted traffic transparently to the backend, so you cannot possible tell anything about the layer 7 protocol, whether or not it’s HTTP and especially not what the request headers look like.

First of all, don’t put everything in one frontend. Matching destination ports is a good indication that you are doing something wrong, because the same thing could be achieved way easier.

If ports 8027,8383,8022,8020,8021,8443,8444,8031 are dedicated to itam, just dedicated a frontend for it and direct everything to that specific backend. There is no point in complicating your config with dst_port ACL’s, because you can just group that into a frontend.

For example:

frontend itam
 bind :8027,:8383,:8022,:8020,:8021,:8443,:8444,:8031
 default_backend bk_itam

backend bk_itam
 server itam X.X.X.X

By not specifying a port in the backend server, haproxy will use the destination port from the frontend.

Same goes for the other ones with dedicated ports.

As for the port with 2 services on it, again, the socket API has no concept of hostnames and neither does the TCP stack. You need to extract that information from somewhere. If both of those services are SSL/TLS, and if you can guarantee that all clients are SNI enabled, then you can use that to route between one or the other backend. Is that the case for those services?

Brizoo · June 3, 2019, 8:42pm

Thank you very much for these explanations, it help me a lot. I will rewrite my whole config file following your indications tomorrow.

I’ve never heard of “Server Name Indication” before you and I really have no clue to know if my clients are, or not, SNI enabled. Even after a good search on Google, I’ve found nothing about the forticlient or the Desktop central agent and SNI. How could I check this?

Another thing is that I have multiple public IP addresses available to bind onto, so that means I could separate the port with 2 services on it on different public IP address in the frontend by binding the port on a specific IP address, for example:

frontend itam
    bind x.x.x.y:8443, :8027, :8383, etc.

frontend jss
    bind x.x.x.z:8443

Do you think it could be ok?

lukastribus · June 3, 2019, 11:05pm

If you have multiple public IPs at your disposal that will indeed make everything easier for you. Just like you said you can bind to specific IP:port combinations and this will be a clear distinction between one and the other service.

Brizoo · June 4, 2019, 8:14am

Thanks to @lukastribus, I have rewrited my conf file and it is now a lot better than before, and it has multiple frontends.

global
	maxconn 4096
	log /dev/log local0
	log /dev/log local1 notice
	user haproxy
	group haproxy
	daemon

defaults
	timeout connect 10s
	timeout client 30s
	timeout server 30s
	log global
	mode tcp

#-------------------------------#
#          Frontends            #
#_______________________________#

frontend FortiEMS
	bind :8013,:8014
	default_backend bk_fortiems

frontend ITAM
	bind :8027,:8383,:8022,:8020,:8021,:8443,:8444,:8031
	default_backend bk_itam

frontend JamfJSS
	bind :8443
	default_backend bk_jss

frontend ITSM
	bind :443,:9000
	default_backend bk_itsm

frontend Assets
	bind :8085,:9443
	default_backend bk_assets

#-------------------------------#
#          Backends             #
#_______________________________#

backend bk_fortiems
	server fortiems X.X.X.X

backend bk_itam
	server desktopcentral X.X.X.X

backend bk_jss
	server jamfjss X.X.X.X

backend bk_itsm
	server supportcenter X.X.X.X

backend bk_assets
	server assets X.X.X.X

Though, I still struggle to find a way to strongly identify my sources and so forward connections to the proper backends.

I have tried to add public IP address before the port in the bind section like following:

frontend JamfJSS
bind X.Y.Z.A:8443

frontend ITAM
bind X.Y.Z.B:8443

But nothing is working anymore after this change. How can I check the logs to see if the public IP is correctly coming to HaProxy server?

Another thing, now since I have mutliple frontends, I might be able to use HTTP mode for some of them when appropriate, and after an SSL termination on HAProxy, I could use the hdr(dom) value to identify and forward to the good backends.
What do you think?

Another thing, despite the fact FortiEMS backend is alone using ports 8013 and 8014, I have a timeout when I telnet fortiems.domain.com 8013. And my clients are still not connecting to that service.
This is not related to another service using the same port or anything, It looks like HaProxy is just never forwarding the connections.

Thanks

lukastribus · June 4, 2019, 10:45am

The question is only, are the ip addresses X.Y.Z.A and X.Y.Z.B properly configured on this box and reachable from your network?

If you bind to something that doesn’t work anyway, it won’t work of course.

I’m not sure all of those services are actually HTTP, and also, you would need to have all the certificates with private keys configured with haproxy. This becomes way more complicated than the current configuration.

Whether that’s a good idea depends on what those services are, but it sounds to me like they are more complicated than simple HTTP(S) sites.

Enable logging (but use option tcplog instead of http mode) and check out the logs.

Brizoo · June 4, 2019, 2:50pm

The HAProxy server is inside a DMZ network which is behind a firewall. So I don’t have any public ip address on a netwrok card inside HAProxy server. I NAT from outside to inside onto the HAProxy using my firewall.
If I get what you mean, I should configure multiple network cards onto my HAProxy and assign them one public IP Address each and bypass the firewall by exposing the HAProxy to the outside directly, so I would be able to do what I want. Is that correct?

Nope, you are wright, not all are HTTP, some are TCP, some other I just don’t know because this is not specifically told by the manufacturer, I could verify that using wireshark, but that souds complicated. So I would prefer to stick with the first plan to identify using public IP address.

I’m doing it right now, and I’ll let you know the results.

lukastribus · June 6, 2019, 7:09am

No. Configure multiple private, reachable IPs on your NIC, and forward from the respective public IPs to those. That way you differentiate the traffic.

Brizoo · June 6, 2019, 7:30am

Awesome ! Thanks lukas, I didn’t think about that.

Topic		Replies	Views
Arbitrary TCP (and HTTP) over port Help!	3	4469	March 22, 2016
Backend using https on different port Help!	5	4862	November 9, 2018
Ssl redirection issue Help!	6	1695	November 3, 2017
HAProxy in `mode tcp` accepts HTTP with HTTPS backend Help!	1	610	March 3, 2020
Need help with a HTTP and HTTPS Proxy Config for multiple backend servers Help!	1	602	January 19, 2021

HAProxy forwarding HTTPS OK, TCP not OK. Please help!

Related Topics