Mixing mode tcp and http - SSL termination and Passthrough

Help!
1

Hello!

My last thread is here for reference: Cannot bind socket 80 / 443

That got everything working just fine. At the time I wanted to terminate all SSL at HAProxy. Works beautifully. I’m standing up a new service which seems to really hate having SSL terminated upstream. It seems I require two frontends. One in http mode for sites which are terminating SSL at HAProxy. One in tcp mode for sites which are having SSL passed through to them.

The rub: I know I can’t bind the same port twice. Both terminated and passthrough servers are using ports 80 and 443. I’ve seen a few other threads, one where someone was using varnish to solve some of this, another where they only used tcp with and SNI route (this thread: 2 frontends SSL over HTTP and TCP?) to get the desired result. I’m not quite sure what’s going on with the tcp only solution.

I’ve been through the documentation a few times trying to sort this out. Can’t seem to make heads or tails of it (just not familiar enough with HAProxy yet). Any help or suggestions are appreciated.

Thanks! <3

1 Like
2

Just a friendly bump. Anyone have a solution or hint for getting http and tcp frontends in the same config / proxy?

Alternatively: a way to use tcp mode for everything and SSL termination on only select sites (pretty sure I can’t do that though).

3

SNI based switching is the way to go, when you have only 1 public IP address. If you have multiple IP addresses, then just bind to different IP addresses in your frontends…

For the SNI solution, you have to match either the SNI value of the SSL terminating hostnames, or the the value of the hostname(s) for the passthrough.

You can find a better example here:

It’s about different SSL parameters (client auth), in your case one of the second tier frontends would be in tcp mode …

1 Like
4

Hey Lukas,

Understood. I gave this another whirl. Here’s what I’ve got so far:
https://hastebin.com/jukejobihe.vbs

This seems like a reasonable configuration to me, however; the SSL Terminated servers are no longer working. I either receive a timeout or connection refusal.

Is the configuration improper?

5

I’ve edited further and commented out most of the config. just trying to get the bare bones working.

Here’s the current (broken) config: https://hastebin.com/kamoyiyaqi.vbs

Can you see anything wrong? I’m not sure why I’m now getting refused connection and “ERR_SSL_PROTOCOL_ERROR” messages.

6

Alright… Given the lack of traction getting this to work properly I’ve opted to try using only TCP mode.

Here’s the new configs.
HAProxy: https://hastebin.com/lawogupepe
nginx: https://hastebin.com/vigirafogi.nginx

This seems to want to work, however; now HAProxy is complaining that the backend servers are down:
Server blechinger.io_wordpressServers/pnWordpress01 is DOWN, reason: Layer6 invalid response, check duration: 3ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue..
Issuing nc -zv 10.0.10.200 80 returns:
Connection to 10.0.10.200 80 port [tcp/http] succeeded! - so I’m not quite sure what’s wrong here.

Thoughts? Any way to grab more sophisticated logs from the haproxy side?

An aside: is there are more active location for community assistance?

Thanks!

7

All I want at this point is a working ssl passthrough setup. I’ve tried a bunch of different recommended configurations both both HAProxy and nginx. None of them seem to work.

Requirements:

  1. HAProxy passes SSL through to whatever backends I have set up.
  2. 301 redirects for http to https (should be done on backend nginx servers (HAProxy shouldn’t care about this)).
  3. HSTS is enabled (should be done on backend nginx servers (HAProxy shouldn’t care about this)).
  4. Restrict protocols to TLS V1.1 and TLS 1.2
  5. Restrict ciphers to EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH.

This is what I’ve come up with in an attempt to achieve the above goals:

HAProxy config: https://hastebin.com/ejanazuyay
nginx config: https://hastebin.com/eqatewewil.nginx

Currently HAProxy doesn’t see the backend servers. Connection fails from the browser.

8

Haproxy makes a layer 6 check (SSL) here, while you expect a layer 4 check, and of course the backend has no SSL layer on port 80, so it fails.

Remove option ssl-hello-chk from blechinger.io_wordpressServers.

9

I have the same issue as you have
But in TCP everything is working and I want mixed conf.

https://hastebin.com/exasufakal.pl

As soon as I had a crt to the bind 443 … nothing works anywore.
Wih this conf, SSL is passthrough and certains servers are answering on TLS 1.0

Thank for help :slight_smile:

10

@stanthewizzard don’t hijack other peoples threads. If you have a problem, open your own thread please.

11

@lukastribus Yep. True. HAProxy now sees the backend. Very silly of me.

Now I can see requests coming in from HAProxy via nginx access logs, however; everything coming from HAProxy just shows a bunch of hex values in the nginx access log…

10.1.51.30 - - [11/Jul/2018:14:43:18 -0400] "\x16\x03\x01\x00\xC0\x01\x00\x00\xBC\x03\x03\xA0P\xD8f\x97\xDF\xF0\xE04\xF6\xEB\xE1\xA1,\xB1w\xFC;\x0C\...truncated...\xCC\xA8\xC0\x13\xC0\x14\x00\x9C\x00\x9D\x00/\x005\x00" 400 166 "-" "-"

I’ve googled around trying to figure out what this means but… No luck.

Running an ssllabs test against any of my domains shows that no security protocols are enabled. I only have TLS 1.2 enabled (following Mozilla’s Server Side TLS guide for Modern browser compatibility: https://wiki.mozilla.org/Security/Server_Side_TLS) - something funky is going on. Browsers either get a closed connection or a SSL_ERROR_RX_RECORD_TOO_LONG error. I know the .pem files I’m using are good. They are the same ones I was using for the few servers I had set up previously using https and ssl truncation at HAProxy. The certs are valid. I’ve tried from multiple devices and browsers and see the same results.

I’ve modified the nginx configuration slightly a few times regarding the SSL commands. Removing all of the ssl configuration lines (and just allowing nginx to handle SSL in its default way) still yields the same results.

If I cut HAProxy out of my stack and access the webserver directly (host file points the domain name to a specific internal IP) I am able to access my domains.

Updated configs:
HAProxy: https://hastebin.com/bevemifusi
nginx: https://hastebin.com/evamugunur.nginx

nginx config has ssl lines for the blechinger.io site commented out just for testing… Didn’t make any difference as stated above.

Is the HAProxy config not allowing HTTPS to reach the server? I had thought that:

,--------------------------------------------------------------
| Browser request for x
|    |
| Internet
|    |
| Firewall--> HAProxy
|                |                                        +---------------------------+
|                +-> SNI Rules                            |                           |  
|                       |                                 V                           |
|                       +-> Proxy to x over port 80 --> nginx 301 return to port 443 -+
|                       +-> Proxy to y over port 80       |
|                       +-> Proxy to z over port 80       +--> nginx ssl over port 443 --> serve up x for browser

BUT it seems like I’m getting encrypted requests over port 80… Any idea what’s going on?

EDIT:

Changing some of the SSL only backends from port 80 to port 443 saw a slight improvement. I can now hit some domains some of the time. It seems that sometimes the wrong certificate is being used. Also: pterodactyl.blechinger.io is a completely different webserver from blechinger.io, however; it seems HAProxy is directing pterodactyl.blechinger.io to the blechinger.io webserver.

Very confusing.

12

It"s not an hijack at all

I’m giving my TCP settings that are working (and I think it’s one of the requirement for @blechinger
and saying that those settings don’t work for http (same issue that he has)

this a warm welcome here :((((

1 Like
13

Sorry for this, it was my impression your where asking for help. In that case, your post is obviously welcome.

The config forces everything to port 80 on the backends. It also cannot content-switch HTTP (non-SSL) based on SNI, because SNI is part of SSL.

So you need to make a separate frontend for unencrypted HTTP Traffic, use http mode and content switch based on the Host header. You will also need distinct backends for that. And you need to use port 80 as destination port for HTTP and port 443 as destination port for HTTPS.

I assume your certificates are overlapping. For SNI content-switching to work, your backend server must not have overlapping SANs in the certificate.

For example: if one if your servers has a wildcard certificate of *.blechinger.io, and the browser hits this server first, the SSL connection will still be open and the browser will also request pterodactyl.blechinger.io through that very same SSL session, because the certificate covered this as well. Haproxy only sees encrypted traffic at this point. So you cannot have certificates with overlapping SANs when you content-switch based on SNI.

1 Like
14

@lukastribus
Ok so cool :slight_smile:

So btw, the config I posted is working well but only on TCP. If it can helps …

Just (in order to help) you could try a wildcard cert with letsencrypt to avoid cert overlapping ?

15

The certs overlapping because of SNI makes perfect sense. If that’s the case then I cannot use SNI redirection and will have to stick with use_backend if statements based on HDR. Which means http mode. Which means I’m back where I started.

I will not be using letsencrypt. I already have good working certificates. I just can’t use SNI switching for my HAProxy use-case. Unfortunate.

I’ve moved everything back to mode http and am terminating SSL at HAProxy. Most of the webservers are working fine. The difference is that I have a unique frontend for each individual port. Even though, for instance, ports 8443 and 443 should be treated the same (as far as HAProxy is concerned) having both ports in the same frontend caused problems.

I know that’s not particularly useful information for those that find this thread in the future hoping for a fix for TCP -> HTTP modes but it’s what’s working for me.

The few things I need to have as TCP have a dedicated frontend and backend that’s tcp only. This doesn’t achieve my end-goal of ssl termination on some http sites and ssl passthrough for others. It only allows some services through.

16

Old post, but I think I solve that same problem, it was driving me crazy. The solution is to CONCATENATE a backend with a frontend, like this:

#######haproxy.cfg_BEGIN##################
…
…
frontend SSL_PassThrough
mode tcp
bind *:80
bind *:443
tcp-request inspect-delay 5s
tcp-request content accept if { req_ssl_hello_type 1 }

use_backend backend1 if { req_ssl_sni -i aaa.bbb.com }
use_backend backend2 if { req_ssl_sni -i ccc.ddd.com }
use_backend bbackend3 if { req_ssl_sni -i eee.fff.com }
default_backend bk_tcp_to_https

backend bk_tcp_to_https
mode tcp
server haproxy-https 127.0.0.1:8443 check

frontend SSL_Termination
mode http
bind *:8443 ssl crt /etc/haproxy/certs/ggg.hhh.com.pem crt /etc/haproxy/certs/iii.kkk.com.pem

use_backend     backend4  if { hdr(host) -i ggg.hhh.com }
use_backend     backend5  if { hdr(host) -i iii.kkk.com }

#SSL Passthrough Backends (every backend manage their own SSL termiantion)
backend backend1
mode tcp
server server1 192.168.0.100:443 check

backend backend2
mode tcp
server server2 192.168.0.101:443 check

backend backend3
mode tcp
server server3 192.168.0.102:443 check

#SSL Terminated by HAProxy Backends (plain http traffic between HAProxy and these backends)
backend backend4
mode http
server server4 192.168.0.104:80 check
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }

backend backend5
mode http
server server5 192.168.0.105:80 check
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
#########haproxy.cfg_END#################

The Trick here is “default_backend bk_tcp_to_https” in frontend1 that concatenates all the requests that are not going to be passthrough, to the “backend bk_tcp_to_https”.
It is basically telling if the request is not for aaa.bbb.com or ccc.ddd.com or eee.fff.com, then send it to the default backend, which is himself: 127.0.0.1 but in other port: 8443, and then, Frontend2 is listening in 8443 to take care of the termination of the SSL.

This is the mixed TCP/HTTP combination to passthrough some SSL and terminate others using one single configuration.

3 Likes
17

thanks alot :smiley:

18

I have recently just done this with a concatenated config to have some systems ssl terminated and others not. It works quite nicely except for the fact that I can’t get the client IP address to the ones I want ssl terminated. I can see the client IP hitting the TCP frontent in the PFsense HAProxy dashboard. I have tried to use send-proxy to the backend that loops bact to the 127.0.0.1 address and then all my sites break that are ssl terminated. Just not sure if I am attaching it to the right place or if I need to add accept-proxy to a front-end. I’m using the haproxy install in pfsense

Any help would be great

UPDATE:
All good. Looks like all I needed to do was add ‘accept-proxy’ to the Bind Pass thru on my shared frontend and also ‘send-proxy’ on my looped back port 8443 backend

I can now set ACL’s properly to limit access by IP

Figured it out from this Link