Transition HA Proxy from mode TCP with SNI only to passthrough TLS encryption

Help!
1

Hello HAProxy Community,

I have 2 X HA Proxies running on RHEL8 VMs. They are in our DMZ. I use keepalived to cluster them together. They are working well.They ultimately direct internet traffic to our backend OpenShift Clusters.At the moment they are set-up with SNI only. This means the mode is TCP. Right now there is no http only https.All clients connect over HTTPS from outside my company. There are Firewalls and wildcard DNS etc between the HA Proxies and the internet.I have RH OpenShift as backend and all traffic back to OpenShift is HTTPS i.e. encrypted.We have CA signed certificates for the OpenShift Wildcard DNS .e.g. *.fancy.cool.com.Each individual application gets its own URL e.g. app1.apps.fancy.cool.com and app2.apps.fancy.cool.comThe CA signed certificates are currently only stored at the backend. In fact OpenShift uses HA Proxy as it ingress loadbalancer. It runs in PODs on the Cluster.

Overtime the limits of SNI have become apparent. For example it is not possible to find out source IP adddress and/or redirect anything as we have SNI and mode TCP. This means full encryption all the way through to the backend OpenShift Clusters.

The solution appears to be to transition to passthrough TLS encryption on the HA Proxies. If I just wanted to know the source IP then I could try send-proxy protocol () but as I want to redirect some URLS I think passthrough encyption is best way to go.

So first thing is confirm passthrough is best way to go? Given my requirements?

Next update I will share my haproxy.cfg file etc. So we can dig into the detail.

Thanks in advance for discussion and help.

Kevin.

2

“SNI only” doesn’t really mean any particular configuration, SNI is used in almost any configuration.

TCP mode too can be used in many configuration, although I do understand what you likely mean.

Those words are opposites of each other.

Let’s get the basics right first:

SSL/TLS passthrough means that haproxy is passing through the TLS session as-is, not decrypting or encrypting anything. It is acting at layer 4, moving the content on port 443 from the local host to a destination server. By buffering the first few bytes in the request, SNI based routing decisions are possible, but you do not have any access to plaintext values like HTTP headers of course. PROXY protocol can inject the informations about source IP address (because that comes before the TLS session) as you correctly indicate, but you are not able to

This is the configuration you currently have and it is called SSL/TLS passthrough.

If you need to intervene at HTTP layer, then you need to decrypt and encrypt TLS, by installing the certificates on haproxy and switching to HTTP mode.

That is called terminating TLS or TLS encryption/decryption.

You are mentioning a URL but your examples indicates hostnames.

Do you mean URL or hostnames?

If that means HTTP redirects, then yes, you need to terminate TLS and switch to HTTP mode.

You are already in passthrough mode. You need TLS termination by installing certificates on Haproxy.

3

Hi Lukastribus,

Thanks for prompt response. Apologies I did not explain my current set-up very well.

I will get my current configuration files shortly and share them. You have guessed correctly my configuration and what I want to do but I will confirm.

I currently don’t have the TLS certs stored on the HA Proxy RHEL VMs. HTTPS traffic arrives to them from the internet and gets directed to the backend OpenShift Clusters. There is no decryption etc done on the HA Proxys. I use SNI because I have a wildcard TLS cert and the OpenShift Cluster applications use wildcard DNS to publish applications.

This all works but means no intervention at HTTP layer (decrypt and encrypt). No manipulation can happen at the HA Proxy. I cannot redirect URLs and I cannot see the source IPs.

If you need to intervene at HTTP layer, then you need to decrypt and encrypt TLS, by installing the certificates on haproxy and switching to HTTP mode.

If that means HTTP redirects, then yes, you need to terminate TLS and switch to HTTP mode.

If I switch to HTTP mode is it still possible to receive HTTPS (encrypted) from the internet, intervene on the HAProxy (decrypt and encrypt) and then pass the traffic to the backend OpenShift Cluster as HTTPS (encrypted)? Or do I need to publish HTTP URLs instead of HTTPS URLS?

I am guessing I can simple keep some URLs as full HTTPS with SNI but what if I need to manipulate (make plain HTTP) some URLS but the traffic must arrive and leave HA Proxy encrypted?

You are mentioning a URL but your examples indicates hostnames.

OpenShift has an internal (to my company) wildcard DNS entry which publishes applications via HTTPS on port 443.

So app1.apps.fancy.cool.com and app2.apps.fancy.cool.com are https://app1.apps.fancy.cool.com and https://app2.apps.fancy.cool.com on OpenShift internally and they get published on the internet in the same manner. So on the internet we have a public wildcard DNS with those applications set-up. We currently use SNI on the HAProxies to direct the traffic back to appropriate OpenShift Cluster.

I will take out any sensitive information from my config files and share shortly. So you get a better picture.

Best regards,

Kevin.

4 
# Adjust the timeout to your needs
defaults
  timeout client 30s
  timeout server 30s
  timeout connect 5s

# Single VIP with sni content switching
frontend atomic-openshift-api
  bind 10.1.3.113:443
  mode tcp

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  use_backend %[req_ssl_sni,lower,map(/etc/haproxy/mappings/uat.map)] if { req_ssl_sni,lower,map(/etc/haproxy/mappings/uat.map) -m found }
  use_backend %[req_ssl_sni,lower,map(/etc/haproxy/mappings/prod.map)] if { req_ssl_sni,lower,map(/etc/haproxy/mappings/prod.map) -m found }

# UAT backend
backend backend_internal_lb_4_uat
  mode tcp
  balance source

  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk

  server X.Y.Z.252 X.Y.Z.252:443 check

# PROD backend
  backend backend_internal_lb_4_prod
  mode tcp
  balance source

  # maximum SSL session ID length is 32 bytes.
  stick-table type binary len 32 size 30k expire 30m

  acl clienthello req_ssl_hello_type 1
  acl serverhello rep_ssl_hello_type 2

  # use tcp content accepts to detects ssl client and server hello.
  tcp-request inspect-delay 5s
  tcp-request content accept if clienthello

  # no timeout on response inspect delay by default.
  tcp-response content accept if serverhello

  stick on payload_lv(43,1) if clienthello

  # Learn on response if server hello.
  stick store-response payload_lv(43,1) if serverhello

  option ssl-hello-chk

  server X.Y.W.252 X.Y.W.252:443 check



# Prod - map route to backend. key/value pairs


# app1
app1.apps.fancy.cool.com backend_internal_lb_4_prod

# UAT - map route to backend. key/value pairs


# app2
app2.apps.fancy.cool.com backend_internal_lb_4_uat
5

Yes, that’s possible, you would just re-encrypt again on the backend by adding the ssl keyword, optionally disabling SSL certificate verification (if the path to the backends is trusted).

server X.Y.W.252 X.Y.W.252:443 ssl verify none check

You will have to check if SNI is necessary for healthchecks and traffic, if that is the case add:

check-sni www.example.org sni str(www.example.org)
6

Ok this is a great information. Just to confirm here when you say “re-encrypt again on the backend” you mean the backend as defined on the HAProxy which is pointing to OpenShift Cluster which is the “real” backend.

Just to confirm the traffic will get re-encrypted as it leaves the HA Proxies and then arrive at my OpenShift Cluster encrypted.

You will have to check if SNI is necessary for healthchecks and traffic, if that is the case add:

Here do you mean if HA Proxy needs the SNI to do its healthchecks.

7

Yes, the ssl keyword in the backend section of the haproxy configuration will encrypt the plaintext HTTP again, so your openshift backends can stay on port 443 listening to HTTPS requests.

If your Openshift backends require specific SNI values in the SSL hello, which browsers previously send, then you need to add this configuration to the server backend configuration in haproxy, so that healtcheck (check-sni example.org) and SNI of traffic (sni str(www.example.org) is send.

8

Thanks for the confirmation. It helps my understanding a lot.

Great to know I can keep HTTPS (443) on backend OpenShift Cluster and decrypt and reencrypt on HA Proxy.

I don’t think the OpenShift backend requires SNI but I guess the application front ends (browsers on the internet) will be sending SNI as traffic is HTTPS and we have multiple DNS entries (one for each application) on a single internet IP. I am going to check/research this more.

The question /clarification from a HAProxy (and OpenShift backend) viewpoint is do we need SNI? Also if we just add it using check-sni can it do any harm? I think the the answer is no but would welcome your view on it.

Another question is is it possible to test the new set-up (decrypt on HA Proxy and reencrypt when leaving HAProxy) on a single URL. Or is it all or nothing?

Thanks for your help so far on this.

I am not sure if OpenShift end requires the specific SNI

Now on the SNI settings …

Honestly I don’t know if the browsers (application front end) do send SNI. I know they all hit

https://app1.apps.fancy.cool.com

9

Today, if the browser is pointing to https://app1.apps.fancy.cool.com it is sending SNI app1.apps.fancy.cool.com which your backend receives.

You can absolutely just configure it, because it will not do any harm.

In this case it would look like:

check-sni app1.apps.fancy.cool.com sni str(app1.apps.fancy.cool.com)

Per URL never, because haproxy cannot breakup a encrypted SSL session.

You can do per SNI rules, like you are doing today, but instead of pointing to a openshift backend server you point it to a different haproxy frontend that actually terminate TLS.

So for example you can do this for app1.apps.fancy.cool.com specifically.

10

Per URL never, because haproxy cannot breakup a encrypted SSL session.

You can do per SNI rules, like you are doing today, but instead of pointing to a openshift backend server you point it to a different haproxy frontend that actually terminate TLS.

So for example you can do this for app1.apps.fancy.cool.com specifically.

OK cool. Now we are getting to meat of how I could migrate to the new solution safely and/or test it.

So lets take app1.apps.fancy.cool.com at a high level I still accept it has as https://app1.apps.fancy.cool.com on the HAProxy. I then decypt, do whatever manipulation is required or send on source IPs etc. Then I point it to another HAProxy frontend which recrypts. I set the new individual frontend (with reencrypt) to have the original OpenShift backend. They talk HTTPS between each other.

Is there a name for this method so I can search for an example?

11

Sometimes it is referred to as recirculation, but it varies.

This would be an approximate example for your case:


# Single VIP with sni content switching
frontend atomic-openshift-api
  bind 10.1.3.113:443
  mode tcp

  tcp-request inspect-delay 5s
  tcp-request content accept if { req_ssl_hello_type 1 }

  use_backend recirculation_tls_termination if { req_ssl_sni,lower app1.apps.fancy.cool.com }
  use_backend %[req_ssl_sni,lower,map(/etc/haproxy/mappings/uat.map)] if { req_ssl_sni,lower,map(/etc/haproxy/mappings/uat.map) -m found }
  use_backend %[req_ssl_sni,lower,map(/etc/haproxy/mappings/prod.map)] if { req_ssl_sni,lower,map(/etc/haproxy/mappings/prod.map) -m found }

# backend to reroute to TLS terminating frontend
backend recirculation_tls_termination
  mode tcp
  server 127.0.0.1:10443

# regular TLS termination frontend
frontend atomic-openshift-api-tls-termination
 mode http
 bind 127.0.0.1:10443 ssl crt /path/to/ssl/cert.pem
 http-request set-header X-Bla 123
 use_backend backend_internal_lb_4_prod_tls_termination


backend backend_internal_lb_4_prod_tls_termination
  mode http
  server X.Y.W.252 X.Y.W.252:443 ssl check check-sni app1.apps.fancy.cool.com sni str(app1.apps.fancy.cool.com)

We can see:

  • in the existing frontend we only add a new use_backend rule before other rules for a specific sni value, and then we redirect this to a particular recirculation_tls_termination backend
  • this backend only points to another port of haproxy itself; where we DO actually terminate TLS
  • on the frontend atomic-openshift-api-tls-termination we do regular TLS termination in HTTP mode, as you can see we can interviene on the HTTP headers
  • on the backend backend_internal_lb_4_prod_tls_termination we do regular backend stuff, but reencrypt the traffic before going to the openshift backend servers.