I am setting up a new haproxy server (I have some haproxy experience years ago at a different job)
It will not be load balancing, it is only doing reverse proxy (forwarding requests to appropriate webserver based on domain name used in URL).
I am planning to use SSL passthrough (at this point I don’t think I have to terminate it at haproxy for any reason and I still have to have it enabled on the webservers so passthrough would be the simplest. But to do SSL passthrough I think I have to use TCP mode and I can’t get it to work.) probably just missing something really simple but I haven’t found it.
for simplicity here I have removed my https front end and backends and am now only testing using http on port 80, once I have that working I’ll go on to https… So with http it works no problem in http mode using the following config (edited to actual server and domain names):
but if I test switching to tcp mode I can’t access the websites anymore (my browser tells me “ERR_EMPTY_RESPONSE”) at this point all I am doing is setting “mode tcp” in my front end and backends. No change to Global or Defaults sections and the front and backends become:
frontend haproxy-80
bind *:80
mode tcp
option tcplog
use_backend AM_http if { hdr(host) -i AM-dr.example.com }
use_backend AR_http if { hdr(host) -i AR-dr.example.com }
use_backend WS_http if { hdr(host) -i dr.WS-example.com }
backend AM_http
mode tcp
server C33-WEBDR 10.2.33.10:80 check
backend AR_http
mode tcp
server C01-WEBDR 10.2.1.10:80 check
backend WS_http
mode tcp
server C17-WEBDR 10.2.17.10:80 check
In your frontend your are trying to access the Host HTTP header to make your content switching decision (use_backend based on this HTTP header).
You cannot parse a HTTP request to access a HTTP header (like the Host header) if you are in TCP mode.
That is why haproxy is unable to select a backend server, and so there is no HTTP response.
Now, you don’t really need TCP mode for HTTP, but you do need it for HTTPS (SSL passthrough). In the SSL passthrough case you can access the SNI value of the ssl client_hello. This is a commonly used configuration that allows SSL passthrough but still permits to content switch based on the “hostname” aka the SNI value.
However there are some caveats with this. All your backend server certificates need to be unique, they can’t use overlapping certificates, otherwise the browser will try existing SSL session to access a different server.
So in short:
keep using HTTP mode for plaintext port 80 traffic
for SSL passthrough use SNI based content switching see below
Thanks so much. That works now if I just use https and forget about http. But unfortunately the overlapping certificates issue may be a problem for me. Can you explain more or point me to documentation on this?
When I first tested this new config. the first 2 sites in my config (the 2 using the same wildcard certificate) both came up with the webpage for the first one (AM-dr.example.com). At hat point my test websites were not enforcing SNI. I changed them to enforce SNI and then the second URL (https://AR-dr.example.com) failed to load at all. But opening it in a new incognito browser window it did load the correct site. Then I went back to the other browseer where it had previouosly failed to load and this time it does load. In other words I may have an issue with overlapping certificate or I may not. I do have many sites that will use the same wildcard certificate, and a few sites that each have their own cert. dr.WS-example.com fro example has its own cert
SNI based routing only works because the SNI value is in the very first packet (and in cleartext) of the TLS connection.
So haproxy can look at that and make routing decisions and then select the correct backend.
But once the routing decisions is made and the connection is established, there is no going back. The connection is now fully encrypted and what is done is done.
However the browser only sees:
i have established a secure connection to ar-dr.example.com IP address 1.2.3.4, the wildcard certificates covers the entire example.com domain
the user now want me to connect to dr-ws.example.com but which is also on IP address 1.2.3.4 and remember, the wildcard certificate already covers both hostnames
so the browser will use the existing TLS connection, because from a browser perspective, there is no difference. The IP address is the same, and the certificate covers it all
Haproxy has no way to know that the same TLS connection which had ar-dr in the SNI and is routed and still is routed to the ar-dr backend server is now no longer indented for ar-dr but for dr-ws.
Only when the browser forces a new connection (for example due to a keep-alive timeout or by switching to icognito mode), the problem will resolve itself because a new connection will go through the routing decision again.
Solutions:
don’t use overlapping certificates
disable H2 / H3 on the backend server, sticking to HTTP1 (because browser do this kind of connection reuse only on H2 and H3)
have your backend server send a “421 Misdirected Request” response, when facing unexpected Host headers
If none of this is an option, than you can’t use SSL passthrough and you will have to put a wildcard certificate on haproxy and use HTTP mode.
It looks like I will have to install certificates and use http mode. I wouldn’yt mind so much if just the one wildcard, but we also have some sites each with their own certtificate and maintaining them all in various places is cumbersome. or, i can do passthrough with the sites that have their own cert and https for the wildcard sites. I think to do that, i’d have to do 2 levels of sorting: one front end in TCP mode with backends listed for each of the sites that have their own certificate and a default backend that then redirects to a new http mode front end to sort everything that uses the wildcard. Or start in https mode and send the remainder to a new tcp mode front end. I’m not sure if one way is more efficient than the other,
Thanks again Lukas. And yes, as you said it would, it does work (to do TCP mode followed by http mode). But now I have a number of other issues that I need to solve. So in the end I don’t care if I have to install a bunch more certificates to get everything working right. I will do TCP only or HTTP only or a mixture, whatever it takes.
Here are my 3 current issues hopefully someone can clarify what I did wrong:
-1) for the tcp mode site, I can only load the website once. then if I refresh the page, I get “ERR_SSL_PROTOCOL_ERROR” in my browser. pasting the url inot a new incognito browser window works, but again, refrreshing the page fails
-2) for both the http mode and the tcp mode sites, I cannot enable SNI on the backend webserver. If I enable SNI on the backend webserver, then immediately the console of my ssh session with haproxy prints “haproxy[45395]: backend WS_https has no server available!”. And indeed if I try to browse to the site I get a 503 error. As soon as I turn SNI off on the backend, I can browse to the site. This issue is that some of our backend servers are hosting 2 diferent sites using SNI
-3) I am unable to connect to my websites tha run on port 5005, using http mode does not connect, when I browse to it I get a “404 - File or directory not found.” error
I have confirmed in each of the above cases that:
-1) if I have my browser bypass haproxy by entering the ip address of the backend servers in my local hosts file, and then using the exact same URL (by copy and paste) the webpage sucessfully displays. So the backend servers are working and with the URL’s I am using
-2) for above issues number 1 & 2 if I make simple test configs using only one front end, I get the same error.
-3) for above issue number 3, if I make simple test configs using only one front end, to be best of my memory, the port 5005 websites work, But if I do test configs with single port 443 front end and then a single port 5005 front end, that also fails but this last set of control tests is not documented and not 100% clear in my mind.
My current config is:
global
log /dev/log local0
log /dev/log local1 notice
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
stats timeout 30s
user haproxy
group haproxy
daemon
maxconn 4096
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
ssl-default-bind-ciphers AES-256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDHE-ECDHE-ECDSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_256_GCM_SHA384:TLS_ECDHE_RSA_WITH_AES_256_CGM_SHA384:TLS_ECDHE_ECDSA_WITH_AES_256_CGM_SHA384
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
defaults
log global
mode http
option httplog
option dontlognull
# option forwardfor
maxconn 200
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
#handles those sites with their own dedicated certificate, overflows to next sorting
frontend haproxy-https
bind *:443
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }
use_backend WS_https if { req.ssl_sni -i dr.WS-example.com }
default_backend https_SSL_terminated
#this extra step required to switch to an http mode frontend in order to process our wildcard sites
backend https_SSL_terminated
mode tcp
server loopback-for-https abns@haproxy-https send-proxy-v2
# next sorting level: handles wildcard certificate sites
frontend for-https
bind abns@haproxy-https accept-proxy ssl crt /etc/ssl/certs/example.com.pem
mode http
option http-keep-alive
timeout http-request 5s
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https
use_backend AM_https if { hdr(host) -i AM-dr.example.com }
use_backend AR_https if { hdr(host) -i AR-dr-api.example.com }
use_backend AR_https if { hdr(host) -i AR-dr.example.com }
default_backend Default_https
#now start sorting port 5005 sites with their own certs and then overflow to the wildcard sites
frontend haproxy-5005
bind *:5005
mode tcp
option tcplog
tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }
use_backend WS_5005 if { req.ssl_sni -i dr.WS-example.com }
default_backend https_5005_terminated
#this extra step required to switch to an http mode frontend in order to process our wildcard sites
backend https_5005_terminated
mode tcp
server loopback-for-5005 abns@haproxy-5005 send-proxy-v2
# next sorting level: handles wildcard certificate sites
frontend for-5005
bind abns@haproxy-5005 accept-proxy ssl crt /etc/ssl/certs/example.com.pem
mode http
option http-keep-alive
timeout http-request 5s
option httplog
option forwardfor
http-request set-header X-Forwarded-Proto https
use_backend AM_5005 if { hdr(host) -i AM-dr.example.com }
default_backend Default_https
backend Default_https
mode http
server C14-WEBDR 10.252.14.10:443 ssl check verify none
backend AM_https
mode http
server C33-WEBDR 10.252.33.10:443 ssl check verify none
backend AM_5005
mode http
server C33-WEBDR 10.252.33.10:5005 ssl check verify none
backend AR_https
mode http
server C01-WEBDR 10.252.1.10:443 ssl check verify none
backend WS_https
mode tcp
server C17-WEBDR 10.252.17.10:443 ssl check verify none
backend WS_5005
mode tcp
server C17-WEBDR 10.252.17.10:5005 ssl check verify none
Can you make a list of domains that you want to handle with SSL termination on haproxy, and a list of domain names that you want to SSL passthrough?
There is no need to change ports to 5005 or anything. HTTP remains on port 80, HTTPS on port 443.
There is also no need to enable SNI on the backend servers.
The idea is this:
The primary frontend listening on port 443 is a “mode tcp” frontend that routes the SSL passthrough domains directly to the backend with the real servers (a configuration that already works for you, except for the refresh issue due to overlapping certs). Only the domains you want to SSL terminate are redirect to haproxy itself will go through a “local connect” backend (like https_SSL_terminated), which then points to a SSL terminating frontend in “mode http”, that has all certificates and http configurations.
Yes, the current config is a proof of concept using test backend servers. Once I get it working I will deploy it to our 25 - 30 actual backend servers. In reality we have 7 backend servers each hosting their own domain which have their own individual certificates. I was planning on doing SSL passthough with them to avoid having to install (and re-install each year) their certificates on ha proxy. But if SSL termination works better I can certainly install all the certificates. We have another 16 sites, each on their own backend server but these 16 sites all use the same wild card certificate (each site is a subdomain). These 16 sites will have to do SSL terminate to avoid to the refresh issue due to overlapping certs For the above current testing config of three backend servers, the 2 that are using ssl terminate in http mode are: AM-dr.example.com AR-dr.example.com
and the one that is using ssl passthrough in tcp mode is: dr.WS-example.com
Our application has 2 parts a front end on 443 and an api on port 5005. clients make an initial connection on port 443 and then the front end sends them over to port 5005 where most of the work is done. I have to be able to filter for both 443 and 5005 and incoming requests for 5005 have to get forwarded to port 5005 for that backend server.
In my test config, the backends using port 5005 for api are: dr.WS-example.com AM-dr.example.com
As explained above but we have some customers where this doesn’t work because their corporate firewalls are blocking https requests to (or from) port 5005. so in these cases we have our application configured to run the api on port 443 but using a different subdomain. we use SNI for the webserver to know if requests are for the front end 443 site or the api 443 site. In my current test config we have only one backend server doing this, it is hosting the 2 sites: AM-dr.example.com AM-dr-api.example.com
So, yes for the backend servers using port 5005 for the api, we don’t have to enforce sni. But for those using 443 for both front end and qpi, we do have to enforce SNI
That is what I attempted to do in my config. But of course I added in port 5005 for some servers and SNI is required for others.
But in reality the 7 sites using individual certificates, will not be using port 5005 at all. they are an older application that has no api. I should have done port 5005 only on http mode as I would not need an initial tcp mode to sort through. It turns out all our webservers using port 5005 will be using a wildcard cert and using http mode. I will adjust this next test.
The last thing I should say is that eventually this will have to support websockets as well. Our api sites use websockets. I do not have the application running on my test backend webservers, just static websites with a single page saying “this is test __example site running on port 443 (or 5005)”.
That fact that you need to look at SNI at haproxy to be able to route requests to the correct backend server on the correct port does not mean you need to “enforce SNI” at the backend server.
Enforcing SNI at the backend servers serves no purpose here.
Here are some suggestions
Do not use health checks at all, unless you plan to load balance or failover between different servers. There is no point in generating health check load and config complexity when there is no server to failover to.
When you are using SSL passthrough, the traffic must not pass through any haproxy section with the SSL keyword enabled. Not on the frontend and not on the backend. Generally speaking in your case that should be backends in “mode tcp”. So you will have to remove the SSL keyword, and while you are at it, remove “check verify none” to disable health checks.
404 Errors come from backend servers, haproxy does not generate them. In most cases, this is because the Host headers is unexpected for the backend server, when non-standard ports are used. For example when the backend servers expects www.example.org as a Host header but gets www.example.org:1234, the vserver configuration doesn’t match and you get the backend servers default vhosts.
Thank you very much @lukastribus for all the help. I implemented all your latest suggestions and everything is now working as expected. Removing the “ssl” keyword on the tcp mode backends was what made the biggest difference, and of course not having to enable SNI on the backend server (I thought we had to but was totally wrong). One of the other errors I had was a misconfig on a backend server (at least it appears so as I just blew away the config and re-created it and it worked - not sure why).