Two workloads on the same port


#1

I have a use case where I am expecting two work loads on the same 443 port. One expects client certs to be validated whereas, other doesn’t expect that.

is it possible to have two workloads on the same port like below? And backend is common for both.

frontend https
bind 104.191.20.22:443 ssl crt /opt/haproxy/ssl/host_key_cert.pem

frontend https
bind 35.196.140.91:443 ssl crt /opt/haproxy/ssl/host_key_cert.pem ca-file /opt/haproxy/ssl/ca_cert.pem verify optional

IP address in bold above are the IPs of the load balancers from where the two workloads are coming.

A quick response is appreciated. Thanks.


#2

In the bind line you specify the (destination) IP address that you want haproxy to listen to. If you have 2 IP addresses on the box running haproxy and you want to use client cert authentication on one and no client authentication on the other, than you can do it the way you are proposing.

However, you do not specify the source IP “from where the two workloads are coming” in the bind line.

I believe you don’t want the former, but the latter …

So based on the client IP address?

Then no, it is not gonna work like this. I assume you cannot use “verify optional” on both, because causes the browser ask for the client cert in both cases, is that a correct assumption?

In that case, you will have to configure a TCP frontend without SSL termination, and route the traffic based on an IP ACL to specific backends that then reroute the traffic to a SSL terminating frontend, appropriately configured.

Something like this:

frontend port443
 bind :443
 use_backend recir_noauth if { src 104.191.20.22 }
 use_backend recir_default if { src 35.196.140.91 }
 #default_backend recir_default

backend recir_noauth
 server loopback-for-tls abns@haproxy-noauth send-proxy-v2
backend recir_default
 server loopback-for-tls abns@haproxy-default send-proxy-v2

frontend fe-ssl-noauth
 mode http
 bind abns@haproxy-noauth accept-proxy ssl crt /opt/haproxy/ssl/host_key_cert.pem
 # actual frontend configuration 
frontend fe-ssl-default
 mode http
 bind abns@haproxy-default accept-proxy ssl crt /opt/haproxy/ssl/host_key_cert.pem ca-file /opt/haproxy/ssl/ca_cert.pem verify optional
 # actual frontend configuration

#3

Thanks for your response. Thats what I thought as well. But when I tested the same request from the two LBs, I saw logs are telling that they are being routed to their respective frontends without any rerouting. So, it seems like, in the bind line, it is honoring the client ip as well. Here is my configuration again with a small test that I performed.

Haproxy configuration

frontend https-1
35.196.239.755.196.140.91:443 ssl crt /opt/haproxy/ssl/host_key_cert.pem ca-file /opt/haproxy/ssl/ca_cert.pem verify optional
#actual fronend configuration
frontend https-2
bind *:443 ssl crt /opt/haproxy/ssl/host_key_cert.pem
#actual frontend configuration

Here are my test request

curl -k https://104.191.20.22/js/api.enlighted.js
curl -k https://35.196.140.91/js/api.enlighted.js

Here is the output from haproxy.log

[08/Dec/2017:19:47:32.389] https-2~ api-server/www-1 231/0/1/3/490 200 298841 - - ---- 1/1/0/1/0 0/0 “GET /js/api.enlighted.js HTTP/1.1”
[08/Dec/2017:19:47:59.005] https-1~ api-server/www-1 195/0/0/3/447 200 298841 - - ---- 1/1/0/1/0 0/0 “GET /js/api.enlighted.js HTTP/1.1”

Do you see any issue here?


#4

No, this is wrong. You may get load-balanced between the 2 sockets and by sheer luck hit the right frontend.


#5

I made changes suggested by you and modified my haproxy.cfg as below but I keep getting this error.

log snippet:

Mar  5 20:11:29 beta1-haproxy1-n01-us-east1-c haproxy[3157]: 40.211.11.12:50362 [05/Mar/2018:20:11:29.397] https-all htt
ps-all/<NOSRV> -1/-1/-1/-1/74 400 188 - - PR-- 0/0/0/0/0 0/0 "<BADREQ>"

haproxy.cfg snippet:

frontend https-all
    bind *:443
    acl is_gateway_host hdr(host) -i device.mycompany.com
    use_backend recir-client-auth if is_gateway_host
    default_backend recir-default
 
backend recir-client-auth
    server loopback-for-tls abns@haproxy-client-auth send-proxy-v2

backend recir-default
    server loopback-for-tls abns@haproxy-default send-proxy-v2

frontend https-devices
    mode http
    bind abns@haproxy-client-auth accept-proxy ssl crt /opt/haproxy/ssl/host_key_cert.pem ca-file /opt/haproxy/ssl/ca_cert.pem verify optional
    default_backend gateway-server

frontend https-api
    mode http
    bind abns@haproxy-default accept-proxy ssl crt /opt/haproxy/ssl/host_key_cert.pem
    default_backend api-server

Also, what is send-proxy-v2 here?


#6

You are probably setting “mode http” in the default section. Either remove it there, or statically put the first 3 sections from your snippet in “mode tcp”.

This is wrong, you are not in HTTP mode here, you cannot match a HTTP header. Remember, haproxy only sees encrypted SSL traffic here. I proposed the ACL’s matching source IP addresses as per your original requirement in my example. Would you like to match something else?

Please clarify what data you would use to decide between client-authentication or no client-authentication.

It will make sure you are seeing the original client-IP on your backend.


#7

Thanks for the quick response. Originally I was trying to use source IP addresses but then I realized that I could use the hostname that is coming in the header. This way I don’t need to hardcode ip addresses per environment.

The two workloads are coming from two different domains:

device.mycompany.com
and 
api.mycompany.com

#8

No, you cannot match the Host header, because this is SSL and that means it is encrypted.
Please try one thing at the time.

So, put your section in the correct mode like I previously suggested, make it work and then go on with improvements you think are useful.

If matching SNI is acceptable for you, that is something that will work as it is unencrypted in the client_hello. But please do one thing at the time here, and make the basics work first.


#9

Ok, I removed the Host header check and mode http as it was there in the default section. Now trying with source IP based routing. But still getting below error in the log.

Mar  6 17:44:51 beta1-haproxy1-n01-us-east1-c haproxy[11318]: 40.211.11.12:63872 [06/Mar/2018:17:44:51.822] https-all https-all/<NOSRV> -1/-1/0 0 SC 0/0/0/0/0 0/0

haproxy.cfg:

defaults
    log     global
    mode    http
    option httpclose
    option forwardfor
    option http-server-close
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend https-all
    bind *:443
    mode tcp
    use_backend recir-client-auth if { src 104.191.20.22 }
    use_backend recir-default if { src 35.196.140.91 }

backend recir-client-auth
    mode tcp
    server loopback-for-tls abns@haproxy-client-auth send-proxy-v2

backend recir-default
    mode tcp
    server loopback-for-tls abns@haproxy-default send-proxy-v2

frontend https-devices
    bind abns@haproxy-client-auth accept-proxy ssl crt /opt/haproxy/ssl/host_key_cert.pem ca-file /opt/haproxy/ssl/ca_cert.pem verify optional
    default_backend gateway-server

frontend https-api
    bind abns@haproxy-default accept-proxy ssl crt /opt/haproxy/ssl/host_key_cert.pem
    default_backend api-server

#10

Ok, that’s a different error message and a different root cause though.

You are coming from the IP 40.211.11.12, however you have only configured haproxy to deal with the IPs 104.191.20.22 and 35.196.140.91.

So either create a default_backend rule, or handle 40.211.11.12.


#11

Looks like things are working for me after making few changes. Thanks for your help here.

Now coming back to SNI in the client_hello, how do I capture it in the tcp mode?


#12

Use req.ssl_sni:

tcp-request inspect-delay 5s
tcp-request content accept if { req.ssl_hello_type 1 }

use_backend recir-client-auth if { req.ssl_sni -i clients.example.org }
use_backend recir-default if { req.ssl_sni -i www.example.org }

How to send https traffice to correct destination based on the subdomain it was sent to?
#13

Cool … one of my request is still not able to meet this check.
How do I print the value of req.ssl_sni in haproxy logs?

Also, if I have “option httplog” set at default level, can I override it to no-logs in the frontend block where I handle tcp mode? It automatically switches to tcplog there.


#14

Use the custom log format and access the req.ssl_sni variable.

I suggest to remove “mode http” and “option httplog” from you first default section, and create a second default section between the last tcp backend the the first http section, and in that default section you add:

default
 mode http
 option httplog