mTLS tunnel between two HAProxy instances?

Hi, I’ve read several client side examples which involve a client and one HAProxy server. However, I’m wondering if it is possible to do the following with HAProxy.

I would like to have two HAProxy instances, call one client and one server. The server will act as a hub to many HAProxy clients. I would like the client side’s backend to present a client cert to the server sides front end as well as the server side front end present a ‘server’ cert to the client such that the 2 HAProxy instances establish a mTLS link between them.

Then on the client side I would like the front end to perform TCP pass thru from any clients (such as curl) which connect to it and I want that passed thru connection to go across this mTLS tunnel to the server which would have a backend config that forwards it to a server behind it.

Here’s a diagram of what I am trying to describe. TIA!

Not sure what mTLS means? Do you mean mutual authentication?

Yes, you can configure haproxy to send a client certificate to the server, check the documentation (specifically the ssl and crt keywords in the section "server and default server options).

mTLS == mutual TLS. I saw the sections on ssl and crt. However, I am not trying to have HAProxy send a client cert to the HTTPS server in my diagram. I want the 1st HAProxy instance one the left to send a client cert to the 2nd HAProxy instance on the right to secure the connection between the two HAProxy servers (the fat red arrow between the HAProxy instances).

I want to have the client on the far left connect (the blue line) to the HAProxy server on the left and then get forwarded over to the HAProxy on the right using the mutually authenticated connection between the two HAProxy servers and then from there on to the HTTPS server. The fat red arrow (mTLS connection) should be transparent to the client and the HTTPS server it is connecting to.

The configs I have tried so far look like this

Client side (left) HAProxy

    global
        maxconn 256

    defaults
        mode http
        timeout connect 5000ms
        timeout client 50000ms
        timeout server 50000ms

    frontend http-in
        mode tcp
        bind *:443
        default_backend servers

    backend servers
        mode tcp
        server server1 172.16.42.4:9090 ssl crt /usr/local/etc/haproxy/client1.crt maxconn 32

Server side (right) HAProxy

global
   maxconn 256

defaults
    timeout connect 5000ms
    timeout client 50000ms
    timeout server 50000ms

frontend http-in
    mode tcp
    bind *:443 ssl crt /usr/local/etc/haproxy/server.pem ca-file /usr/local/etc/haproxy/ca.crt verify required
    default_backend servers

backend servers
    mode tcp
    option tcplog
    server server1 10.200.12.94:443 check maxconn 32

Yeah what you mean is mutual authentication.

Not sure what you mean, that is exactly what you seem to need and also what you configured?

Config is wrong, you are missing the ssl keyword on the server configuration in the client side left haproxy instance and your port does not match the other one (listening on 443 but connecting to 9090).

Thanks for pointing out the missing ssl keyword. I added that but still seeing the same behavior. The ports are different because these HAProxy instances are docker containers. The right side one is listening to 443 in the container which is mapped to 9090 on the host. I should have mentioned that in the original post. Sorry for the confusion.

In any case even adding the ssl keyword I still see the same issue. Namely when I curl (the client box in the diagram) the HTTPS Server I get:

*   Trying 172.16.42.4...
* TCP_NODELAY set
* Connected to foo.bar.net (172.16.42.4) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/etc/openssl/cert.pem
  CApath: /usr/local/etc/openssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS alert, Server hello (2):
* SSL certificate problem: self signed certificate
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (60) SSL certificate problem: self signed certificate

The only self signed certificates are the ones referenced by my HAProxy configs. The one that foo.bar.net presents is not. So it seems that curl is seeing one of these HAProxy certs.

What I thought I had configured was HAProxy on the left would accept a connection to and forward to the backend server 172.16.42.4:9090 which is the frontend to HAProxy (right). I wanted the backend of HAProxy left to present the client cert to HAProxy (right) frontend and HAProxy (right) frontend to present the referenced cert (server) to HAProxy (left) backend, the HAProxy server itself, not the client connection that is being proxied across it. It seems that the HAProxy server right’s server cert is being presented to the CURL client instead of HAProxy (left backend) itself. Maybe HAProxy can’t work like this?

Thanks!

Haproxy supports this setup just fine, there must be some other problem. Double check the actual crt files, the final HTTPS server configuration, docker port mappings, etc.

For example, do you see the correct certificate when accessing https://10.200.12.94:443 ?