Mutual-authentication in distroless container

lasse · September 2, 2020, 6:49pm

I’m trying to use Google Distroless to tighten the security of my HAProxy container workload.

Almost everything is working. Meaning general SSL termination, redirects, acl’s and what not. The only thing that is giving me a headache is mutual-authentication when using the distroless approach. N.B. mutual-authentication works when not using distroless!

I enabled verbose logging and the “error”/result I get is:

SOME_WAN_IP:40472 [30/Aug/2020:20:54:11.213] mutual_auth_clientcert/1: SSL handshake failure

This is my setup

The dockerfile

FROM haproxy:2.2.2 as build

#
# Initial configuration
#
ARG HAPROXY_USER=haproxy

#
# - Permissions: run as non-root user
# - File copying: needed HAProxy files to /opt for distroless in-sourcing
#
RUN addgroup --gid 1024 ${HAPROXY_USER} && \
    adduser --ingroup ${HAPROXY_USER} --system --shell /bin/bash ${HAPROXY_USER} && \
    mkdir -p /opt/etc && \
    cp -a --parents /usr/local/etc/haproxy /opt && \
    cp -a --parents /usr/local/sbin/haproxy /opt && \
    cp -a --parents /lib/x86_64-linux-gnu/libz.so.* /opt && \
    cp -a --parents /lib/x86_64-linux-gnu/libdl.so.* /opt && \
    cp -a --parents /lib/x86_64-linux-gnu/libpthread.so.* /opt && \
    cp -a --parents /lib/x86_64-linux-gnu/libm.so.* /opt && \
    cp -a --parents /lib/x86_64-linux-gnu/libc.so.* /opt && \
    cp -a --parents /usr/lib/x86_64-linux-gnu/libcrypto.so.* /opt && \
    cp -a --parents /usr/lib/x86_64-linux-gnu/liblua5.3.so.* /opt && \
    cp -a --parents /usr/lib/x86_64-linux-gnu/libpcre2-8.so.* /opt && \
    cp -a --parents /usr/lib/x86_64-linux-gnu/libpcre2-posix.so.* /opt && \
    cp -a --parents /usr/lib/x86_64-linux-gnu/libssl.so.* /opt && \
    cp -a --parents /etc/passwd /opt && \
    cp -a --parents /etc/group /opt

#################
# GO DISTROLESS #
#################
FROM gcr.io/distroless/base-debian10
COPY --from=build /opt /

# Set the container to run as the just created user
USER ${HAPROXY_USER}

#
# Boot value. The -W and -db options are needed in order for HAProxy to work
# with the distroless image
# -W    == indicates to the HAProxy process that it should run in `master-worker mode`
# -db   == disables background mode
#
CMD ["haproxy", "-W", "-db", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]

The HAProxy config

# ---------------------- #
# CONFIG & OVERALL SETUP #
# ---------------------- #
global
        # Global log options
        log stdout format raw local0 info

        # HAProxy configuration security
        daemon
        maxconn 1024

        # Stats
        stats socket /haproxysocket/haproxy.sock mode 660 level admin
        stats timeout 2m

        # Encryption conf.
        ssl-default-bind-options force-tlsv12 prefer-client-ciphers
        ssl-default-bind-ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1
        ssl-default-server-options force-tlsv12
        ssl-default-server-ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1
        tune.ssl.default-dh-param 2048

defaults
        log global
        mode http
        option httplog
        option dontlognull
        option forwardfor
        option http-server-close
        timeout connect 5000
        timeout client 50000
        timeout server 50000
        stats realm Haproxy\ Statistics
        stats refresh 10s
        stats uri /haproxy?stats
        stats auth user:pass

# --------- #
# FRONTENDS #
# --------- #
frontend http_front
        bind *:80

        # -------------- #
        # Endpoint rules #
        # -------------- #
        #
        # Other domains letsencrypt request and renew config
        #
        acl letsEncrypt path_beg /.well-known

        # a.domain.com
        acl a-domain hdr_dom(Host) -i a.domain.com
        acl a-domain-path path_beg /test
        http-request redirect code 301 location /test if !a-domain-path a-domain !letsEncrypt 

        # ----------- #
        # Add headers #
        # ----------- #
        http-request set-header X-Forwarded-Proto http

        # ------------------------------------------------ #
        # Set the backend to use for the various endpoints #
        # ------------------------------------------------ #
        # Lighttpd, with HTTP
        use_backend lighttpd_back if a-domain !letsEncrypt

        # Default backend
        default_backend fun-and-games

frontend port443_splitter
        bind :443
        mode tcp

        # -----
        # Layer 4 inspects to split requests to different backends, 
        # for further re-circulation to dedicated TLS frontends.
        # In order to support mutual auth.
        # -----
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }

        # ------------- #
        # ROUTING RULES #
        # ------------- #
        use_backend recir_clientcertenabled if { req_ssl_sni -i a.domain.com }
        default_backend recir_clientcertenabled

frontend mutual_auth_clientcert
        # Bind & conf. for a.domain.com 
        bind abns@mutual-auth-clientcert accept-proxy ssl crt /usr/local/etc/haproxy/pems/ ca-file /usr/local/etc/haproxy/mutualauth/ca.crt verify required crl-file /usr/local/etc/haproxy/mutualauth/ca_crl.pem alpn h2,http/1.1

        # -------------- #
        # Endpoint rules #
        # -------------- #
        # a.domain.com
        acl a-domain hdr(Host) -i a.domain.com
        acl a-domain-path path_beg /test
        http-request redirect code 301 location /test if !a-domain-path a-domain

        # ------------------------------------------------ #
        # Set the backend to use for the various endpoints #
        # ------------------------------------------------ #
        # Lighttpd backend, with HTTPS
        use_backend lighttpd_back if a-domain

frontend https_front
        # Setup HTTPS, load certificate files & advertise support for both HTTP2 & 1
        #bind *:443 ssl crt /usr/local/etc/haproxy/pems/ alpn h2,http/1.1
        bind abns@https-front accept-proxy ssl crt /usr/local/etc/haproxy/pems/ alpn h2,http/1.1

        # -------------- #
        # Endpoint rules #
        # -------------- #
        <JUST IMAGINE THAT THERE IS SOMETHING HERE>

        # ----------- #
        # Add headers #
        # ----------- #
        http-request set-header X-Forwarded-Proto https

        # ------------------------------------------------ #
        # Set the backend to use for the various endpoints #
        # ------------------------------------------------ #

        # Default backend
        default_backend fun-and-games

# -------- #
# BACKENDS #
# -------- #
backend recir_default
        # Re-direct/loopback to all frontends that is not using mutual authentication
        mode tcp
        server loopback-for-tls abns@https-front send-proxy-v2

backend recir_clientcertenabled
        # Used for a.domain.com
        mode tcp
        server loopback-for-tls abns@mutual-auth-clientcert send-proxy-v2

backend lighttpd_back
        redirect scheme https if !{ ssl_fc }
        http-response set-header Strict-Transport-Security "max-age=63072000; includeSubDomains;"
        compression algo gzip
        compression type text/html text/plain text/css application/javascript
        server localhost_lighttpd lighttpd-something:80 check

backend lighttpd_letsencrypt_back
        server localhost_lighttpd 127.0.0.1:8888 check

backend fun-and-games
        redirect scheme https if !{ ssl_fc }
        server localhost_pihole 192.168.0.20:808 check

Versions

HAProxy: 2.2.2
Docker: Docker version 17.09.1-ce, build 0bbe3ac
The base image used, FROM haproxy:2.2.2 as build, in the dockerfile can be glansed here HAProxy 2.2.2 base image dockerfile. As it can be seen from that dockerfile HAProxy is built with OpenSSL.

Checked and tried

The certificate is valid (mutual-auth works when I’m not using a distroless image)
I ran ldd on the HAProxy binary and it looks like I’m good in regards to needed libraries
I tried running strace on a working container HAProxy workload, with mutual-auth, I was not able to conclude anything really usefull
There are no errors at container image build time
To me this is likely somethign I’m not including/copying into the distroless end result in the container image. I’m at fault on what that might be
I tried my luck the HAProxy Slack channel. I got some initial help but I think this forum fits better for my issue and troubleshooting it. My thread is just slowly disappearing over on Slack.

My hope is that someone here on this forum is able and willing to be me in the right direction. What I’m missing to include in the end-result distroless image or potentially give some insight into how and what componets that are involved when a self-signed certificate is used for mutual-authentication in HAProxy

Thank you very much in advance. Any help is highly regarded

lasse · September 10, 2020, 7:02pm

Anyone with a tip?

Thank you

lukastribus · September 11, 2020, 7:35am

You have a complex configuration that is not working when you make that change to the container.

To troubleshoot it, you need to start with an empty configuration, and add feature by feature so you understand at which point exactly it begins to fail.

Topic		Replies	Views
Mutual TLS with SSL Termination Help!	3	2823	March 28, 2019
HAP Load Balancer forward to HAP Reverse Proxy: SSL handshake failure Help!	2	177	September 21, 2023
HAProxy SSL offloading/termination Help!	1	1407	March 31, 2017
Mutual authentication without TLS termination? Help!	0	184	October 23, 2023
Using SSL with HAProxy for docker containers Help!	33	23848	December 6, 2016

Mutual-authentication in distroless container

This is my setup

Related Topics