Mixed http tcp works, but

So I have a work in progress configuration. Which works if you the comment the bind lines that are used instead of the unix bind lines.

I figured this would work but, it simply does not.

How can I configure this so that sockets are used instead of tcp?

My haproxy -vv

HAProxy version 2.9.9-ad75c48 2024/06/14 - https://haproxy.org/
Status: stable branch - will stop receiving fixes around Q1 2025.
Known bugs: http://www.haproxy.org/bugs/bugs-2.9.9.html
Running on: FreeBSD 14.1-RELEASE-p3 FreeBSD 14.1-RELEASE-p3 GENERIC amd64
Build options :
  TARGET  = freebsd
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wnull-dereference -fwrapv -Wno-unknown-warning-option -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment -DFREEBSD_PORTS
  OPTIONS = USE_GETADDRINFO=1 USE_OPENSSL=1 USE_ACCEPT4=1 USE_ZLIB=1 USE_CPU_AFFINITY=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 -BACKTRACE +CLOSEFROM +CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ENGINE -EPOLL -EVPORTS +GETADDRINFO +KQUEUE -LIBATOMIC +LIBCRYPT -LINUX_CAP -LINUX_SPLICE -LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL -PRCTL +PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT -RT +SHM_OPEN -SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD -TFO +THREAD -THREAD_DUMP +TPROXY -WURFL +ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=1).
Built with OpenSSL version : OpenSSL 3.0.12 24 Oct 2023
Running on OpenSSL version : OpenSSL 3.0.13 30 Jan 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with the Prometheus exporter as a service
Built with zlib version : 1.3
Running on zlib version : 1.3.1
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with PCRE2 version : 10.43 2024-02-16
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 16.0.6 (https://github.com/llvm/llvm-project.git llvmorg-16.0.6-0-g7cbf1a259152)

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
global
        default-path config

        maxconn 5000

        chroot /var/run/haproxy/
        user haproxy
        group haproxy

        daemon
        pidfile /var/run/haproxy.pid

        hard-stop-after 5m

        # Stats
        stats socket /var/run/haproxy-svc1.sock level admin mode 600 user haproxy expose-fd listeners
        stats timeout 1h

        # send logs to stderr for logging via the service manager
        log stderr local0 info
        log 127.0.0.1:514 local0

        # intermediate security for SSL, from https://ssl-config.mozilla.org/
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options prefer-client-ciphers no-tls-tickets

# default settings common to all HTTP proxies below
defaults http
        mode http
        log global
        timeout client 1m
        timeout server 1m
        timeout connect 10s
        timeout http-keep-alive 2m
        timeout queue 15s
        timeout tunnel 4h  # for websocket

# provide a stats page on port 8181
frontend stats
        bind :8181
        # provide advanced stats (ssl, h2, ...)
        stats uri /
        stats show-modules
        # some users may want to protect the access to their stats and/or to
        # enable admin mode on the page from local networks
        #  stats auth admin:mystats
        #  stats admin if { src 10.0.0.0/8 172.16.0.0/12 192.168.50.0/16 127.0.0.0/8 }

cache cache
        total-max-size 200        # RAM cache size in megabytes
        max-object-size 10485760  # max cacheable object size in bytes
        max-age 3600              # max cache duration in seconds
        process-vary on           # handle the Vary header (otherwise don't cache)

# TCP frontend, that splits https to a https backend and passes TCP to tcp
# frontend fe_main_tcp
#         mode tcp
# #       option tcp-smart-accept
#         option tcplog
#         #bind :80,:443
#         tcp-request inspect-delay 10s

#         use_backend web_staging if { req_ssl_sni -i www.someurl.com }

#         default_backend be_main_tcp

# backend be_main_tcp
#         mode tcp
# #       option tcp-smart-connect
#         server common-http unix@/var/run/haproxy/common-http.sock
#         #server common-http unix@common-http.sock send-proxy-v2

listen tcp_to_https
        mode tcp
        bind :80
        bind :443 
        
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }

        #server httpsbackend 127.0.0.1:8443 check
        server httpsbackend unix@/var/run/haproxy/httpsocket.sock check

frontend https

        mode http
        option httpslog
        bind unix@/var/run/haproxy/httpsocket.sock user haproxy mode 600 ssl crt /usr/local/share/certificates/ ssl-min-ver TLSv1.2 alpn h2,http/1.1
        #bind *:8443 ssl crt /usr/local/share/certificates/ ssl-min-ver TLSv1.2 alpn h2,http/1.1
        
        option socket-stats  # provide per-bind line stats

        ## set HSTS for one year after all responses 
        # http-after-response set-header Strict-Transport-Security "max-age=31536000"
        http-request redirect scheme https code 301 if !{ ssl_fc }

        # silently ignore connect probes and pre-connect without request
        option http-ignore-probes

        # pass client's IP address to the server and prevent against attempts
        # to inject bad contents
        http-request del-header x-forwarded-for
        option forwardfor

        # enable HTTP compression of text contents
        compression algo deflate gzip
        compression type text/ application/javascript application/xhtml+xml image/x-icon

        # enable HTTP caching of any cacheable content
 #       http-request  cache-use cache
 #       http-response cache-store cache

        use_backend %[req.hdr(host),lower,map(/usr/local/etc/haproxy/maps/backendselect.map)] if { req.hdr(host),lower,map(/usr/local/etc/haproxy/maps/backendselect.map) -m found }
        default_backend web_staging

# backend backend2
# mode tcp
# server server2 192.168.0.101:443 check

# backend backend3
# mode tcp
# server server3 192.168.0.102:443 check

# #SSL Terminated by HAProxy Backends (plain http traffic between HAProxy and these backends)
# backend backend4
# mode http
# server server4 192.168.0.104:80 check
# http-request set-header X-Forwarded-Port %[dst_port]
# http-request add-header X-Forwarded-Proto https if { ssl_fc }

# backend backend5
# mode http
# server server5 192.168.0.105:80 check
# http-request set-header X-Forwarded-Port %[dst_port]
# http-request add-header X-Forwarded-Proto https if { ssl_fc }
#########haproxy.cfg_END#################

backend web_staging
        mode http
        # Algorithm:
        #  - roundrobin is usually better for short requests,
        #  - leastconn is better for mixed slow ones, and long transfers,
        #  - random is generally good when using multiple load balancers
        balance random

        # abort if the client clicks on stop.
        option abortonclose

        # insert a session cookie for user stickiness
        cookie app1 insert indirect nocache

        # check the servers' health using HTTP requests
        option httpchk

        http-check send meth GET uri / ver HTTP/1.1 hdr host staging.someurl.com
        http-request set-header hdr staging.someurl.com

        http-request set-header X-Forwarded-Host %[req.hdr(Host)]
        http-request set-header X-Forwarded-Port 443
        http-request set-header X-Forwarded-Proto https
        #http-request set-header Host %[req.hdr(Host)]
        http-request set-header Host staging.someurl.com

        default-server ssl verify none

        # do not overload the servers (100 concurrent conns max each)
        server staging-windows-webserver 192.168.50.50:443 cookie s1 maxconn 100 check inter 1s
        #server srv2 192.0.2.2:80 cookie s2 maxconn 100 check inter 1s
        #server srv3 192.0.2.3:80 cookie s3 maxconn 100 check inter 1s
        #server srv4 192.0.2.4:80 cookie s4 maxconn 100 check inter 1s

You are chrooting:

chroot /var/run/haproxy/

meaning a frontend bind of:

bind unix@/var/run/haproxy/httpsocket.sock

does exactly what you intend to do, as sockets are bound before chroot()ing.

A server statement of:

server httpsbackend unix@/var/run/haproxy/httpsocket.sock check

actually connects to /var/run/haproxy/var/run/haproxy/httpsocket.sock because you are already in the chroot when you are accessing a backend server. That’s why you’d have to remove the /var/run/haproxy/ prefix from the server statement.

The next thing you likely run into is a permission issue. Same thing applies:

You bind the unix socket as root, but after the bind you downgrade user and group to haproxy, and if the socket is not world readable than you cannot access it. Specify user haproxy group haproxy on the unix socket bind line would fix this.

However, you can avoid both chroot and permission issues if you just use abstract namespace sockets.

frontend ...
 bind abns@httpsocket.sock

backend ...
 server httpsbackend abns@httpsocket.sock

Abstract namespace sockets are not a path and a such are not subject to permission or chroot issues.

1 Like

Thank you so much, for your informative post! I spent most of the day trying to get this to work. I could have guessed the chroot could be an issue. (But I didn’t).

I wouldn’t have known that the 2 paths(bind and server) would actually need to be different, very good to know.
The reason for not using abns is that this is FreeBSD. And abns seems to be a linux only thing. I have a working haproxy with indeed abns and a similar configuration to this one.