HAProxy no responses when built with wolfssl, while working with openssl

Help!
1

I am trying to use haproxy 2.8.5-stable or 2.9.0 with WolfSSL 5.6.4-stable but I am not getting any replies from HAProxy. If I compile haproxy with OpenSSL 3.0.2 or 1.1.1w it works fine.
I also tried this patch:

which is for HAProxy 2.4-dev18 with same results.

I am using ubuntu 22.04 server.

In both cases (openssl, wolfssl) I am hitting HAProxy with:

$ wrk -t12 -c400 -d100s -H"Connection: Close" "https://my-public-reachable-hostname?code=hi"

The only things that differs in the following two scenarios is the SSL shared library the haproxy binary is linked with:

1. With openssl

compiled openssl 1.1.1w, and also used apt’s 3.0.2 same behaviour, haproxy builds and works fine:

$ make clean && make -j $(nproc) TARGET=linux-glibc USE_OPENSSL=1 USE_SYSTEMD=1

$ ldd haproxy
        linux-vdso.so.1 (0x00007ffdacd4f000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x000015213dbf3000)
        libssl.so.1.1 => /usr/local/lib/libssl.so.1.1 (0x000015213db5a000)
        libcrypto.so.1.1 => /usr/local/lib/libcrypto.so.1.1 (0x000015213d86b000)
        libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x000015213d7a4000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000015213d57c000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x000015213d54f000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x000015213d480000)
        liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x000015213d460000)
        libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x000015213d455000)
        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x000015213d317000)
        /lib64/ld-linux-x86-64.so.2 (0x000015213e1b4000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x000015213d2ef000)

$ sudo ./haproxy -V -d -f /etc/haproxy/haproxy.cfg

[NOTICE]   (137738) : haproxy version is 2.8.5-aaba8d0
[NOTICE]   (137738) : path to executable is ./haproxy
[WARNING]  (137738) : config : Proxy 'pub-https': no-sslv3/no-tlsv1x are ignored for bind 'HAPROXY_PUBLIC_REACHABLE_IP:443' at [/etc/haproxy/haproxy.cfg:77]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
[WARNING]  (137738) : config : Proxy 'int-https': no-sslv3/no-tlsv1x are ignored for bind 'HAPROXY_PRIVATE_REACHABLE_IP:443' at [/etc/haproxy/haproxy.cfg:119]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Note: setting global.maxconn to 799929.
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.

Available filters :
	[BWLIM] bwlim-in
	[BWLIM] bwlim-out
	[CACHE] cache
	[COMP] compression
	[FCGI] fcgi-app
	[SPOE] spoe
	[TRACE] trace
Using epoll() as the polling mechanism.
00000000:int-https.accept(000b)=00b2 from [IP_OF_WRK:50664] ALPN=<none>
00000000:int-https.clireq[00b2:ffffffff]: GET / HTTP/1.1
00000000:int-https.clihdr[00b2:ffffffff]: host: my_reachable_hostname
00000000:abc_c39_session_id.srvrep[00b2:0115]: HTTP/1.1 200 OK
00000000:abc_c39_session_id.srvhdr[00b2:0115]: content-type: text/plain
00000000:abc_c39_session_id.srvhdr[00b2:0115]: content-length: 0
00000000:abc_c39_session_id.srvhdr[00b2:0115]: date: Wed, 13 Dec 2023 17:27:14 GMT
00000000:abc_c39_session_id.srvcls[00b2:0115]
00000000:abc_c39_session_id.clicls[00b2:0115]
00000000:abc_c39_session_id.closed[00b2:0115]
...

getting responses to client (wrk)

$ make clean && sudo make uninstall

2. with WolfSSL

Downloaded wolfssl-5.6.4-stable from:
https://github.com/wolfSSL/wolfssl/archive/refs/tags/v5.6.4-stable.tar.gz

$ ./autogen.sh

$ ./configure --enable-haproxy

also tried:

$ ./configure --enable-opensslextra --enable-haproxy

$ make -j $(nproc)

no errors during building, but the following check reports one:


$ make check  

...
SKIP: scripts/openssl.test
SKIP: scripts/external.test
PASS: scripts/trusted_peer.test
PASS: scripts/resume.test
PASS: scripts/google.test
PASS: scripts/tls13.test
PASS: scripts/crl-revoked.test
PASS: scripts/ocsp.test
PASS: scripts/pem.test
PASS: scripts/ocsp-stapling.test
PASS: scripts/ocsp-stapling-with-ca-as-responder.test
PASS: scripts/ocsp-stapling2.test
PASS: testsuite/testsuite.test
FAIL: scripts/unit.test
============================================================================
Testsuite summary for wolfssl 5.6.4
============================================================================
# TOTAL: 14
# PASS:  11
# SKIP:  2
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
...

$ sudo make install

*** Back in haproxy 2.8.5 (clean dir) ***


$ make clean && make -j $(nproc) TARGET=linux-glibc USE_OPENSSL_WOLFSSL=1 USE_SYSTEMD=1 SSL_INC=/usr/local/include/wolfssl SSL_LIB=/usr/local/lib

$ sudo make install

$ ldd haproxy 
        linux-vdso.so.1 (0x00007fffa8dd0000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x000014ddf962a000)
        libwolfssl.so.41 => /usr/local/lib/libwolfssl.so.41 (0x000014ddf931a000)
        libsystemd.so.0 => /lib/x86_64-linux-gnu/libsystemd.so.0 (0x000014ddf9253000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014ddf902b000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x000014ddf8f44000)
        /lib64/ld-linux-x86-64.so.2 (0x000014ddf9bea000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x000014ddf8f17000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x000014ddf8e48000)
        liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x000014ddf8e28000)
        libcap.so.2 => /lib/x86_64-linux-gnu/libcap.so.2 (0x000014ddf8e1d000)
        libgcrypt.so.20 => /lib/x86_64-linux-gnu/libgcrypt.so.20 (0x000014ddf8cdf000)
        libgpg-error.so.0 => /lib/x86_64-linux-gnu/libgpg-error.so.0 (0x000014ddf8cb7000)

Now I get no HTTP_responses/output:

$ sudo ./haproxy -V -d -f /etc/haproxy/haproxy.cfg

[NOTICE]   (155142) : haproxy version is 2.8.5-aaba8d0
[NOTICE]   (155142) : path to executable is ./haproxy
[WARNING]  (155142) : config : Proxy 'pub-https': no-sslv3/no-tlsv1x are ignored for bind 'haproxy_public_ipv6_here:443' at [/etc/haproxy/haproxy.cfg:77]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
[WARNING]  (155142) : config : Proxy 'int-https': no-sslv3/no-tlsv1x are ignored for bind 'haproxy_private_ip_here:443' at [/etc/haproxy/haproxy.cfg:119]. Use only 'ssl-min-ver' and 'ssl-max-ver' to fix.
Note: setting global.maxconn to 799929.
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Using epoll() as the polling mechanism.

… no more output here … stats page also empty

system info

uname -a
Linux myhostname 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2 15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

/etc/haproxy/haproxy.cfg

global
	daemon
        quiet

	nbthread 48
        cpu-map auto:1/1-48 24-47,48-71
#	cpu-map auto:1/1-48 0-47

	user haproxy
	group haproxy
        chroot /var/lib/haproxy

	# these were commented out so that it starts with wolfssl too:
	
        # tune.ssl.cachesize 100000000    
        # tune.ssl.cachesize 20000000
        # tune.ssl.ssl-ctx-cache-size 100000
        # tune.ssl.lifetime 3000s

	log /dev/log	local0
	log /dev/log	local1 notice
        no log

	# these were commented out so that it starts with wolfssl too:
	
        # ssl-default-bind-options ssl-min-ver TLSv1.0 no-sslv3 no-tls-tickets
        # ssl-default-bind-ciphers 6:kEDH+AESGCM:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:@SECLEVEL=0
        # ssl-default-server-options ssl-min-ver TLSv1.0 no-sslv3 no-tls-tickets
        # ssl-default-server-ciphers ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES256-GCM-SHA384:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA


defaults
        no log
	mode	http
	option	dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        option redispatch
        retries 3
        maxconn 300000
#        bind-process 1

listen stats
        bind 0.0.0.0:8888
        stats enable
        stats uri /haproxy?stats
        stats auth usr:pass
        stats refresh 5s
#	bind-process 1-64
#        acl network_allowed src 10.0.0.0/16
#        tcp-request connection reject if !network_allowed


frontend pub-http
#        bind-process 1-64
	bind haproxy_public_ip_here:80
        bind haproxy_public_ipv6_here:80 v6only

        mode http
        option forwardfor
        http-request add-header X-Forwarded-Proto https
        maxconn 6000000

        timeout http-keep-alive 25000ms
        timeout client 25000ms
        timeout http-request 16000ms
        option http-ignore-probes

        acl abc urlp(sessionRoot) -m found
        acl def      urlp(code)        -m found
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc1.map,bk_default)] if abc
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc2.map,bk_default)] if def
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc3.map,bk_default)] if !abc !def


frontend pub-https
        # bind-process 1-64
        bind haproxy_public_ip_here:443
        bind haproxy_public_ipv6_here:443 v6only ssl no-sslv3 crt /etc/haproxy/certs/ ssl-min-ver TLSv1.0 alpn h2,http/1.1
        mode http
        option forwardfor
        http-request add-header X-Forwarded-Proto https
        maxconn 6000000

        timeout http-keep-alive 25000ms
        timeout client 25000ms
        timeout http-request 16000ms
        option http-ignore-probes

        acl abc urlp(sessionRoot) -m found
        acl def      urlp(code)        -m found
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc1.map,bk_default)] if abc
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc2.map,bk_default)] if def
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc3.map,bk_default)] if !abc !def


frontend int-http
        # bind-process 1-64
        bind haproxy_private_ip_here:80

        mode http
        option forwardfor
        http-request add-header X-Forwarded-Proto https
        maxconn 6000000

        #timeout http-keep-alive 25000ms
        #timeout client 25000ms
        timeout client 8000ms
        #timeout http-request 16000ms
        #option http-ignore-probes

        acl abc urlp(sessionRoot) -m found
        acl def      urlp(code)        -m found
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc1.map,bk_default)] if abc
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc2.map,bk_default)] if def
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc3.map,bk_default)] if !abc !def


frontend int-https
        # bind-process 1-64
        bind haproxy_private_ip_here:443 ssl no-sslv3 crt /etc/haproxy/certs/ ssl-min-ver TLSv1.0 alpn h2,http/1.1

        mode http
        option forwardfor
        http-request add-header X-Forwarded-Proto https
        maxconn 6000000

        #timeout http-keep-alive 25000ms
        #timeout client 25000ms
        timeout client 8000ms
        #timeout http-request 16000ms
        #option http-ignore-probes

        acl hname urlp(sessionRoot) -m found
        acl def      urlp(code)        -m found
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc1.map,bk_default)] if abc
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc2.map,bk_default)] if def
        use_backend %[req.hdr(host),lower,map_dom(/etc/haproxy/abc3.map,bk_default)] if !abc !def

...
backends configuration following ...
...

Any ideas how to troubleshoot this?
Thank you all

2

The install instructions are in the INSTALL file:

You need to follow the install instruction exactly, line for line, word for word.

Wolfssl remains a experimental feature so YMMV.

3

Hello @lukastribus

Thank you for your prompt reply!

These are the instructions I followed, indeed. I went through every iteration and posted the best point I managed to reach.
When I follow them word-by-word:

  $ cd ~/build/wolfssl
  $ ./configure --enable-haproxy --enable-quic --prefix=/opt/wolfssl-5.6.4/
  $ make -j $(nproc)
  $ make install   # well this needs sudo in a default ubuntu installation

  $ cd ~/build/haproxy
  $ make -j $(nproc) TARGET=generic USE_OPENSSL_WOLFSSL=1 USE_QUIC=1 \
    SSL_INC=/opt/wolfssl-5.6.4/include SSL_LIB=/opt/wolfssl-5.6.4/lib

it’s even worse because the haproxy binary doesn’t even get the wolfssl object file linked:

$ ldd haproxy 
        linux-vdso.so.1 (0x00007ffdb07f5000)
        libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x000014e54f079000)
        libwolfssl.so.41 => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x000014e54ee51000)
        /lib64/ld-linux-x86-64.so.2 (0x000014e54f6a9000)

And It’s not a permission issue, I checked

4

when hitting the http frontend, it works fine.

If I use openssl client to connect to the https frontend it just returns:

$ openssl s_client -connect myhostname:443
CONNECTED(00000003)

which seems that the TCP handshake succeeds but the TLS fails.

I also tried to use a single certificate in the bind line:

bind 10.0.27.8:443 ssl crt /etc/haproxy/certs/myhostname.pem ssl-min-ver TLSv1.0 alpn h2,http/1.1

but the TLS handshake will not go forward, and the weird thing is I am not getting any output from haproxy despite being ran like this:

$ sudo ./haproxy -d -V -f /etc/haproxy/haproxy.cfg

Note: setting global.maxconn to 799929.
Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result FAILED
Total: 3 (2 usable), will use epoll.

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Using epoll() as the polling mechanism.

Is there a way to get more verbose logging?

Thanks

5

you have configured wolfssl with prefix

/opt/wolfssl-5.6.4/

in the configure of haproxy you should link to /opt/wolfssl-5.6.4 as include and lib dir

6

Hi @mrit

Thanks for your reply.
You are seeing different iterations above. Please check the initial post instead. I have installed wolfssl to many locations in the various iterations.

Every time I compile haproxy I link the correct locations with SSL_INC and SSL_LIB flags and I then check with ldd and ./haproxy -vv. It just happened the one time I used /opt/, it didn’t link correctly, which is just another issue, different from the one I am facing here.

Following the exact same procedure changing the paths for OpenSSL and AWS-LC, works fine.

Haproxy is compiled and linked with wolfssl correctly (with no errors reported). It just fails to respond to TLS handshake W/O LOGS.

7

During wolfssl compilation on a fresh Ubuntu 22.04 we can see:

Libraries have been installed in:
   /opt/wolfssl-5.6.4/lib

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the '-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable
     during execution
   - add LIBDIR to the 'LD_RUN_PATH' environment variable
     during linking
   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag
   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.

As such, we need to add LD_LIBRARY_PATH:

root@ubuntu-2gb-hel1-1:~/haproxy-2.8# ./haproxy -vv
./haproxy: error while loading shared libraries: libwolfssl.so.41: cannot open shared object file: No such file or directory
root@ubuntu-2gb-hel1-1:~/haproxy-2.8# ldd ./haproxy
        linux-vdso.so.1 (0x00007ffff8749000)
        libwolfssl.so.41 => not found
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f080aaad000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f080b02e000)
root@ubuntu-2gb-hel1-1:~/haproxy-2.8#
root@ubuntu-2gb-hel1-1:~/haproxy-2.8# LD_LIBRARY_PATH=/opt/wolfssl-5.6.4/lib ldd haproxy
        linux-vdso.so.1 (0x00007ffd621ed000)
        libwolfssl.so.41 => /opt/wolfssl-5.6.4/lib/libwolfssl.so.41 (0x00007fcad83e6000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcad81b8000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcad80d1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fcad8a53000)
root@ubuntu-2gb-hel1-1:~/haproxy-2.8# LD_LIBRARY_PATH=/opt/wolfssl-5.6.4/lib ./haproxy -vv
HAProxy version 2.8.5 2023/12/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.5.html
Running on: Linux 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64
Build options :
  TARGET  = generic
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_OPENSSL_WOLFSSL=1 USE_QUIC=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES -ACCEPT4 -BACKTRACE -CLOSEFROM -CPU_AFFINITY -CRYPT_H -DEVICEATLAS -DL -ENGINE -EPOLL -EVPORTS -GETADDRINFO -KQUEUE -LIBATOMIC -LIBCRYPT -LINUX_CAP -LINUX_SPLICE -LINUX_TPROXY -LUA -MATH -MEMORY_PROFILING -NETFILTER -NS -OBSOLETE_LINKER +OPENSSL +OPENSSL_WOLFSSL -OT -PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL -PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION +QUIC -QUIC_OPENSSL_COMPAT -RT -SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD -TFO -THREAD -THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : wolfSSL 5.6.4
Running on OpenSSL version : wolfSSL 5.6.4
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built without multi-threading support (USE_THREAD not set).
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built without PCRE or PCRE2 support (using libc's regex instead)
Encrypted password support via crypt(3): no
Built with gcc compiler version 11.4.0

Available polling systems :
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 2 (2 usable), will use poll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
       quic : mode=HTTP  side=FE     mux=QUIC  flags=HTX|NO_UPG|FRAMED
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : none

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

root@ubuntu-2gb-hel1-1:~/haproxy-2.8#

A quick test with SSL configuration in the frontend (classic SSL termination example) on the haproxy server reveals that everything works just fine.

defaults
 mode http
 timeout client 10s
 timeout server 10s
 timeout connect 10s

global
 maxconn 10

frontend a
bind :443 ssl crt /root/openssl-cert/combined.pem
default_backend b

backend b
 http-request return status 200 content-type "text/plain" string "ok"

Running haproxy:

root@ubuntu-2gb-hel1-1:~/haproxy-2.8# LD_LIBRARY_PATH=/opt/wolfssl-5.6.4/lib ./haproxy -f /root/openssl-cert/haproxy.cfg -d
Available polling systems :
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 2 (2 usable), will use poll.

Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace
Using poll() as the polling mechanism.
00000000:a.accept(0003)=0006 from [127.0.0.1:54620] ALPN=h2
00000000:a.clireq[0006:ffffffff]: GET https://localhost/ HTTP/2.0
00000000:a.clihdr[0006:ffffffff]: host: localhost
00000000:a.clihdr[0006:ffffffff]: user-agent: curl/7.81.0
00000000:a.clihdr[0006:ffffffff]: accept: */*
00000000:b.clicls[0006:ffff]
00000000:b.closed[0006:ffff]
00000001:a.accept(0003)=0006 from [127.0.0.1:59182] ALPN=h2
00000001:a.clireq[0006:ffffffff]: GET https://localhost/ HTTP/2.0
00000001:a.clihdr[0006:ffffffff]: host: localhost
00000001:a.clihdr[0006:ffffffff]: user-agent: curl/7.81.0
00000001:a.clihdr[0006:ffffffff]: accept: */*
00000001:b.clicls[0006:ffff]
00000001:b.closed[0006:ffff]
^C

Running a client against it:

root@ubuntu-2gb-hel1-1:~# curl -vvvk https://localhost
*   Trying 127.0.0.1:443...
* Connected to localhost (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=NG; ST=Rivers; L=PHC; O=Mono Finance; OU=Finance; CN=*.monofinance.net; emailAddress=mrikehchukwuka@gmail.com
*  start date: Dec 18 16:05:24 2023 GMT
*  expire date: Feb 16 16:05:24 2024 GMT
*  issuer: C=NG; ST=Rivers; L=Choba; O=Mono Institution; OU=Finance; CN=*.monoinstitute.net; emailAddress=monoinstitute@gmail.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55ca66d12e90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: localhost
> user-agent: curl/7.81.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 200
< content-length: 2
< content-type: text/plain
<
* Connection #0 to host localhost left intact
okroot@ubuntu-2gb-hel1-1:~#
root@ubuntu-2gb-hel1-1:~#
root@ubuntu-2gb-hel1-1:~# curl -k https://localhost
okroot@ubuntu-2gb-hel1-1:~#
8

And alternative approach as per the wolfssl message: compile haproxy with LD_RUN_PATH=/opt/wolfssl-5.6.4/lib:

make clean; LD_RUN_PATH=/opt/wolfssl-5.6.4/lib make -j $(nproc) TARGET=generic USE_OPENSSL_WOLFSSL=1 USE_QUIC=1     SSL_INC=/opt/wolfssl-5.6.4/include SSL_LIB=/opt/wolfssl-5.6.4/lib
9

If the object file is not linked correctly haproxy will emit a corresponding error upon execution

10

The problem turned out to be the

chroot /var/lib/haproxy

line… No idea how is this relevant with SLL

11

I noticed Willy mentioned on another issue that openssl only opens /dev/urandom on first crypto use, and so there is a need to perform a dummy call to force it to initialize before chrooting. I wonder if something similar might be the case with wolf?

FWIW I got the same issue (CONNECTED(00000003), no other output until the chroot is removed) with either a dynamic or static linked wolfSSL 5.6.6 with HAProxy 2.9.2, with wolf built with the same arguments as you ./configure --prefix=/tmp/wolfssl --enable-haproxy --enable-quic (I use a /tmp dir as I was compiling on a different machine to where I was running HAProxy so need to ship the .so out and set the LD_LIBRARY_PATH on the run machine, which is why I tried static compilation first). Static linking to OpenSSL removes the issue, as does removing the chroot.

If the only answer to this is removing the chroot, then it seems to me the issue should not be marked as solved. I am happy to provide more information if anyone wants to investigate further, there is obviously some difference between our setup and Lukas one.

12

There are two different issues discussed in this thread.
One is a compilation issue and the other one is the chroot issue.

The chroot issue is definitely something that needs further diagnosis. Did anyone of your noticed increased CPU usage when wolfssl with chroot enabled faces a hanging TLS connection?

There’s a bug report about this here, but we don’t know yet if this has the same root cause as the chroot thing:

13

Hi Lukas,

Only had time just then to do a quick test, under a test load of about 50 SSL requests/sec and around 500 non SSL/s, the baseline usage for that is about 25% top reported CPU of a modern 8 core machine.

That is running wolfSSL with no chroot jail, so I added the jail and restarted. Requests started dropping, and I observed that an openssl s_client -connect would just hang, so I think that is the issue we are talking about.

Sorry to bury the lede, but the CPU usage dropped down to around 4-5% while it was in this state. I observed for around a minute that the requests definitely seem to all have hung, and then I put it back.
If you can point me towards any theories or tests I can run to help figure this out, that would be much appreciated.

15

Confirmed this is about /dev/[u]random access:

gettimeofday({tv_sec=1706642203, tv_usec=117454}, NULL) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=20852494}) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=3981304, tv_nsec=26693841}) = 0
accept(4, {sa_family=AF_INET, sin_port=htons(34994), sin_addr=inet_addr("127.0.0.1")}, [128->16]) = 7
fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
brk(0x55905ba7e000)     = 0x55905ba7e000
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/random", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/urandom", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/dev/random", O_RDONLY) = -1 ENOENT (No such file or directory)

But there is really nothing for haproxy to do here. OpenSSL at some point did something more dangerous in this case, which was reading random data initially and then silently breaking SSL later, when more random data was required (haproxy broke, but only after running for half an hour without any issues).

You can mount random/urandom within your chroot to solve this; but really a modern libssl should use getrandom() as opposed to accessing random files.

To do the latter you need to enable it in wolfssl manually (adding EXTRA_CFLAGS=-DWOLFSSL_GETRANDOM=1 as configure argument):

$ ./configure --enable-haproxy --enable-quic --prefix=/opt/wolfssl-5.6.6/ EXTRA_CFLAGS=-DWOLFSSL_GETRANDOM=1

And indeed it works fine in chroot by no longer accessing the files but calling getrandom() instead:

gettimeofday({tv_sec=1706642846, tv_usec=9373}, NULL) = 0
clock_gettime(CLOCK_THREAD_CPUTIME_ID, {tv_sec=0, tv_nsec=28593327}) = 0
clock_gettime(CLOCK_MONOTONIC, {tv_sec=3981946, tv_nsec=918110946}) = 0
accept(4, {sa_family=AF_INET, sin_port=htons(34998), sin_addr=inet_addr("127.0.0.1")}, [128->16]) = 7
fcntl(7, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
brk(0x55e7c3ea1000)     = 0x55e7c3ea1000
getrandom("\x5c\x1a\x6b\x23\x2d\xe8\x4b\x0b\x79\x9c\x3f\x46\x55\xda\xe3\xc1\xb8\x96\x48\x29\xe1\x79\x33\xf8\xdc\xd3\xb5\x14\x2b\x9d\x5d\x93"..., 52, 0) = 52
16

Thanks for the speedy response Lukas, I can reproduce what you see in strace with and without that macro, and that fixes the issue for me as well.

Given that chroot is recommended, it would be good to have a note in the HAProxy wolfSSL compilation instructions about adding this macro (or disabling chroot for those few without kernel 3.17+ and glibc 2.25+). I understand wolf has a lot of options and perhaps not everyone needs the random calls, but it seems the default might bite enough people abandoning the OpenSSL ship in the future to be worth mentioning?

17

Absolutely, I already sent a DOC patch to the mailing list:

https://www.mail-archive.com/haproxy@formilux.org/msg44543.html

18

And alternative could be to mount the random devices into the chroot.

mknod -m 444 /path/to/chrootdir/dev/random c 1 8
mknod -m 444 /path/to/chrootdir/dev/urandom c 1 9

DOC change committed: