Openssl 3.x speed issues

Hi,

We’re having issues upgrading from ubuntu 20.04 to ubuntu 22.04.

After test different haproxy versions (the one shipped by ubuntu, 2.4, and 2.7.3) and different openssl versions (the one shipped by ubuntu, 3.0.2, and 3.0.7) we always see blocking issues in the TLS handsake under high load.

Same configuration, ubuntu 20.04 with OpenSSL 1.1.1, no problem at all.

¿any hints?

Unfortunately there is no solution, other than to downgrade. Openssl 3.0 is extremely problematic for multi-threading loads, this has nothing to do with haproxy but with changes within openssl 3.0 itself, please see:

As well as:

https://www.mail-archive.com/haproxy@formilux.org/msg42992.html

Yes, exactly what we saw.

If we were about to test the today release of openssl, which introduces locking/performance ([0]) improvements… do we need to compile haproxy against the new openssl? or it’s enough if we substitute the operative system openssl and libssl? (haproxy -vv shows it’s compiled against openssl 3.0.2)

[0] OpenSSL 3.1 Final Release - OpenSSL Blog

Replacing a system openssl library is a bad idea as it could break other applications on the system.

I suggest you build haproxy with a static openssl build, as per 4.5) Cryptography in the INSTALL file in the tarball.

Hi, out of curiosity, do you load any directories that contain many SSL certificates? If so, are any of those certificates expired? We had a ton of issues similar to what you are describing, and we stumbled onto something that helped… deleting the expired certificates from the folder, and then restarting HAproxy seemed to fix the issues for us! I have no idea why this would would work.

we are actually going to just move to debian11. Don’t want to deal with openssl 3.0 issues until it’s more mature.