HAProxy community

HAProxy caching keys/response from HSM


#1

Hi,
I am trying to use openssl engine “xyz” to store the private keys inside HSM and using reference of private key along with certicate in pem file.
Below is my haproxy.cfg file:
global
log 127.0.0.1 local2

chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon
ssl-engine xyz
tune.ssl.default-dh-param 2048
tune.ssl.cachesize 400000
tune.ssl.lifetime 60
tune.bufsize 32768

defaults
mode http
log global
option httplog
option dontlognull
retries 3
timeout http-request 10s
timeout queue 10s
timeout connect 10s
timeout client 10s
timeout server 10s
timeout http-keep-alive 10s
timeout check 5m

frontend https-in
bind *:80
use_backend static
option forwardfor
bind *:443 ssl crt /etc/pki/tls/certs/haproxy2.pem

backend static
balance roundrobin
server node1 10.164.115.79:80 check
server node2 10.164.76.117:80 check

and output of haproxy -vv is:

[root@noi-9y0fk72 bin]# /opt/rh/rh-haproxy18/root/usr/sbin/haproxy -vv
HA-Proxy version 1.8.4-1deb90d 2018/02/08
Copyright 2000-2018 Willy Tarreau ‘willy@haproxy.org’

Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1

Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with network namespace support.

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace

[root@noi-9y0fk72 bin]#

Output of openssl s_client -connect localhost:443 is:
[root@noi-9y0fk72 bin]# openssl s_client -connect localhost:443
Retrieving passphrase for slot id 0x1… success.
CONNECTED(00000004)
depth=0 C = IN, ST = UP, L = Noida, O = XYZ, OU = HSM, CN = wwwhaproxyhsmcom, emailAddress = abc@kem.ino
verify error:num=18:self signed certificate
verify return:1
depth=0 C = IN, ST = UP, L = Noida, O =XYZ, OU = HSM, CN = wwwhaproxyhsmcom, emailAddress = abc@kem.ino
verify return:1

Certificate chain
0 s:/C=IN/ST=UP/L=Noida/O=XYZ/OU=HSM/CN=wwwhaproxyhsmcom/emailAddress=abc@kem.ino
i:/C=IN/ST=UP/L=Noida/O=XYZ/OU=HSM/CN=wwwhaproxyhsmcom/emailAddress=abc@kem.ino

Server certificate
-----BEGIN CERTIFICATE-----
MIID2zCCAsOgAwIBAgIJAO+FMrkIK1k/MA0GCSqGSIb3DQEBCwUAMIGDMQswCQYD
VQQGEwJJTjELMAkGA1UECAwCVVAxDjAMBgNVBAcMBU5vaWRhMRAwDgYDVQQKDAdT
YWZlbmV0MQwwCgYDVQQLDANIU00xGzAZBgNVBAMMEnd3dy5oYXByb3h5aHNtLmNv
bTEaMBgGCSqGSIb3DQEJARYLYWJjQGtlbS5pbm8wHhcNMTgxMTE5MDgxMjAxWhcN
MTkxMTE5MDgxMjAxWjCBgzELMAkGA1UEBhMCSU4xCzAJBgNVBAgMAlVQMQ4wDAYD
VQQHDAVOb2lkYTEQMA4GA1UECgwHU2FmZW5ldDEMMAoGA1UECwwDSFNNMRswGQYD
VQQDDBJ3d3cuaGFwcm94eWhzbS5jb20xGjAYBgkqhkiG9w0BCQEWC2FiY0BrZW0u
aW5vMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtmLqlzlEhuED9uia
wBE1UYgqg1PMZeCRZ/ud30HhSM8nKeKbJzdoZxpH2ifo4ViV9ftOdij+Yfol1akz
5QHyQazu+Rp6R8YvlAytQNHceZBUiA+TmCaTS+2VE4BdI5qhShNhICkI8+lSocfn
ry0z9tZvOxnz8R01JiaR2CGdKowvYIJHOU09PWeuAdCNKf20NzBT0fdbmrZzgaaR
c5OInROQlTSR2d3Da0usQtipz0rOe8PtpXdBNpb3U0LqC14vMmwPD4r2YgNzcD6z
xB1XyhnXy4WyDx8uRQYhkiREIOd5eBgnoqgzUdg2kSKo3XVLbgMpcUbYZTvgtZxB
KNO1xQIDAQABo1AwTjAdBgNVHQ4EFgQUHTYDdXwyhz7lxyDOhzNCP+w05GcwHwYD
VR0jBBgwFoAUHTYDdXwyhz7lxyDOhzNCP+w05GcwDAYDVR0TBAUwAwEB/zANBgkq
hkiG9w0BAQsFAAOCAQEAFMyz4bNTOCUn0Ehnub6OKul0KgnG1RINMZMUNurFuAe9
jD0Fadm+PsXrdpg7+f21eByR18WTcNsC5YAqjkYKMyF6JsP6lYlFnYFLbZAdvSEY
sJP+PSRnoW3t1wOWNDWefvk5Y/KOPN+dst/VyYrojD3y8VekkK+7k/XNUWBixZ9m
80FKd8xvrxPx/Gxv1Q9PTSuDSuNQUMGpwz9VpDdlNAastEGiKbFySPh8BD9i1T6z
fDw19m2OWWGqQy92SUg4fs7HoQZ7l6dwdkX5ShZyv3wz3AdkA4kwcSM7THU5vW0C
Fbn5BY1N8m994U/2/+jvX6rUwAQsncm/nDKK71mD4g==
-----END CERTIFICATE-----
subject=/C=IN/ST=UP/L=Noida/O=XYZ/OU=HSM/CN=wwwhaproxyhsmcom/emailAddress=abc@kem.ino
issuer=/C=IN/ST=UP/L=Noida/O=XYZ/OU=HSM/CN=wwwhaproxyhsmcom/emailAddress=abc@kem.ino

No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 1650 bytes and written 415 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: FDEE9908F72E6E40150CCBAA7430AC3B330913D2F644EFE70FA71E6E7D1EE5AD
Session-ID-ctx:
Master-Key: 769F732987326B2F1A69CC5426ADDF2C10ABA395AA64A089967304E495C02379DE21335A661B5FC59C8770ED08EF47B4
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 60 (seconds)
TLS session ticket:
0000 - db 27 4f 54 ac bf e3 db-a5 12 53 a8 d3 fd bf 5f .‘OT…S…_
0010 - e2 04 c4 9e 82 f1 c5 b3-65 ae b2 87 dd 4e 2b 8f …e…N+.
0020 - b5 a8 3a 06 9c 4f b6 15-94 a3 8a 48 cf ea 74 7f …:…O…H…t.
0030 - 7c e7 6f 66 8d 6a 18 3f-3e 2b 7c b3 21 84 b9 6e |.of.j.?>+|.!..n
0040 - e5 fd 36 47 63 6a 9d 03-89 1d fd 6a fd 92 30 4a …6Gcj…j…0J
0050 - c0 79 26 78 a2 3e d0 e3-1e 71 20 06 64 af 90 21 .y&x.>…q .d…!
0060 - a7 6c ae cd 9c c1 42 5b-6c b4 22 fb de 3e a7 4b .l…B[l."…>.K
0070 - f0 00 3b 71 28 1b 76 6d-89 23 7f 44 6d eb a0 b9 …;q(.vm.#.Dm…
0080 - 58 42 1a c6 58 e3 b2 dc-1e 84 de e4 c3 58 89 92 XB…X…X…
0090 - 6f 98 3f 95 0f 20 6b 5b-6c 28 27 8a 86 7f 9d c5 o.?.. k[l(’…

Start Time: 1543474928
Timeout   : 300 (sec)
Verify return code: 18 (self signed certificate)

HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<.html><.body><.h1>408 Request Time-out<./h1>
Your browser didn’t send a complete request in time.
<./body><./html>
closed

I am able to start the service successfully.When i am starting the service it is going to the HSM for private key with private key reference and service is up with no error.
Logs:
[root@noi-9y0fk72 bin]# systemctl status -l rh-haproxy18-haproxy.service
● rh-haproxy18-haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/rh-haproxy18-haproxy.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2018-11-29 12:31:54 IST; 17min ago
Process: 29210 ExecStartPre=/opt/rh/rh-haproxy18/root/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
Main PID: 29212 (haproxy)
Tasks: 2
CGroup: /system.slice/rh-haproxy18-haproxy.service
├─29212 /opt/rh/rh-haproxy18/root/usr/sbin/haproxy -Ws -f /etc/opt/rh/rh-haproxy18/haproxy/haproxy.cfg -p /run/rh-haproxy18-haproxy.pid
└─29214 /opt/rh/rh-haproxy18/root/usr/sbin/haproxy -Ws -f /etc/opt/rh/rh-haproxy18/haproxy/haproxy.cfg -p /run/rh-haproxy18-haproxy.pid**

Nov 29 12:31:54 noi-9y0fk72 systemd[1]: Starting HAProxy Load Balancer…
Nov 29 12:31:54 noi-9y0fk72 haproxy[29210]: Retrieving passphrase for slot id 0x1… success.
Nov 29 12:31:54 noi-9y0fk72 systemd[1]: Started HAProxy Load Balancer.
Nov 29 12:31:54 noi-9y0fk72 haproxy[29212]: Retrieving passphrase for slot id 0x1… success.
[root@noi-9y0fk72 bin]#

But the problem is that while opening withhttps://HAProxyIp ,it is not going to HSM for private key.
So it seems that HAProxy is somewhere storing the retrieved private key/response from HSM at the time of starting the service in its own cache.Even if i am deleteing the keys from HSM it is still able to open https://HAProxyIp on https without any error.

So my questions are:

  1. is there any way i can force HAProxy to go to HSM for private key each time i hit https://HAProxyIp??
    2.If not, then can i set some time like after every 5 minutes it clears the key/response from cache and go to HSM for private key and update itself?

#2

I have no idea what the expected behavior is, the code in haproxy is probably agnostic to this.

Does this work as per your expectation in a test server by using only the “openssl” command itself?


#3

Yes,with openssl it is working as expected


#4

Can you please verify the configuration sent above?Any suggestions to make configuration more robust are welcome.


#5

I don’t think this is something that is actually supported, or whether or not we can support it.

There is a patch from 3 years ago on the mailing list, which probably does not work with current code anymore:

https://www.mail-archive.com/haproxy@formilux.org/msg18806.html

I would suggest you use the mailing list, maybe someone there will be able to explain the current situation with this in haproxy. I have doubts about this being supported though, and the patch mentioned earlier has not been applied to mainline haproxy.


#6

Thank you for your suggestion.
Do HAProxy Enterprise edition supports PKCS#11 for cryptographic keys storage?
If yes then i will go for it.
If no,then HAProxy developers should add the support so that the connection would be more secured and unhackable.Other competitors to HAProxy such as NGINX do have support for PKCS#11.
Hoping the same to be added in HAProxy.


#7

I simply don’t know. I’m not affiliated with the company behind haproxy which handles the enterprise edition in any way.

I suggest:

  • contact the mailing list at haproxy@formilux.org regarding this issue (this is for the open source project), also make sure to mention the patch on the mailing list from 2015 (here: https://www.mail-archive.com/haproxy@formilux.org/msg18806.html)
  • contact the company regarding the Enterprise Edition