Hi,
I am trying to use openssl engine “xyz” to store the private keys inside HSM and using reference of private key along with certicate in pem file.
Below is my haproxy.cfg file:
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
ssl-engine xyz
tune.ssl.default-dh-param 2048
tune.ssl.cachesize 400000
tune.ssl.lifetime 60
tune.bufsize 32768
defaults
mode http
log global
option httplog
option dontlognull
retries 3
timeout http-request 10s
timeout queue 10s
timeout connect 10s
timeout client 10s
timeout server 10s
timeout http-keep-alive 10s
timeout check 5m
frontend https-in
bind *:80
use_backend static
option forwardfor
bind *:443 ssl crt /etc/pki/tls/certs/haproxy2.pem
backend static
balance roundrobin
server node1 10.164.115.79:80 check
server node2 10.164.76.117:80 check
and output of haproxy -vv is:
[root@noi-9y0fk72 bin]# /opt/rh/rh-haproxy18/root/usr/sbin/haproxy -vv
HA-Proxy version 1.8.4-1deb90d 2018/02/08
Copyright 2000-2018 Willy Tarreau ‘willy@haproxy.org’
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_SYSTEMD=1 USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
[root@noi-9y0fk72 bin]#
Output of openssl s_client -connect localhost:443 is:
[root@noi-9y0fk72 bin]# openssl s_client -connect localhost:443
Retrieving passphrase for slot id 0x1… success.
CONNECTED(00000004)
depth=0 C = IN, ST = UP, L = Noida, O = XYZ, OU = HSM, CN = wwwhaproxyhsmcom, emailAddress = abc@kem.ino
verify error:num=18:self signed certificate
verify return:1
depth=0 C = IN, ST = UP, L = Noida, O =XYZ, OU = HSM, CN = wwwhaproxyhsmcom, emailAddress = abc@kem.ino
verify return:1
Certificate chain
0 s:/C=IN/ST=UP/L=Noida/O=XYZ/OU=HSM/CN=wwwhaproxyhsmcom/emailAddress=abc@kem.ino
i:/C=IN/ST=UP/L=Noida/O=XYZ/OU=HSM/CN=wwwhaproxyhsmcom/emailAddress=abc@kem.ino
Server certificate
-----BEGIN CERTIFICATE-----
MIID2zCCAsOgAwIBAgIJAO+FMrkIK1k/MA0GCSqGSIb3DQEBCwUAMIGDMQswCQYD
VQQGEwJJTjELMAkGA1UECAwCVVAxDjAMBgNVBAcMBU5vaWRhMRAwDgYDVQQKDAdT
YWZlbmV0MQwwCgYDVQQLDANIU00xGzAZBgNVBAMMEnd3dy5oYXByb3h5aHNtLmNv
bTEaMBgGCSqGSIb3DQEJARYLYWJjQGtlbS5pbm8wHhcNMTgxMTE5MDgxMjAxWhcN
MTkxMTE5MDgxMjAxWjCBgzELMAkGA1UEBhMCSU4xCzAJBgNVBAgMAlVQMQ4wDAYD
VQQHDAVOb2lkYTEQMA4GA1UECgwHU2FmZW5ldDEMMAoGA1UECwwDSFNNMRswGQYD
VQQDDBJ3d3cuaGFwcm94eWhzbS5jb20xGjAYBgkqhkiG9w0BCQEWC2FiY0BrZW0u
aW5vMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtmLqlzlEhuED9uia
wBE1UYgqg1PMZeCRZ/ud30HhSM8nKeKbJzdoZxpH2ifo4ViV9ftOdij+Yfol1akz
5QHyQazu+Rp6R8YvlAytQNHceZBUiA+TmCaTS+2VE4BdI5qhShNhICkI8+lSocfn
ry0z9tZvOxnz8R01JiaR2CGdKowvYIJHOU09PWeuAdCNKf20NzBT0fdbmrZzgaaR
c5OInROQlTSR2d3Da0usQtipz0rOe8PtpXdBNpb3U0LqC14vMmwPD4r2YgNzcD6z
xB1XyhnXy4WyDx8uRQYhkiREIOd5eBgnoqgzUdg2kSKo3XVLbgMpcUbYZTvgtZxB
KNO1xQIDAQABo1AwTjAdBgNVHQ4EFgQUHTYDdXwyhz7lxyDOhzNCP+w05GcwHwYD
VR0jBBgwFoAUHTYDdXwyhz7lxyDOhzNCP+w05GcwDAYDVR0TBAUwAwEB/zANBgkq
hkiG9w0BAQsFAAOCAQEAFMyz4bNTOCUn0Ehnub6OKul0KgnG1RINMZMUNurFuAe9
jD0Fadm+PsXrdpg7+f21eByR18WTcNsC5YAqjkYKMyF6JsP6lYlFnYFLbZAdvSEY
sJP+PSRnoW3t1wOWNDWefvk5Y/KOPN+dst/VyYrojD3y8VekkK+7k/XNUWBixZ9m
80FKd8xvrxPx/Gxv1Q9PTSuDSuNQUMGpwz9VpDdlNAastEGiKbFySPh8BD9i1T6z
fDw19m2OWWGqQy92SUg4fs7HoQZ7l6dwdkX5ShZyv3wz3AdkA4kwcSM7THU5vW0C
Fbn5BY1N8m994U/2/+jvX6rUwAQsncm/nDKK71mD4g==
-----END CERTIFICATE-----
subject=/C=IN/ST=UP/L=Noida/O=XYZ/OU=HSM/CN=wwwhaproxyhsmcom/emailAddress=abc@kem.ino
issuer=/C=IN/ST=UP/L=Noida/O=XYZ/OU=HSM/CN=wwwhaproxyhsmcom/emailAddress=abc@kem.ino
—
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
—
SSL handshake has read 1650 bytes and written 415 bytes
—
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: FDEE9908F72E6E40150CCBAA7430AC3B330913D2F644EFE70FA71E6E7D1EE5AD
Session-ID-ctx:
Master-Key: 769F732987326B2F1A69CC5426ADDF2C10ABA395AA64A089967304E495C02379DE21335A661B5FC59C8770ED08EF47B4
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 60 (seconds)
TLS session ticket:
0000 - db 27 4f 54 ac bf e3 db-a5 12 53 a8 d3 fd bf 5f .‘OT…S…_
0010 - e2 04 c4 9e 82 f1 c5 b3-65 ae b2 87 dd 4e 2b 8f …e…N+.
0020 - b5 a8 3a 06 9c 4f b6 15-94 a3 8a 48 cf ea 74 7f …:…O…H…t.
0030 - 7c e7 6f 66 8d 6a 18 3f-3e 2b 7c b3 21 84 b9 6e |.of.j.?>+|.!..n
0040 - e5 fd 36 47 63 6a 9d 03-89 1d fd 6a fd 92 30 4a …6Gcj…j…0J
0050 - c0 79 26 78 a2 3e d0 e3-1e 71 20 06 64 af 90 21 .y&x.>…q .d…!
0060 - a7 6c ae cd 9c c1 42 5b-6c b4 22 fb de 3e a7 4b .l…B[l."…>.K
0070 - f0 00 3b 71 28 1b 76 6d-89 23 7f 44 6d eb a0 b9 …;q(.vm.#.Dm…
0080 - 58 42 1a c6 58 e3 b2 dc-1e 84 de e4 c3 58 89 92 XB…X…X…
0090 - 6f 98 3f 95 0f 20 6b 5b-6c 28 27 8a 86 7f 9d c5 o.?.. k[l(’…
Start Time: 1543474928
Timeout : 300 (sec)
Verify return code: 18 (self signed certificate)
HTTP/1.0 408 Request Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<.html><.body><.h1>408 Request Time-out<./h1>
Your browser didn’t send a complete request in time.
<./body><./html>
closed
I am able to start the service successfully.When i am starting the service it is going to the HSM for private key with private key reference and service is up with no error.
Logs:
[root@noi-9y0fk72 bin]# systemctl status -l rh-haproxy18-haproxy.service
● rh-haproxy18-haproxy.service - HAProxy Load Balancer
Loaded: loaded (/usr/lib/systemd/system/rh-haproxy18-haproxy.service; disabled; vendor preset: disabled)
Active: active (running) since Thu 2018-11-29 12:31:54 IST; 17min ago
Process: 29210 ExecStartPre=/opt/rh/rh-haproxy18/root/usr/sbin/haproxy -f $CONFIG -c -q (code=exited, status=0/SUCCESS)
Main PID: 29212 (haproxy)
Tasks: 2
CGroup: /system.slice/rh-haproxy18-haproxy.service
├─29212 /opt/rh/rh-haproxy18/root/usr/sbin/haproxy -Ws -f /etc/opt/rh/rh-haproxy18/haproxy/haproxy.cfg -p /run/rh-haproxy18-haproxy.pid
└─29214 /opt/rh/rh-haproxy18/root/usr/sbin/haproxy -Ws -f /etc/opt/rh/rh-haproxy18/haproxy/haproxy.cfg -p /run/rh-haproxy18-haproxy.pid**
Nov 29 12:31:54 noi-9y0fk72 systemd[1]: Starting HAProxy Load Balancer…
Nov 29 12:31:54 noi-9y0fk72 haproxy[29210]: Retrieving passphrase for slot id 0x1… success.
Nov 29 12:31:54 noi-9y0fk72 systemd[1]: Started HAProxy Load Balancer.
Nov 29 12:31:54 noi-9y0fk72 haproxy[29212]: Retrieving passphrase for slot id 0x1… success.
[root@noi-9y0fk72 bin]#
But the problem is that while opening withhttps://HAProxyIp ,it is not going to HSM for private key.
So it seems that HAProxy is somewhere storing the retrieved private key/response from HSM at the time of starting the service in its own cache.Even if i am deleteing the keys from HSM it is still able to open https://HAProxyIp on https without any error.
So my questions are:
- is there any way i can force HAProxy to go to HSM for private key each time i hit https://HAProxyIp??
2.If not, then can i set some time like after every 5 minutes it clears the key/response from cache and go to HSM for private key and update itself?