Hi,
What mechanism other than plain file exists, if any, to load certificates? I find having files around in a workload is dangerous; a malware could access them and exfiltrate the private key.
So I want to explore alternatives but first I need to understand what is available in haproxy.
If you OS is compromised a malware can read your private key from memory.
The solution to this is keyless SSL, something wich afaik is not really a thing in OSS solutions currently.
But you can add remove certificates from the unix socket:
http://docs.haproxy.org/2.6/management.html#9.3
Many limitations apply, you will have to read the documentation carefully and evaluate whether this makes actual sense in your case.
edit: a HSM would solve this, but I’m not sure if that is suitable for high performance.
Depending on your exact requirements, maybe it would suffice to store certs on tmpfs, as oppossed to disk.
Thanks, yes someone suggested the unix socket. But now a have a link, thanks again.
As for reading the key from the memory; there is vast difference in simply reading a file and parsing / scraping memory. But obviously, hackers have a lot of tools and expertise so it is definitely not impossible. However, memory obfuscation techniques exists, which significantly improve (without eliminating) the security issues. But, this is another debate…
I think it would make a lot of sense for haproxy to start thinking about adding internal protection against memory scraping attacks.
I just read about keyless SSL. What an overstatement!!! The fact of moving the key elsewhere does not make SSL keyless!
And on top of this both actors (the edge/proxy and the ultimate key server) needs additional privates keys, which could leak… So what is the gain?
Can you elaborate what you are specifically suggesting?
Whether you like the name or not, what Cloudflare does with their keyless SSL implementation is exactly what you asked for:
That’s a Cloudflare specific implementation.
RFC9345 is a proposal for a standard implementation, but I don’t know the details.
Keys are specific to a session and very short lived as opposed to a certificate private key that could be valid a year.
What you can do with such a key you can do anyway when the box this is running on is compromised.
About memory protection. Some of this is provided by the HW (memory encryption where the kernel plays a minimal role), by the OS (actually it is the dynamic linker doing randomization) all of which the app does not play a role. But what the app could do is spread the key into multiple parts, obfuscate the parts (using may be the good old One Time Pad technique)(It is really Pad, not Password). Combining all of this makes compromising the key from memory several order of magnitude more complex than accessing a file (even if read permissions are negated the hacker may somehow gain CAP_DAC_OVERRIDE, or get haproxy user identity, or whatever hole they find). Just static access to memory is not enough.
No, keyless SSL still requires another private key for the client certificate that the Cloudflare server needs to access the key server. So it is not delivering what I am asking for. And that extra private key is not that short lived. It can certainly be much shorter than 1 year but certainly not 1 time use. Or, if it is 1 time use then it is likely that some API is used to get a new key (or certificate actually) each time. That API requires another secret (may be a token in that case).
You are correct, that extra key allows a compromised box to query the key server for the site private key and then exfiltrate it.
So, in the end, an alternative is (and I am not saying it is the ultimate solution since nothing will ever be the ultimate answer) this:
The important point to the alternative is that getting the material (from the tang server) is gated / controlled (externally) and only available at start up time of the workload. If the workload gets compromised later on, the outside material access is blocked. Ideally, the gating is unlocked by a human actor (like when your bank send a request to your phone after you logged in from your desktop’s browser). But other criteria can be devised.
The resulting effect is that there is nothing in the workload available to hacker to exfiltrate a primary key, or a secondary secret to obtain the primary key. The exception is haproxy memory. Hence my statement about making it more secure.
The beauty of all this is that it is mostly driven by the series of RFCs on JWT.
To come back to the origin question, you can add certificates through api.
However, it makes no sense to try to protect haproxy if the server is compromised.
Haproxy needs the key during runtime, so it must be in the memory. If you do some of your suggest encryption/obfuscation (tang etc.), all the needed keys must be stored in the current accessable memory. If you know the code, all reverse engineering is trivial in terms of a hacker.
You put lot of energy in just feeling more secure without being more secure.
Just do the normal stuff. Privilege separation, do updates, block/filter unneeded traffic, do updates, block potential attackers (fail2ban), do updates, don’t run unknown 3rd party code, do updates….
Thanks for the pointer for the API.
As for the keys in memory; if the keys are never in clear text (so obfuscated) in the heap, that the obfuscation material is also not in the heap in clear text, that the loader is doing memory address randomization (pretty standard these days) and that the only time the keys are assembled / clear text is on the stack then no, it is not trivial for a hacker to extract the keys. Not impossible, but significantly harder. Libraries were even created for this purpose (but forgot the name sorry).
The normal stuff you mention are really good and a must. But hacking is still possible: Azure probably does them all but yet keys / account access capability were leaked in last July. All this to say that never have too much layer of security.
Now, I understand that haproxy does not none of the stuff I mention above in term of key protection while in memory but may it is worth considering.