Thanks, yes someone suggested the unix socket. But now a have a link, thanks again.
As for reading the key from the memory; there is vast difference in simply reading a file and parsing / scraping memory. But obviously, hackers have a lot of tools and expertise so it is definitely not impossible. However, memory obfuscation techniques exists, which significantly improve (without eliminating) the security issues. But, this is another debate…
I think it would make a lot of sense for haproxy to start thinking about adding internal protection against memory scraping attacks.
About memory protection. Some of this is provided by the HW (memory encryption where the kernel plays a minimal role), by the OS (actually it is the dynamic linker doing randomization) all of which the app does not play a role. But what the app could do is spread the key into multiple parts, obfuscate the parts (using may be the good old One Time Pad technique)(It is really Pad, not Password). Combining all of this makes compromising the key from memory several order of magnitude more complex than accessing a file (even if read permissions are negated the hacker may somehow gain CAP_DAC_OVERRIDE, or get haproxy user identity, or whatever hole they find). Just static access to memory is not enough.
No, keyless SSL still requires another private key for the client certificate that the Cloudflare server needs to access the key server. So it is not delivering what I am asking for. And that extra private key is not that short lived. It can certainly be much shorter than 1 year but certainly not 1 time use. Or, if it is 1 time use then it is likely that some API is used to get a new key (or certificate actually) each time. That API requires another secret (may be a token in that case).
You are correct, that extra key allows a compromised box to query the key server for the site private key and then exfiltrate it.
So, in the end, an alternative is (and I am not saying it is the ultimate solution since nothing will ever be the ultimate answer) this:
Encrypt all the secrets, including API token, certificate private keys, etc. May be using JWE
Deliver the encrypted secret to the workload
Decrypt the JWE using material obtained from outside the workload (tang server for instance)
Protect the decrypted secret that sits in memory as much as possible
Destroy any intermediate copy of the decrypted secret (as well as the material obtained from outside).
The important point to the alternative is that getting the material (from the tang server) is gated / controlled (externally) and only available at start up time of the workload. If the workload gets compromised later on, the outside material access is blocked. Ideally, the gating is unlocked by a human actor (like when your bank send a request to your phone after you logged in from your desktop’s browser). But other criteria can be devised.
The resulting effect is that there is nothing in the workload available to hacker to exfiltrate a primary key, or a secondary secret to obtain the primary key. The exception is haproxy memory. Hence my statement about making it more secure.
The beauty of all this is that it is mostly driven by the series of RFCs on JWT.
To come back to the origin question, you can add certificates through api.
However, it makes no sense to try to protect haproxy if the server is compromised.
Haproxy needs the key during runtime, so it must be in the memory. If you do some of your suggest encryption/obfuscation (tang etc.), all the needed keys must be stored in the current accessable memory. If you know the code, all reverse engineering is trivial in terms of a hacker.
You put lot of energy in just feeling more secure without being more secure.
Just do the normal stuff. Privilege separation, do updates, block/filter unneeded traffic, do updates, block potential attackers (fail2ban), do updates, don’t run unknown 3rd party code, do updates….
As for the keys in memory; if the keys are never in clear text (so obfuscated) in the heap, that the obfuscation material is also not in the heap in clear text, that the loader is doing memory address randomization (pretty standard these days) and that the only time the keys are assembled / clear text is on the stack then no, it is not trivial for a hacker to extract the keys. Not impossible, but significantly harder. Libraries were even created for this purpose (but forgot the name sorry).
The normal stuff you mention are really good and a must. But hacking is still possible: Azure probably does them all but yet keys / account access capability were leaked in last July. All this to say that never have too much layer of security.
Now, I understand that haproxy does not none of the stuff I mention above in term of key protection while in memory but may it is worth considering.