I am facing an issue with Haproxy. I am trying to send cert files to Harshicorp vault through Haproxy. It works fine if I run cur directly hitting vault servers. But its failing through Haproxy.
You are using client certificate authentication in curl. You need to either put that certificate on the haproxy box, or configure haproxy transparently.
Thats the part I am not getting. I do have a certificate already in /etc/pki/trust/anchors for ssl. But this certificate you are seeing in curl needs to send to vault which has RootCA cert already stored.
How can I send these certs to backend vault from Haproxy. Same is working if I hit vault server directly.
Below is the response directly hitting vault
------------response------------
curl -vvv --request PUT --cacert rootCAcert.pem --cert clientcrt.pem --key clientkey.pem } --data ‘{“name”: “rootca”}’ https://ito028708.hosts.com/v1/auth/cert/login
curl: (3) [globbing] unmatched close brace/bracket in column 1
backend Vault
option httpchk GET /v1/sys/health
http-check expect rstatus ^(429|200)$
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port 443
{{~#eachAlive bind.backend.members as |member|}}
server {{member.sys.hostname}} {{member.sys.ip}}:{{member.cfg.port}} check ssl verify none inter 2s
{{~/eachAlive}}
{{#if cfg.dr ~}}
backend Vault_dr
option httpchk GET /v1/sys/health
http-check expect rstring {{cfg.backend_data.dr_expect}}
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port 443
{{~#each cfg.dr as |member|}}
server {{member.server}} {{member.ip}}:{{member.port}} check ssl verify none inter 2s{{~/each}}
{{~/if}}
backend Consul
mode tcp
option tcp-check
{{~#eachAlive bind.consul.members as |member|}}
server {{member.sys.hostname}} {{member.sys.ip}}:{{member.cfg.port-https}} check inter 2s
{{~/eachAlive}}
either you configure haproxy transparently (without TLS termination), so that the TLS session is end-to-end (and the client certificate reaches your backend)
or your configure the client certificate on haproxy (clientcrt.pem and clientkey.pem on your backend configuration, so that it is send as client certificate to the backend) - see the crt keyword on the server line
Thanks very much.
Can you please provide or guide me to examples or documentation for setting up TLS end to end session. I am very new to Haproxy and certs.
Can you please explain second method a little bit.
As the purpose of sending client cert to vault by end user is get back token from vault. End user wont have access to haproxy boxes, then this might not a right method. Please confirm if I got it right
If you have different end-users, each with different certificates that are supposed to be send to the backend server, then you don’t have a choice; you need to configure Haproxy transparently, so that the TLS connections is established end-to-end.
To do that your remove the entire ssl configuration and put both front and backend into mode tcp.
So:
replace mode http with mode tcp everywhere (or just in the default section and remove the other mode statements)
remove ssl no-sslv3 crt /etc/pki/trust/anchors/{{sys.hostname}}.{{cfg.server.domain}}.pem from the bind statement
remove ssl verify none from the backend servers
you may have to double check that health checks are still working as you remove the ssl layer. Maybe use check-ssl here
backend Vault
option httpchk GET /v1/sys/health
http-check expect rstatus ^(429|200)$
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port 443
{{~#eachAlive bind.backend.members as |member|}}
server {{member.sys.hostname}} {{member.sys.ip}}:{{member.cfg.port}} check inter 2s
{{~/eachAlive}}
{{#if cfg.dr ~}}
backend Vault_dr
option httpchk GET /v1/sys/health
http-check expect rstring {{cfg.backend_data.dr_expect}}
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port 443
{{~#each cfg.dr as |member|}}
server {{member.server}} {{member.ip}}:{{member.port}} check inter 2s{{~/each}}
{{~/if}}
backend Consul
mode tcp
option tcp-check
{{~#eachAlive bind.consul.members as |member|}}
server {{member.sys.hostname}} {{member.sys.ip}}:{{member.cfg.port-https}} check inter 2s
{{~/eachAlive}}
I have another question:
How can I see all access logs or requests in haproxy logs going to journalctl. I changed config from notice to info but that didnt helped.
Okey my bad. I was using https in command. I just tried with below command with http but got
curl: (52) Empty reply from server
and I am also seeing “http: TLS handshake error from haproxyIP:34900: tls: first record does not look like a TLS handshake” in vault server logs. I am unable to get any logs in haproxy for any requests made to Haproxy.
Like I said you need to be careful with health checks. Read the log and fix health checks. Use check-ssl on the servers and confirm that the health check is successful.
You NEED to use https in your curl call, you can’t use a client certificate when you don’t use HTTPS.
If you add new outputs, please put them in code tags </> so they are readable.
You configured haproxy to log to 127.0.0.1, so there needs to be a syslog daemon listening for it. journalctl can only log what is going to stderr and stdout, which is not much.
I got that I still need to use https and as you have suggested with removing mode http to mode tcp and other configurations from config, it should enable TLS end to end session. Then how I should be able to post certs to vault from haproxy.
Do I still need to keep those PEM files on servers.
You don’t need any certificates on the haproxy box. Haproxy will just establish a TCP connection to your backend server, permitting end-to-end HTTPS connectivity, including client certificate authentication.
I am confused I guess. Here are my questions
If we are terminating SSL at haproxy how other auth methods are working with vault and only cert method is not working?
Example curl -X POST --data '{"role_id":"3c740d73-f69b-1f3f-a923-a51f8c4eacc4","secret_id": "52e41fe6-d6cc-6aad-8868-fd937cfd447a"}' https://ito024571.hosts.com/v1/auth/approle/login
How can we make pass through work with haproxy and keep using ssl certs at haproxy or we cant use cert at haproxy?
Is mode http not allowing certs and terminating ssl or what settings are terminating the SSL. I mean how I explain that SLL is terminating at HAproxy layer
Also I have tried removing settings (like mode http and others) mentioned by you
curl --request PUT --cacert rootCAcert.pem --cert clientcrt.pem --key clientkey.pem --data '{"name": "rootca"}' https://ito024574.hosts.om/v1/auth/cert/login curl: (35) Unknown SSL protocol error in connection to ito024574.hosts.com:443
You are not using SSL client certificate authentication here, which is why it works even when terminating SSL at haproxy.
You can’t.
You either terminate SSL, which by definition also means the client certificate is terminated there, or not.
Please read my previous posts carefully. Read the logs, make sure health checks are working.
If you can figure it out, instead of starting with an complicated configuration, trim it down and start with a simple configuration, without health checks.
Hi
Health checks are working. I modified the config and able to pass through.Auth method is working. Need your guidance on below two things now.
1). I am getting below error if I dont add -k flag to curl call.
2). The reason of above error what I understand is backend server which is vault has cert with CN of it own hostname but its expecting CN of Haproxy server.
Either adjust the certificate so it matches what curl expects, or you adjust DNS so that it points to haproxy instead of the origin server, so you can use hostname for this.
Or, if you are just trying to test this with curl, instead of pointing to haproxy via the haproxy hostname, rewrite the resolution of the origin server in curl by using the –resolve directive.
Change the IP address of ito024571.hosts.com from the destination server to the haproxy instance, so that you can use that in the curl call. Of course, you need to configure the IP addresses in the haproxy configuration then (no the hostname, otherwise you point haproxy to itself).
I have no idea what your end goal is, and what you plan on doing with this setup, so I can only make generic suggestion, you need to choose what works best for you out of those 3 suggestions.