Haproxy not sending files to backend server

Hi Everyone,

I am facing an issue with Haproxy. I am trying to send cert files to Harshicorp vault through Haproxy. It works fine if I run cur directly hitting vault servers. But its failing through Haproxy.

Command: curl -vvv --request PUT --cacert rootCAcert.pem --cert clientcrt.pem --key clientkey.pem --data ‘{“name”: “rootca”}’ https:///v1/auth/cert/login

Error

1). {“errors”:[“tls connection required”]}

2).{“errors”:[“client certificate must be supplied”]}

Verbose output:

  • Hostname was NOT found in DNS cache

  • Trying …

  • Connected to () port 443 (#0)

  • successfully set certificate verify locations:

  • CAfile: rootCAcert.pem

CApath: /etc/ssl/certs/

  • SSLv3, TLS Unknown, Unknown (22):

  • SSLv3, TLS handshake, Client hello (1):

  • SSLv2, Unknown (22):

  • SSLv3, TLS handshake, Server hello (2):

  • SSLv2, Unknown (22):

  • SSLv3, TLS handshake, CERT (11):

  • SSLv2, Unknown (22):

  • SSLv3, TLS handshake, Server key exchange (12):

  • SSLv2, Unknown (22):

  • SSLv3, TLS handshake, Server finished (14):

  • SSLv2, Unknown (22):

  • SSLv3, TLS handshake, Client key exchange (16):

  • SSLv2, Unknown (20):

  • SSLv3, TLS change cipher, Client hello (1):

  • SSLv2, Unknown (22):

  • SSLv3, TLS handshake, Finished (20):

  • SSLv2, Unknown (20):

  • SSLv3, TLS change cipher, Client hello (1):

  • SSLv2, Unknown (22):

  • SSLv3, TLS handshake, Finished (20):

  • SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

  • Server certificate:

  • subject: C=US; ST=Michigan; L=Dearborn; O=; CN=

  • start date: 2019-05-07 15:35:42 GMT

  • expire date: 2021-05-07 15:35:42 GMT

  • subjectAltName: matched

  • issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Organization Validation CA - SHA256 - G2

  • SSL certificate verify ok.

  • SSLv2, Unknown (23):

PUT /v1/auth/cert/login HTTP/1.1

User-Agent: curl/7.37.0

Host:

Accept: /

Content-Length: 18

Content-Type: application/x-www-form-urlencoded

  • upload completely sent off: 18 out of 18 bytes

  • SSLv2, Unknown (23):

< HTTP/1.1 400 Bad Request

< Cache-Control: no-store

< Content-Type: application/json

< Date: Mon, 13 May 2019 19:31:32 GMT

< Content-Length: 51

<

{“errors”:[“client certificate must be supplied”]}

  • Connection #0 to host left intact

I see value for Content-Length above is 550 if I hit vault server directly

Share your haproxy configuration please.

You are using client certificate authentication in curl. You need to either put that certificate on the haproxy box, or configure haproxy transparently.

1 Like

Thats the part I am not getting. I do have a certificate already in /etc/pki/trust/anchors for ssl. But this certificate you are seeing in curl needs to send to vault which has RootCA cert already stored.
How can I send these certs to backend vault from Haproxy. Same is working if I hit vault server directly.
Below is the response directly hitting vault
------------response------------
curl -vvv --request PUT --cacert rootCAcert.pem --cert clientcrt.pem --key clientkey.pem } --data ‘{“name”: “rootca”}’ https://ito028708.hosts.com/v1/auth/cert/login
curl: (3) [globbing] unmatched close brace/bracket in column 1

  • Hostname was NOT found in DNS cache
  • Trying 19.14.83.124…
  • Connected to ito028708.hosts.com (ip) port 443 (#0)
  • successfully set certificate verify locations:
  • CAfile: rootCAcert.pem
    CApath: /etc/ssl/certs/
  • SSLv3, TLS Unknown, Unknown (22):
  • SSLv3, TLS handshake, Client hello (1):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, Server hello (2):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, CERT (11):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, Server key exchange (12):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, Request CERT (13):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, Server finished (14):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, CERT (11):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, Client key exchange (16):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, CERT verify (15):
  • SSLv2, Unknown (20):
  • SSLv3, TLS change cipher, Client hello (1):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, Finished (20):
  • SSLv2, Unknown (20):
  • SSLv3, TLS change cipher, Client hello (1):
  • SSLv2, Unknown (22):
  • SSLv3, TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
  • Server certificate:
  • subject: C=US; ST=Michigan; L=Dearborn; O; CN=ito028708.hosts…com
  • start date: 2019-05-07 15:34:43 GMT
  • expire date: 2021-05-07 15:34:43 GMT
  • subjectAltName: ito028708.hosts.com matched
  • issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Organization Validation CA - SHA256 - G2
  • SSL certificate verify ok.
  • SSLv2, Unknown (23):

PUT /v1/auth/cert/login HTTP/1.1
User-Agent: curl/7.37.0
Host: ito028708.hosts.com
Accept: /
Content-Length: 18
Content-Type: application/x-www-form-urlencoded

  • upload completely sent off: 18 out of 18 bytes
  • SSLv2, Unknown (23):
    < HTTP/1.1 200 OK
    < Cache-Control: no-store
    < Content-Type: application/json
    < Date: Tue, 14 May 2019 15:16:18 GMT
    < Content-Length: 550
    <
    {"token response}}
  • Connection #0 to host ito028708.hosts.com left intact


global
maxconn {{cfg.maxconn}}
ulimit-n {{cfg.maxfiles}}
daemon
tune.ssl.default-dh-param 2048

set default parameters to the modern configuration

ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
log 127.0.0.1 local0 notice
log-send-hostname localhost
nbproc 4
cpu-map 1 0
cpu-map 2 1
cpu-map 3 2
cpu-map 4 3

defaults
mode http
maxconn 1000
balance roundrobin
timeout client 5000
timeout server 5000
timeout connect 5000
option log-health-checks
option log-separate-errors
option http-keep-alive
option dontlognull
option splice-auto
option socket-stats
option httplog
option tcplog
option redispatch
option forwardfor
option http-server-close
retries 3
log global
bind-process 1

HAProxy will listen on both 80 and 443

Traffic from 80 will be redirected to 443

frontend https
bind-process 2 3 4
bind 0.0.0.0:443 tfo ssl no-sslv3 crt /etc/pki/trust/anchors/{{sys.hostname}}.{{cfg.server.domain}}.pem process 2
bind 0.0.0.0:443 tfo ssl no-sslv3 crt /etc/pki/trust/anchors/{{sys.hostname}}.{{cfg.server.domain}}.pem process 3
bind 0.0.0.0:443 tfo ssl no-sslv3 crt /etc/pki/trust/anchors/{{sys.hostname}}.{{cfg.server.domain}}.pem process 4
option tcp-smart-connect
{{~#each bind.backend.members as |member|}}
acl {{member.sys.hostname}} srv_is_up(Vault/{{member.sys.hostname}}) {{~#unless @last}}{{/unless}}
{{~/each}}
use_backend Vault if {{~#each bind.backend.members as |member|}} {{member.sys.hostname}} {{~#unless @last}}{{/unless}}{{~/each}}
{{#if cfg.dr ~}}
use_backend Vault_dr if {{~#each bind.backend.members as |member|}} !{{member.sys.hostname}} {{~#unless @last}}{{/unless}}{{~/each}}
{{~/if}}

frontend http
bind-process 1
bind *:80
mode http
redirect scheme https if !{ ssl_fc }

backend Vault
option httpchk GET /v1/sys/health
http-check expect rstatus ^(429|200)$
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port 443
{{~#eachAlive bind.backend.members as |member|}}
server {{member.sys.hostname}} {{member.sys.ip}}:{{member.cfg.port}} check ssl verify none inter 2s
{{~/eachAlive}}

{{#if cfg.dr ~}}
backend Vault_dr
option httpchk GET /v1/sys/health
http-check expect rstring {{cfg.backend_data.dr_expect}}
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port 443
{{~#each cfg.dr as |member|}}
server {{member.server}} {{member.ip}}:{{member.port}} check ssl verify none inter 2s{{~/each}}
{{~/if}}

backend Consul
mode tcp
option tcp-check
{{~#eachAlive bind.consul.members as |member|}}
server {{member.sys.hostname}} {{member.sys.ip}}:{{member.cfg.port-https}} check inter 2s
{{~/eachAlive}}

{{#if cfg.status.enabled}}
listen stats
mode http
bind {{cfg.status.listen}}:{{cfg.status.port}}
stats uri {{cfg.status.uri}}
stats realm Haproxy-Statistics
stats auth {{cfg.status.user}}:{{cfg.status.password}}
stats enable
stats hide-version
stats refresh 5s
stats show-node
{{~/if}


The config confirms what I said:

  • either you configure haproxy transparently (without TLS termination), so that the TLS session is end-to-end (and the client certificate reaches your backend)
  • or your configure the client certificate on haproxy (clientcrt.pem and clientkey.pem on your backend configuration, so that it is send as client certificate to the backend) - see the crt keyword on the server line
1 Like

Thanks very much.
Can you please provide or guide me to examples or documentation for setting up TLS end to end session. I am very new to Haproxy and certs.
Can you please explain second method a little bit.
As the purpose of sending client cert to vault by end user is get back token from vault. End user wont have access to haproxy boxes, then this might not a right method. Please confirm if I got it right

If you have different end-users, each with different certificates that are supposed to be send to the backend server, then you don’t have a choice; you need to configure Haproxy transparently, so that the TLS connections is established end-to-end.

To do that your remove the entire ssl configuration and put both front and backend into mode tcp.

So:

  • replace mode http with mode tcp everywhere (or just in the default section and remove the other mode statements)
  • remove ssl no-sslv3 crt /etc/pki/trust/anchors/{{sys.hostname}}.{{cfg.server.domain}}.pem from the bind statement
  • remove ssl verify none from the backend servers
  • you may have to double check that health checks are still working as you remove the ssl layer. Maybe use check-ssl here
1 Like

You are awesome. Thanks for clarifying. This answers my question but still need to fix it. Please help !

I am getting below error now:
url: (3) [globbing] unmatched close brace/bracket in column 1

  • Hostname was NOT found in DNS cache
  • Trying ip…
  • Connected to ito024574.hosts.com (ip) port 443 (#0)
  • unable to use client certificate (no key found or wrong pass phrase?)

Here is changed config as you suggested.

global
maxconn {{cfg.maxconn}}
ulimit-n {{cfg.maxfiles}}
daemon
tune.ssl.default-dh-param 2048

set default parameters to the modern configuration

ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
log 127.0.0.1 local0 notice
log-send-hostname localhost
nbproc 4
cpu-map 1 0
cpu-map 2 1
cpu-map 3 2
cpu-map 4 3

defaults
mode tcp
maxconn 1000
balance roundrobin
timeout client 5000
timeout server 5000
timeout connect 5000
option log-health-checks
option log-separate-errors
option http-keep-alive
option dontlognull
option splice-auto
option socket-stats
option httplog
option tcplog
option redispatch
option forwardfor
option http-server-close
retries 3
log global
bind-process 1

HAProxy will listen on both 80 and 443

Traffic from 80 will be redirected to 443

frontend https
bind-process 2 3 4
bind 0.0.0.0:443 tfo process 2
bind 0.0.0.0:443 tfo process 3
bind 0.0.0.0:443 tfo process 4
option tcp-smart-connect
{{~#each bind.backend.members as |member|}}
acl {{member.sys.hostname}} srv_is_up(Vault/{{member.sys.hostname}}) {{~#unless @last}}{{/unless}}
{{~/each}}
use_backend Vault if {{~#each bind.backend.members as |member|}} {{member.sys.hostname}} {{~#unless @last}}{{/unless}}{{~/each}}
{{#if cfg.dr ~}}
use_backend Vault_dr if {{~#each bind.backend.members as |member|}} !{{member.sys.hostname}} {{~#unless @last}}{{/unless}}{{~/each}}
{{~/if}}

frontend http
bind-process 1
bind *:80
redirect scheme https if !{ ssl_fc }

backend Vault
option httpchk GET /v1/sys/health
http-check expect rstatus ^(429|200)$
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port 443
{{~#eachAlive bind.backend.members as |member|}}
server {{member.sys.hostname}} {{member.sys.ip}}:{{member.cfg.port}} check inter 2s
{{~/eachAlive}}

{{#if cfg.dr ~}}
backend Vault_dr
option httpchk GET /v1/sys/health
http-check expect rstring {{cfg.backend_data.dr_expect}}
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Port 443
{{~#each cfg.dr as |member|}}
server {{member.server}} {{member.ip}}:{{member.port}} check inter 2s{{~/each}}
{{~/if}}

backend Consul
mode tcp
option tcp-check
{{~#eachAlive bind.consul.members as |member|}}
server {{member.sys.hostname}} {{member.sys.ip}}:{{member.cfg.port-https}} check inter 2s
{{~/eachAlive}}

{{#if cfg.status.enabled}}
listen stats
bind {{cfg.status.listen}}:{{cfg.status.port}}
stats uri {{cfg.status.uri}}
stats realm Haproxy-Statistics
stats auth {{cfg.status.user}}:{{cfg.status.password}}
stats enable
stats hide-version
stats refresh 5s
stats show-node
{{~/if}}

Also tried removing below part from config as well

tune.ssl.default-dh-param 2048

set default parameters to the modern configuration

ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
ssl-default-server-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

I have another question:
How can I see all access logs or requests in haproxy logs going to journalctl. I changed config from notice to info but that didnt helped.

Okey my bad. I was using https in command. I just tried with below command with http but got


curl: (52) Empty reply from server

and I am also seeing “http: TLS handshake error from haproxyIP:34900: tls: first record does not look like a TLS handshake” in vault server logs. I am unable to get any logs in haproxy for any requests made to Haproxy.

command:
curl --request PUT --cacert rootCAcert.pem --cert clientcrt.pem --key clientkey.pem --data ‘{“name”: “rootca”}’ http://ito024574.hosts.com/v1/auth/cert/login

Like I said you need to be careful with health checks. Read the log and fix health checks. Use check-ssl on the servers and confirm that the health check is successful.

You NEED to use https in your curl call, you can’t use a client certificate when you don’t use HTTPS.

If you add new outputs, please put them in code tags </> so they are readable.

You configured haproxy to log to 127.0.0.1, so there needs to be a syslog daemon listening for it. journalctl can only log what is going to stderr and stdout, which is not much.

Hi Thanks for replying.

I got that I still need to use https and as you have suggested with removing mode http to mode tcp and other configurations from config, it should enable TLS end to end session. Then how I should be able to post certs to vault from haproxy.

Do I still need to keep those PEM files on servers.

Hi

I highly appreciate your responses. But I really need your help on how to implement approach 1 discussed above.Please pardon my ignorance.

You don’t need any certificates on the haproxy box. Haproxy will just establish a TCP connection to your backend server, permitting end-to-end HTTPS connectivity, including client certificate authentication.

Exactly what is your question right now?

Thanks this explains a lot.

I am confused I guess. Here are my questions
If we are terminating SSL at haproxy how other auth methods are working with vault and only cert method is not working?
Example curl -X POST --data '{"role_id":"3c740d73-f69b-1f3f-a923-a51f8c4eacc4","secret_id": "52e41fe6-d6cc-6aad-8868-fd937cfd447a"}' https://ito024571.hosts.com/v1/auth/approle/login
How can we make pass through work with haproxy and keep using ssl certs at haproxy or we cant use cert at haproxy?
Is mode http not allowing certs and terminating ssl or what settings are terminating the SSL. I mean how I explain that SLL is terminating at HAproxy layer

Also I have tried removing settings (like mode http and others) mentioned by you

curl --request PUT --cacert rootCAcert.pem --cert clientcrt.pem --key clientkey.pem --data '{"name": "rootca"}' https://ito024574.hosts.om/v1/auth/cert/login curl: (35) Unknown SSL protocol error in connection to ito024574.hosts.com:443

Its not allowing me to connect now

You are not using SSL client certificate authentication here, which is why it works even when terminating SSL at haproxy.

You can’t.

You either terminate SSL, which by definition also means the client certificate is terminated there, or not.

Please read my previous posts carefully. Read the logs, make sure health checks are working.

If you can figure it out, instead of starting with an complicated configuration, trim it down and start with a simple configuration, without health checks.

1 Like

Thanks, things started making sense now. I appreciate your help ]. Thanks very much

Hi
Health checks are working. I modified the config and able to pass through.Auth method is working. Need your guidance on below two things now.
1). I am getting below error if I dont add -k flag to curl call.

curl: (51) SSL: no alternative certificate subject name matches target host name ‘ito024574.hosts.cloud.ford.com’*

Is there a way to make it work without -k flag.

2). The reason of above error what I understand is backend server which is vault has cert with CN of it own hostname but its expecting CN of Haproxy server.

From curl verbose logs:

server certificate:

Am I right here. Is this can be fixed by generating a certificate with CN of Haproxy.

Either adjust the certificate so it matches what curl expects, or you adjust DNS so that it points to haproxy instead of the origin server, so you can use hostname for this.

Or, if you are just trying to test this with curl, instead of pointing to haproxy via the haproxy hostname, rewrite the resolution of the origin server in curl by using the –resolve directive.

Hi
Thanks for answering.
I got the first part from your answer but can you please elaborate the second part where you have mentioned to adjust DNS.

I am using curl to test this otherwise applications are making these api requests to vault cluster of three servers.

Change the IP address of ito024571.hosts.com from the destination server to the haproxy instance, so that you can use that in the curl call. Of course, you need to configure the IP addresses in the haproxy configuration then (no the hostname, otherwise you point haproxy to itself).

I have no idea what your end goal is, and what you plan on doing with this setup, so I can only make generic suggestion, you need to choose what works best for you out of those 3 suggestions.

1 Like

thanks. I have added SAN of haproxy to certificate on vault nodes. Let me know in case you its not a good idea