Haproxy with hashicorp vault

Help!
1

Hello Everyone, I am becoming crazy configuring haproxy for hashicort vault.
I am using ubuntu20.04 with tcp pass thru.
If on the backend servers I install apache it works fine.
If on the backend I use vault it does non works: An error occurred during a connection to vault.nivolapiemonte.it:9443. PR_END_OF_FILE_ERROR
I am using same certificates in both cases.

resolvers vault
nameserver dnsmasq 127.0.0.1:5353

frontend default9443
mode tcp
bind *:9443
tcp-request inspect-delay 5s
acl is_vault req.ssl_sni -i vault.nivolapiemonte.it
use_backend vault if is_vault

backend vault
mode tcp
option ssl-hello-chk

option httpchk GET /v1/sys/health

server node0 10.138.219.69:8200 check maxconn 20 ssl ca-file /etc/haproxy/ssl/niv-ca.pem

server s1 podto1-vault01.nivolapiemonte.it:8200 resolvers vault resolve-prefer ipv4

Please help me.
Ignazio

2

Into your frontend add tcp-request content accept if { req.ssl_hello_type 1 }, otherwise every request is delayed for 5 seconds for no reason.

For the backend servers, I assume port 8200 is a SSL port. The frontend is already accepting SSL traffic (port 9443 and you are routing based on SNI so I’m assuming we are talking about HTTPS) that you pass through.

So the ssl keyword in your backend server node0 configuration is certainly wrong: it would encrypt traffic that is already encrypted.

When you fixed this and it still doesn’t work, remove health checks temporarily. Your backend server may need additional things like correct SNI and host headers, so you want to make confirm that whether the only remaining issue is health check related or not.

1 Like
3

Thanks for reply.
Now my backend is the following:

backend vault
mode tcp
option ssl-hello-chk
server node0 10.138.154.69:8200
But it still not working:
Secure Connection Failed

An error occurred during a connection to vault.nivolapiemonte.it:9443. PR_END_OF_FILE_ERROR

It works only connecting directly to the backend without haproxy :frowning:

  1. List item
4

Check haproxy logs. Are you sure this port is reachable from haproxy? What does curl -vk https://10.138.154.69:8200 show?

5

igna@CSILTF9YLFL3:~$ curl -vk https://10.138.154.69:8200

  • Trying 10.138.154.69:8200…
  • TCP_NODELAY set
  • Connected to 10.138.154.69 (10.138.154.69) port 8200 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Request CERT (13):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Certificate (11):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: C=IT; ST=Torino; L=Torino; O=CSI PIEMONTE; CN=*.nivolapiemonte.it
  • start date: May 3 14:16:36 2022 GMT
  • expire date: May 3 14:16:36 2023 GMT
  • issuer: C=IT; ST=Bergamo; L=Ponte San Pietro; O=Actalis S.p.A.; CN=Actalis Organization Validated Server CA G3
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x55afd6b2e840)

GET / HTTP/2
Host: 10.138.154.69:8200
user-agent: curl/7.68.0
accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

  • Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
    < HTTP/2 307
    < cache-control: no-store
    < content-type: text/html; charset=utf-8
    < location: /ui/
    < strict-transport-security: max-age=31536000; includeSubDomains
    < content-length: 40
    < date: Mon, 24 Oct 2022 15:22:06 GMT
    <
    Temporary Redirect.

  • Connection #0 to host 10.138.154.69 left intact

6

The above is the result from my client where I am using the firefox browser.
If I try from the haproxy server the result is very different:

oot@podvc-haproxyser01:~# curl -vk https://10.138.154.69:8200/

  • Trying 10.138.154.69:8200…
  • TCP_NODELAY set
  • Connected to 10.138.154.69 (10.138.154.69) port 8200 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • OpenSSL SSL_connect: Connection reset by peer in connection to 10.138.154.69:8200
  • Closing connection 0
    curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 10.138.154.69:8200

So seems the client works fine while the haproxy has got some issue

Ignazio

7

Exactly, this confirms that the haproxy box is unable to connect to the backend server.

8

:slightly_frowning_face:
I wonder if there is some solution.
I also tried from another haproxy (centos) and from the previous (ubuntu 20.04): it does not work in both cases.
Ignazio

9

If I install on the backend apache with same certificates, haproxy works fine. So I think vault requires some special configuration.

10

I found the problem: the problem is the communication from haproxy to vault on port 8200.
If I run vault on 443 it works.
So I must check with my collegues what kind of firewall there is between haproxy and vault.
Ignazio

1 Like
11

The issue was caused by the firewall.
Thanks
Ignazio

2 Likes