Using pgsql-check with TLS results in 'SSL handshake failure'

Help!
1

I’m trying to use the pgsql-check for checking my postgres node backends. They provide SSL endpoints only. My configuration looks like this:

global
  # Default SSL material locations
  ca-base /usr/local/etc/haproxy/ca
  crt-base /usr/local/etc/haproxy/cert

  ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
  ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:!ECDHE-RSA-AES256-SHA384:!ECDHE-RSA-AES128-SHA256:!ECDHE-RSA-AES256-SHA:!ECDHE-RSA-AES128-SHA:!DHE-RSA-AES128-SHA256:!DHE-RSA-AES128-SHA:!DHE-RSA-AES256-SHA256:!DHE-RSA-AES256-SHA:!ECDHE-ECDSA-DES-CBC3-SHA:!ECDHE-RSA-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!AES128-GCM-SHA256:!AES256-GCM-SHA384:!AES128-SHA256:!AES256-SHA256:!AES128-SHA:!AES256-SHA:!DES-CBC3-SHA:!DSS

  ssl-default-server-options ssl-min-ver TLSv1.2 no-tls-tickets
  ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:!ECDHE-RSA-AES256-SHA384:!ECDHE-RSA-AES128-SHA256:!ECDHE-RSA-AES256-SHA:!ECDHE-RSA-AES128-SHA:!DHE-RSA-AES128-SHA256:!DHE-RSA-AES128-SHA:!DHE-RSA-AES256-SHA256:!DHE-RSA-AES256-SHA:!ECDHE-ECDSA-DES-CBC3-SHA:!ECDHE-RSA-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!AES128-GCM-SHA256:!AES256-GCM-SHA384:!AES128-SHA256:!AES256-SHA256:!AES128-SHA:!AES256-SHA:!DES-CBC3-SHA:!DSS

  ssl-server-verify none

  tune.ssl.default-dh-param 2048
  tune.h2.initial-window-size 1048576 # https://www.haproxy.com/documentation/hapee/1-8r1/traffic-management/enable-http2-protocol/

defaults
  log global
  mode http
  option httpchk GET /

  timeout connect 5000
  timeout client  30000
  timeout server  30000
  default-server init-addr libc,none

listen postgres
  bind *:5432
  balance roundrobin
  option pgsql-check user myuser

  server db_1 qa-db-1:5432 check inter 10s fastinter 2s downinter 60s fall 3 rise 10 ssl verify required ca-file my-ca.crt
  server db_2 qa-db-2:5432 check inter 10s fastinter 2s downinter 60s fall 3 rise 10 ssl verify required ca-file my-ca.crt backup

The checks fail with the following log output:

[NOTICE]   (8) : New worker #1 (10) forked
[WARNING]  (10) : Server postgres/db_1 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 6ms. 0 active and 1 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING]  (10) : Backup Server postgres/db_2 is DOWN, reason: Layer6 invalid response, info: "SSL handshake failure", check duration: 11ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[NOTICE]   (10) : haproxy version is 2.4.1-1ce7d49
[ALERT]    (10) : proxy 'postgres' has no server available!

Checking the postgres log shows:

2021-08-02 18:54:10.987 CEST [14096] [unknown]@[unknown] LOG:  could not accept SSL connection: no suitable signature algorithm
2021-08-02 18:54:10.987 CEST [14096] [unknown]@[unknown] DEBUG:  SSL connection from "(anonymous)"
2021-08-02 18:54:10.987 CEST [14096] [unknown]@[unknown] DEBUG:  shmem_exit(0): 0 before_shmem_exit callbacks to make
2021-08-02 18:54:10.987 CEST [14096] [unknown]@[unknown] DEBUG:  shmem_exit(0): 0 on_shmem_exit callbacks to make
2021-08-02 18:54:10.987 CEST [14096] [unknown]@[unknown] DEBUG:  proc_exit(0): 1 callbacks to make
2021-08-02 18:54:10.987 CEST [14096] [unknown]@[unknown] DEBUG:  exit(0)
2021-08-02 18:54:10.987 CEST [14096] [unknown]@[unknown] DEBUG:  shmem_exit(-1): 0 before_shmem_exit callbacks to make
2021-08-02 18:54:10.987 CEST [14096] [unknown]@[unknown] DEBUG:  shmem_exit(-1): 0 on_shmem_exit callbacks to make
2021-08-02 18:54:10.987 CEST [14096] [unknown]@[unknown] DEBUG:  proc_exit(-1): 0 callbacks to make

Manually executing the ssl handshake works fine:

# echo "" | openssl s_client -starttls postgres -CAfile /etc/haproxy/ca/my-ca.crt  -connect 10.0.101.6:5432 -showcerts
CONNECTED(00000003)
Can't use SSL_get_servername
[…]
verify return:1
---
Certificate chain
[…]
---
No client certificate CA names sent
Peer signature type: Ed25519
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1042 bytes and written 687 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 253 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
DONE

Am I doing something wrong here or could anyone give me a hint how to troubleshoot this, please? HAProxy version that I’m using is 2.4.1-1ce7d49.

2

Postgres doesn’t provide implicit SSL endpoints, but it’s startssl (explicit via postgresql negotiation, also see your openssl command).

All the ssl related configuration on the server line is therefor wrong, you will have to remove it completely (ssl verify required ca-file my-ca.crt).

I don’t know at which point postgresql wants to upgrade via startssl, but certainly before the authentication, so I suggest you give it a try without the user (just option pgsql-check).

1 Like
3

Thank you for your response. You’re right, I didn’t notice the startssl aspect before. I gave it a try and removed the flags you mentioned. Unfortunately, the user name does not really seem to be as optional as the documentation says:

[ALERT]    (8) : parsing [/usr/local/etc/haproxy/haproxy.cfg:48] : 'option pgsql-check' expects 'user <username>' as argument.

So I added it again and left out the ssl part. The result is a different error:

[WARNING]  (10) : Server postgres/db_1 is DOWN, reason: Layer7 invalid response, info: "FATAL", check duration: 4ms. 0 active and 1 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue

This is apparently caused by the missing pg_hba entry for connections without TLS:

2021-08-02 23:54:34.550 CEST [23716] myuser@mydb FATAL:  no pg_hba.conf entry for host "10.0.101.4", user "myuser", database "mydb", SSL off

But it seems wrong to add this entry just for the health check. Let’s assume I’d do that - could I decouple the check and the backend ssl-wise? Else, removing the ssl part of the backend flags would leave me open to MITM, wouldn’t it? HAProxy does no validation of the backend identity anymore then. That’s probably okay for a health check but not for the backend connection itself.

Maybe my approach is wrong and I should rather go with a tcp passthrough. I’m surprised my use case seems to be so exotic. People don’t use TLS to their DBs? ;/

Edit: I just realized that it doesn’t make much sense what I’m doing, because HAproxy cannot authenticate to the backend anyway. I must take the tcp passthrough path then. Thank you very much for your help anyway. :slight_smile:

4

You are already using the TCP passthrough approach, there is no other way, as haproxy does not implement the postgres protocol.

You can use SSL/TLS end to end, and have your client authenticate the backend. But you cannot make haproxy talk the postgresql protocol or add an additional SSL layer from haproxy. I also don’t see how that would make sense, why would you secure the proxy → backend layer and leave the client side unprotected, that doesn’t make sense to me.

I suggest you downgrade to a TCP healthcheck.

5

Because the proxy is sitting with the app and they communicate on localhost. Aim was to avoid fiddling with the JVM keystore for importing my CA. But you’re right, I guess I’ll go with the tcp healthcheck instead. Thank you!

6

This would require a proxy actually capable of speaking the postgresql protocol, which haproxy is not and probably never will.

7

@tumbleweed tumbleweed

Hey friend, did you manage to overcome this problem?

I searched the internet and found nothing, not even official documentation, it doesn’t go into much detail.