Debug HAProxy Socket Error Redis TLS

Help!
1

We have recently implemented tls and authentication in our redis setup, and are trying to get HAProxy back up and running, but are running into socket errors and can’t find any good documentation for how to troubleshoot the issues.
Since the traffic is encrypted captured packages has been difficult to easily decrypt, to find what response we are getting from our redis servers.

We have used many different configurations found online but none works.

This is our current config:

listen redis
  mode tcp
  bind :6379 ssl crt /path/to/server.crt ca-file /path/to/ca.crt
  option tcplog
  option tcp-check
  tcp-check send AUTH\ user\ password\r\n
  tcp-check expect string +OK
  tcp-check send PING\r\n
  tcp-check expect string +PONG
  tcp-check send info\ replication\r\n
  tcp-check expect string role:master
  tcp-check send QUIT\r\n
  tcp-check expect string +OK
  server redis-0 <IP>:6379 maxconn 1024 check check-ssl inter 1s ssl verify required ca-file /path/to/ca.crt
  server redis-1 <IP>:6379 maxconn 1024 check check-ssl inter 1s ssl verify required ca-file /path/to/ca.crt

This is what we get in /var/log/haproxy.log:

Jan 23 16:50:22 haproxy-0 haproxy[52253]: [WARNING]  (52253) : Former worker #1 (2210121) exited with code 0 (Exit)
Jan 23 16:50:22 haproxy-0 haproxy[2210848]: Server redis/redis-0 is DOWN, reason: Socket error, info: " at step 2 of tcp-check (expect string '+OK')", check duration: 3ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jan 23 16:50:22 haproxy-0 haproxy[2210848]: [WARNING]  (2210848) : Server redis/redis-0 is DOWN, reason: Socket error, info: " at step 2 of tcp-check (expect string '+OK')", check duration: 3ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jan 23 16:50:22 haproxy-0 haproxy[2210848]: [WARNING]  (2210848) : Server redis/redis-1 is DOWN, reason: Socket error, info: " at step 2 of tcp-check (expect string '+OK')", check duration: 4ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jan 23 16:50:22 haproxy-0 haproxy[2210848]: Server redis/redis-0 is DOWN, reason: Socket error, info: " at step 2 of tcp-check (expect string '+OK')", check duration: 3ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jan 23 16:50:22 haproxy-0 haproxy[2210848]: Server redis/redis-1 is DOWN, reason: Socket error, info: " at step 2 of tcp-check (expect string '+OK')", check duration: 4ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue

Would appreciate some guidance from someone who has successfully setup a working HAProxy config to redis with tls and auth.

We had no problem using HAProxy to redis before tls was implemented.
HAProxy version 2.4.24-0ubuntu0.22.04.1
Redis server v=7.4.1

2

Redis logs should tell you why.

3

After checking logs in Redis we find this error

3815:M 28 Jan 2025 09:14:18.958 # Error accepting a client connection: error:0A0000C7:SSL routines::peer did not return a certificate (addr=IPADDR:58110 laddr=IPADDR:6379)

When connecting from the haproxy host to redis using redis-cli we get no connection errors and can query the database

redis-cli --tls --cert /path/to/server.crt --key /path/to/server.key --cacert /path/to/ca.crt --user USER --pass PASSWD -h IPADDR -p 6379

The certs are owned by haproxy on the haproxy host and owned by redis on the redis host, and the server.crt, server.key and ca.crt are identical on both hosts with identical permissions. They are also stored in the exact same path on both hosts.

Continuing the troubleshooting but my instinct tells me that the issue is with haproxy and how it’s trying to connect, not providing the certs thats in the configuration.

4

I have also been troubleshooting if the certs are loaded wrong and tried multiple different ways of providing certs but nothing is solving the issue.
This is the latest update I have made

global
  log /dev/log	local0
  log /dev/log	local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
  stats timeout 30s
  user haproxy
  group haproxy
  daemon

  # Default SSL material locations
  ca-base /etc/ssl/certs
  crt-base /etc/ssl/private

listen redis
  mode tcp
  bind :6379 ssl crt /etc/ssl/private/server.pem ca-file /etc/ssl/certs/ca-certificates.crt
  option tcplog
  option tcp-check
  tcp-check connect port 6379 ssl
  tcp-check send AUTH\ USER\ PASSWD\r\n
  tcp-check expect string +OK
  tcp-check send info\ replication\r\n
  tcp-check expect string role:master
  tcp-check send QUIT\r\n
  tcp-check expect string +OK
  server redis-0 IPADDR:6379 maxconn 1024 check inter 10s ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt
  server redis-1 IPADDR:6379 maxconn 1024 check inter 10s ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt

Anyone got some suggestions?

5

Seems your Redis expects client TLS certificates not username and password for authentication. Or maybe both.

6

Correct, redis server expects the client certificate from the redis client.

You need to copy the client certificate and private key that you use on the redis-client to haproxy and configure haproxy to use this client certificate when connecting to the redis server.

However I would also suggest thinking about why you need to do all of this.

Why do you need to terminate TLS on haproxy in the first place? What benefit do you expect from haproxy that made you decide to terminate TLS on haproxy?

And considering that you no longer authenticate the client to the redis backend server, but you are authenticating haproxy to the redis server, is this really what you want?

Perhaps you can disable TLS termination, authenticating the redis client directly to redis server, and for the health checking / master role detecting you use either another port without SSL client cert requirement or use a dedicated client SSL certificate solely for the health check purposes (but not for production traffic).

7

We are using the same certificates.
I am expecting haproxy to load and provide the cert if we have listed it next to “bind”.
And we still need to authenticate to redis when we have connected, see “send AUTH”.

Everything in our environment needs to use certificates and authentication when connecting to Redis, HAProxy is no exception. There are other guides out there that use a similar configuration as we do, we just can’t get ours to work, and I want to know why HAProxy isn’t providing the certificate that is needed when it is configured.

We will not make any changes on the redis side of things.

8

The cert on the bind line is for haproxy as a server, it gets sent to the clients that connect to haproxy. You need to send the client cert to the backend which is done on the server line via the crt atribute, see
Client certificate authentication | HAProxy config tutorials for example

9

You are amazing! It is working now after I only added “ssl crt /etc/ssl/redis/server.crt” to the server line. I even had that guide open but didn’t look good enough.
Thank you very much!

10

It works, but you also have bypassed SSL authentication in this configuration.

Now every redis-client can connect without certificate through haproxy, because haproxy will provide the client certificate without verifying it from client.

At the very least you need to put verify required on your bind statement, to actually authenticate the client.

A much simpler and more secure way to do this is like I said just passing it through. Then the only cryptographic requirement on haproxy is for health checks.

And no, you don’t have to change any client or server configuration for that, unless you want to.

11

We need HAProxy to verify which redis server is the master, how would HAProxy be able to do that without authenticating. We use HAProxy for other systems than redis to connect to the current redis master, redis slave and sentinels do not use HAProxy.
I will test adding “verify required” to the bind line and do some tests.

Edit:
Your input was valid and “verify required” on the bind was needed. Thank you for your help!
We are happy with our configuration now.

12

Haproxy continues to authenticate, but for healthchecks only.

Healthcheck SSL settings can be configured independently of the pass through traffic, in other words you can enable SSL settings for health checks which do NOT affect client traffic.

This way you can achieve end-to-end SSL without MITM’ing redis traffic, while still detecting which redis server is master and only using that.

Because in this case the SSL traffic would be end-to-end, you would not be able to downgrade security on haproxy, for example to due a configuration mistake (although if the private key is the same everywhere, this is not true either).

Of course the SSL settings move from generic server SSL settings to health check specific SSL settings.

Ok good.