SSL Handshake failures when enabling Proxy-protocol

Help!
1

Hello,

We use a HAProxy loadbalancer in TCP mode with behind it a HAProxy reverse proxy in HTTP mode. (HAProxy version 2.4.22-f8e3218 2023/02/14)

–>HAProxy-LBS—>HAProxy-RPX—>webserver

After enabling the proxy-protocol between the loadbalancer and reverse-proxy we see “SSL handshake failure” errors every 2 seconds(lbs alive check…) in the HAProxy log of the reverse-proxy.

Loadbalancer backend config

backend be_<vip_name>_443
    balance leastconn
    mode tcp
    server server1 10.0.0.1:10444 send-proxy-v2 check
    server server2 10.0.0.2:10444 send-proxy-v2 check

Reverse-proxy:

bind *:10444 ssl crt <some cert.pem< ca-file <ca-bundle.crt> verify optional accept-proxy

If I disable the proxy-protocol on the LBS and RPX machines the errors don’t appear anymore.

Any suggestions how to debug this issue or any solutions?

2

Haproxy is supposed to use the proxy protocol for health checks as well, when it’s enabled.

However you can force it with the check-send-proxy parameter.

3

Thanks Lukas for your answer.

I have tried to add the option but with no success.

server node1 10.0.0.1:10444 check check-send-proxy send-proxy-v2
4

Then the request is not coming from LBS but somewhere else.

Maybe you have a monitoring systems that probes RPX and therefore causes this issue?

5

Thanks, but I only get those messages when enabling proxy protocol by adding “accept-proxy” and “send-proxy-v2”, If I remove the check keyword in the loadbalancer it also stops.

Also the source IP is from the loadbalancer.

6

There is not enough information in here to diagnose the issue.

If I’d need to guess and you are sure that there is nothing else involved, it possible old haproxy instances are running in the background with older configurations. Stop and kill all haproxy processes on both LBS and RBX to make sure. Compare PIDs in the logs to confirm.

Otherwise capture the offending traffic and analyse it manually, or provide full configuration and full log outputs.

7

Thanks Lukas.

I am very very sure the errors are generating because of the tcp-check the LBS is doing. Everything else is working. The strange thing is that I should expect this behaviour also when the proxy protocol is disabled, because of an unfinished SSL handshake, but that isn’t happening.

Hereby the configs anonymised, maybe you can find something in it:

LBS config:

global
    log 127.0.0.1:514 local0

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid

    maxconn     65536
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    tune.ssl.default-dh-param 2048

    # utilize system-wide crypto-policies
    #ssl-default-bind-ciphers PROFILE=SYSTEM
    #ssl-default-server-ciphers PROFILE=SYSTEM

defaults
   log global
   mode http
   option tcplog
   option dontlognull
   timeout connect 5000
   timeout client 900s
   timeout server 900s

frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

frontend fe_vhost
    mode tcp
    bind 10.0.1.1:443

    default_backend be_rpx

backend be_rpx
    balance leastconn
    mode tcp
    server node1 10.0.0.1:10444 send-proxy check check-send-proxy
    server node2 10.0.0.2:10444 send-proxy check check-send-proxy

RPX config anonymised:

global
    log 127.0.0.1:514 local0

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid

    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    ca-base /etc/ssl/hap_ca_base/

    tune.ssl.default-dh-param 2048

    # utilize system-wide crypto-policies
    #ssl-default-bind-ciphers PROFILE=SYSTEM
    #ssl-default-server-ciphers PROFILE=SYSTEM
defaults
   log global
   mode http
   option httplog
   option dontlognull
   timeout connect 5000
   timeout client 300s
   timeout server 300s
   maxconn 2000
   errorfile 403 /etc/haproxy/errors/403.http
   errorfile 404 /etc/haproxy/errors/404.http
   errorfile 500 /etc/haproxy/errors/500.http
   errorfile 503 /etc/haproxy/errors/503.http


frontend stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats admin if LOCALHOST

frontend fe_rpx

    mode http
    bind *:10081 accept-proxy

    bind *:10444 ssl crt /etc/ssl/hap_vhosts/cert.pem ca-file ca-bundle.crt verify optional accept-proxy

    acl acl_is_ssl dst_port 10444

    #Redirects
    http-request redirect scheme https unless { ssl_fc }

    #converts http to https in "Location" header
    acl resp-is-location-hdr res.hdr(Location) -m found
    acl resp-is-content-location-hdr res.hdr(Content-Location) -m found
    http-response replace-header  Location http://(.*) https://\1 if resp-is-location-hdr acl_is_ssl
    http-response replace-header  Content-Location http://(.*) https://\1 if resp-is-content-location-hdr acl_is_ssl

    #Actions (HTTPS)
    default_backend be_webserver

backend be_webserver
    balance roundrobin
    server webserver 10.0.10.1:80 check

Logs RPX:

Sep 10 20:32:06 localhost haproxy[2477679]: 10.0.0.1:39630 [10/Sep/2024:20:32:06.816] stats stats/<STATS> 0/0/0/0/0 200 2408 - - LR-- 1/1/0/0/0 0/0 "GET /stats;csv HTTP/1.1"
Sep 10 20:32:29 localhost haproxy[2477679]: 10.1.0.202:63557 [10/Sep/2024:20:32:29.133] fe_rpx~ be_rpx/<NOSRV> 0/-1/-1/-1/0 503 7714 - - SC-- 1/1/0/0/0 0/0 "GET / HTTP/1.1"
Sep 10 20:32:30 localhost haproxy[2477679]: 10.0.0.17:53966 [10/Sep/2024:20:32:30.397] fe_rpx/2: SSL handshake failure
8

Yeah, I’m thinking that there may be a subtle difference in how haproxy handles this situation.

You could probably upgrade the health check itself to a SSL heatlh check to workaround this, as I’m not sure if there really is a ultimate solution to the core problem.

To do this, add the keywords:

check-ssl verify none

I’d have to reproduce and check that this is really it and if newer release behave the same.

9

The check-ssl workaround does not appear to be effective, there are currently two options:

  • upgrade to a full HTTP check (proxy protocol → TLS → HTTP check)
  • downgrade to a TCP check without the proxy protocol

For the second workaround: we have to trick haproxy into not using proxy protocol for the health check though, as their does not seem to be a proper way. To do this, we specify the port again, which is redundant for the passing traffic but will make sure that the health check does not use the proxy protocol:

backend be_rpx
 balance leastconn
 mode tcp
 server node1 10.0.0.1:10444 send-proxy check port 10444
 server node2 10.0.0.2:10444 send-proxy check port 10444

None of this is particularly nice.

Bugs are already filed for those behaviours:

11

Thanks Lukas for your time and support!

I have implemented the second workaround and this solved the issue.

1 Like
12

I want you to know that we don’t see this behaviour on HA-Proxy version 1.8.27-493ce0b 2020/11/06…

13

It stilll there just different (1.8.27-493ce0b):

Connection error during SSL handshake

instead of:

SSL handshake failure