Connection reset for LDAPS

Help!
1

Hello!

I’m configuring HAProxy for LDAPs load balancing. Actually I’m replacing old HAProxy which is working well now with the new one.
Old one:

HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>

New one:

HAProxy version 2.4.22-f8e3218 2023/02/14 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.22.html
Running on: Linux 3.10.0-1160.90.1.el7.x86_64 #1 SMP Fri Mar 17 08:39:44 UTC 2023 x86_64

Config is pretty simple:

global
  chroot  /var/lib/haproxy
  daemon
  log  127.0.0.1 local2
  maxconn  4000
  pidfile  /var/run/haproxy.pid
  ssl-default-bind-ciphers  ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  ssl-default-bind-options  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  ssl-default-server-ciphers  ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  ssl-default-server-options  no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
  stats  socket /var/lib/haproxy/stats mode 600 level admin
  tune.ssl.default-dh-param  2048

defaults
  log  global
  mode  http
  option  httplog
  option  dontlognull
  option  http-server-close
  option  redispatch
  retries  3
  stats  enable
  timeout  http-request 1m
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 5m
  timeout  server 5m
  timeout  http-keep-alive 10s
  timeout  check 20s

frontend dap_base_prd_frontend
  bind *:636
  default_backend dap_base_prd_backend
  mode tcp
  option socket-stats
  option tcpka

backend dap_base_prd_backend
  balance roundrobin
  log global
  mode tcp
  option tcplog
  option tcpka
  option ssl-hello-chk
  server dapbaseprd01.dap 172.16.130.4:635 verify none inter 12000 rise 3 fall 3
  server dapbaseprd02.dap 172.16.130.6:635 verify none inter 12000 rise 3 fall 3

Old one don’t have different SSL “tuning” parameters like ciphers, but everything else is the same.

If I curl HAProxy locally it works:

curl -v https://dapldapprd.dap:636 --resolve dapldapprd.dap:636:10.195.114.69
* Added dapldapprd.dap:636:10.195.114.69 to DNS cache
* About to connect() to dapldapprd.dap port 636 (#0)
*   Trying 10.195.114.69...
* Connected to dapldapprd.dap (10.195.114.69) port 636 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: CN=dapldapprd.dap
*       start date: Apr 23 15:37:24 2020 GMT
*       expire date: Apr 23 15:37:24 2025 GMT
*       common name: dapldapprd.dap
*       issuer: CN=Puppet CA: dapmgmtprd01.dap
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: dapldapprd.dap:636
> Accept: */*
>
* Empty reply from server
* Connection #0 to host dapldapprd.dap left intact
curl: (52) Empty reply from server

If I do the same curl from any machine inside local network or in another network it reset connection:

 curl -v https://dapldapprd.dap:636 --resolve dapldapprd.dap:636:10.195.114.69
* Added dapldapprd.dap:636:10.195.114.69 to DNS cache
* About to connect() to dapldapprd.dap port 636 (#0)
*   Trying 10.195.114.69...
* Connected to dapldapprd.dap (10.195.114.69) port 636 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file

What it can be?
Any advice how to debug those things? In the logs I don’t see any errors or even access attempts to LDAP backend.

2

Check haproxy logs. You will see whether a request arrives or not. If it doesn’t even arrive, probably the IP address 10.195.114.69 is routed or intercepted somewhere else, before arriving at the haproxy box.

And alternative would be to tcpdump on the haproxy box to check whether requests from other boxes arrive or not:

tcpdump -i eth0 -pnvvs0 host 10.195.114.69 and port 636

Check your iptables/nftables configuration. Maybe there are redirects of port 636 to somewhere else?

From the configuration its not clear whether you intent to do health checking or not. There is no check keyword on the server lines, so in haproxy will not health check. But that is likely unrelated to the issue at hand.

Do you see incoming connects on haproxy from the clients in your network at all? If you don’t see anything, it means it doesn’t arrive.

Also check for old/obsolete haproxy instances on port 636, as in: stop haproxy and make sure all haproxy processes are gone. Sometimes during testing/configuration haproxy process are started manually without ever getting properly closed. They will then lurk in the background with old/wrong configurations and make wrong routing decisions.

3

Thank you for reply.

In tcpdump I see traffic coming in and replies with rejects.
I don’t see any rejects by firewall and no ICMP port unreachable messages.
HAProxy is in the network with “test” machine. And no other servers with same IP is there.

I don’t see anything in HAProxy logs at all if I test it remotely. Locally I see it forward traffic to backend.

Fun fact: I have exactly the same configuration for different backend (pure HTTPS) on port 554 and… this works everywhere.

Any ideas?