SSL handshake failure (no shared ciphers when there are multiple shared ciphers)

Help!
1

I have setup with Haproxy fronting 2 backend servers and TLS termination on Hproxy as well as TLS between haproxy and the backend. Everything is working fine, but for a specific client device. Whenever said device tries to connect, an error is thrown and the connection is closed during SSL handshake (right after client hello). Now the haproxy access log states the reason as no shared cipher, but using ssldump to capture the connection, I can see the client is listing its ciphers, and there are for sure shared ciphers between client and haproxy. Haproxy is running on docker, (LTS alpine 3.18).

Here are the ciphers that are in the Haproxy config:

ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-CHACHA20-POLY1305
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES256-SHA384
ECDHE-RSA-AES256-SHA384
ECDHE-ECDSA-AES128-SHA256
ECDHE-RSA-AES128-SHA256

And here is the client hello, note the ciphers:

New TCP connection #1: 172.16.16.59(36355) ↔ 21b9153213ab(7547)
1 1 0.1055 (0.1055) C>S V3.0(105) Handshake
ClientHello
Version 3.3
random[32]=
64 99 8f 51 5d 82 4c 51 e3 b8 d7 1f 49 92 60 4f
3b e5 f8 ec 6f 3f fe bc 62 85 43 cd b9 eb 5b d8
cipher suites
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_RC4_128_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
compression methods
NULL
extensions
signature_algorithms
1 2 0.1148 (0.0093) S>C V3.3(2) Alert
level fatal
value handshake_failure
1 0.1197 (0.0049) C>S TCP FIN
1 0.1229 (0.0031) S>C TCP FIN

Now if I’m not wrong, ECDHE-ECDSA-AES256-GCM-SHA384 is the same as TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 … A shared cipher, and the first one listed in the config.

Given the above, I’m no expert on TLS and cipher suites, therefore I don’t understand why Haproxy returns no shared cipher?

Now to be fair, I tried skipping Haproxy, letting the client communicate directly with the backend server, and that as well returns an error, stating SSL handshake failure as a reason (though it does not provide detail as to where the handshake failed, but I’m assuming the same thing as above). I realize this may be a problem with my client device, but just wanted to see if anyone might have some insight into this.

2

You’d have to provide the full configuration as well as the output of haproxy -vv

Here we need the full command line that you are using.

Yes, but it also requires a EC certificate. If you have a RSA certificate, the RSA equivalent suite must be used (fortunately with TLSv1.3 this is no longer the case).

SSL is complicated, full haproxy config, output of haproxy -vv as well as the full openssl command line are required at the very least to give a proper answer, but a tcpdump of the entire handshake may also be required.

3

Thanks @lukastribus for the response. From what you said, it seems the most likely explanation is this:

Yes, but it also requires a EC certificate. If you have a RSA certificate, the RSA equivalent suite must be used (fortunately with TLSv1.3 this is no longer the case).

I do know that my certificate is RSA. And I can clearly see that the client and server have no shared cipher with RSA in the name. It would make sense since this is the only client device that is facing problems. I also know that it supports up to TLSv1.2, and other devices that I’ve tried support TLSv1.3
haproxy -vv output:

HAProxy version 2.8.1-a90123a 2023/07/03 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.1.html
Running on: Linux 3.10.0-957.el7.x86_64 #1 SMP Thu Nov 8 23:39:32 UTC 2018 x86_64
Build options :
  TARGET  = linux-musl
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=4).
Built with OpenSSL version : OpenSSL 3.1.1 30 May 2023
Running on OpenSSL version : OpenSSL 3.1.1 30 May 2023
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.3.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 12.2.1 20220924

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [BWLIM] bwlim-in
        [BWLIM] bwlim-out
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

Full haproxy.cfg

global
  maxconn 5000
  log xx.xx.xx.xx:514 local0
  user haproxy
  group haproxy
  stats socket /run/haproxy/admin.sock user haproxy group haproxy mode 660 level admin
  ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
  ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
  timeout connect 10s
  timeout client 30s
  timeout server 300s
  log global
  option httpslog
  error-log-format "%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc"
  mode http
  maxconn 2000

  errorfile 400 /etc/haproxy/errors/400.http
  errorfile 403 /etc/haproxy/errors/403.http
  errorfile 408 /etc/haproxy/errors/408.http
  errorfile 500 /etc/haproxy/errors/500.http
  errorfile 502 /etc/haproxy/errors/502.http
  errorfile 503 /etc/haproxy/errors/503.http
  errorfile 504 /etc/haproxy/errors/504.http

# Enable stats page
listen stats
  bind *:1936 ssl crt /etc/haproxy/certs/cert.pem
  mode http
  log global

  maxconn 10

  stats enable
  stats hide-version
  stats refresh 30s
  stats show-node
  stats auth admin:4PXU78CRnuce
  stats uri /haproxy?stats

  ## Restrict access to /stats page to our IPs only
  acl network_allowed src x.x.x.x/24 x.x.x.x/24
  acl restricted_page path_beg /haproxy
  http-request deny if restricted_page !network_allowed

# Http fontend
frontend proxy-http
  bind *:80
  bind *:443 ssl crt /etc/haproxy/certs
  bind *:7547 ssl crt /etc/haproxy/certs
  bind *:7557 ssl crt /etc/haproxy/certs
  bind *:7567 ssl crt /etc/haproxy/certs
  http-request redirect scheme https unless { ssl_fc }
  mode http
  # Fileset
  acl fileset_acl hdr(host) -i vps.o.infonas.com
  use_backend fileset-docker if fileset_acl
  # Genieacs
  acl genieacs_servers hdr(host) -i acs.o.infonas.com
  acl genieacs_ui dst_port 443
  acl genieacs_cwmp dst_port 7547
  acl genieacs_nbi dst_port 7557
  acl genieacs_fs dst_port 7567
  use_backend genieacs_ui if genieacs_servers genieacs_ui
  use_backend genieacs_cwmp if genieacs_cwmp
  use_backend genieacs_nbi if genieacs_nbi
  use_backend genieacs_fs if genieacs_fs

# Fileset Containers backend
backend fileset-docker
  balance roundrobin
  option forwardfor
  http-request set-header X-Forwarded-Port %[dst_port]
  http-request add-header X-Forwarded-Proto https if { ssl_fc }
  option httpchk
  http-check connect
  http-check send meth HEAD uri /premium/login/ ver  HTTP/1.1 hdr host localhost
  http-check expect status 200-399
  server fileset02 x.x.x.x:80 check

# Genieacs Containers Backend
backend genieacs_ui
  balance roundrobin
  cookie SERVER insert indirect nocache
  server genieacs1 x.x.x.x:3000 ssl verify none check cookie genieacs1
  server genieacs2 x.x.x.x:3000 ssl verify none check cookie genieacs2

backend genieacs_cwmp
  balance roundrobin
  cookie SERVER insert indirect nocache
  server genieacs1 x.x.x.x:7547 ssl verify none check cookie genieacs1
  server genieacs2 x.x.x.x:7547 ssl verify none check cookie genieacs2

backend genieacs_nbi
  balance roundrobin
  cookie SERVER insert indirect nocache
  server genieacs1 x.x.x.x:7557 ssl verify none check cookie genieacs1
  server genieacs2 x.x.x.x:7557 ssl verify none check cookie genieacs2

backend genieacs_fs
  balance roundrobin
  cookie SERVER insert indirect nocache
  server genieacs1 x.x.x.x:7567 ssl verify none check cookie genieacs1
  server genieacs2 x.x.x.x:7567 ssl verify none check cookie genieacs2

ssldump command to capture and debug handshake:

ssldump -A -d -i any host <CLIENT_IP>

4

Either the clients starts supporting RSA ciphers in TLSv1.2 or the client starts supporting TLSv1.3 or you deploy an EC certificate on haproxy.

Can you show the client hello of this client? It seems like a “exotic” configuration supporting only EC ciphers with TLSv1.2.

5

The hello in the topic post is from this client. To make things clearer this client is an Optical Network Terminal (ONT), so it’s not your average client device.

Here’s the client hello from above

New TCP connection #1: 172.16.16.59(36355) ↔ 21b9153213ab(7547)
1 1 0.1055 (0.1055) C>S V3.0(105) Handshake
ClientHello
Version 3.3
random[32]=
64 99 8f 51 5d 82 4c 51 e3 b8 d7 1f 49 92 60 4f
3b e5 f8 ec 6f 3f fe bc 62 85 43 cd b9 eb 5b d8
cipher suites
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA256
TLS_RSA_WITH_AES_256_CBC_SHA256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
TLS_RSA_WITH_RC4_128_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_EMPTY_RENEGOTIATION_INFO_SCSV
compression methods
NULL
extensions
signature_algorithms
1 2 0.1148 (0.0093) S>C V3.3(2) Alert
level fatal
value handshake_failure
1 0.1197 (0.0049) C>S TCP FIN
1 0.1229 (0.0031) S>C TCP FIN
6

There are overlaps between the two (GCM ciphers), so the explanation that the ONT doesn’t support RSA ciphers allowed by haproxy doesn’t make sense.

Capture the entire SSL handshake with tcpdump in a pcap file and provide it.

7

tcpdump isn’t showing useful information, the client hello above is the only useful thing I’ve been able to extract so far. That being said, not sure how to attach a file here :sweat_smile:

here’s how Haproxy is logging it with the non standard error log format:

2023-07-26T15:28:45+00:00 10.1.4.11 haproxy[26]: 172.16.16.59:52524 [26/Jul/2023:15:28:45.503] proxy-http~ proxy-http/<NOSRV> -1/-1/-1/-1/0 0 0 - - PR-- 1/1/0/0/0 0/0 "<BADREQ>" 34/000000000A0000C1/0/0/0 -/TLSv1.2/(NONE)

And here is the default error log format:

2023-07-26T15:38:25+00:00 10.1.4.12 haproxy[42]: 172.16.16.59:44100 [26/Jul/2023:15:38:25.503] proxy-http/3: SSL handshake failure (error:0A0000C1:SSL routines::no shared cipher)
2023-07-26T15:39:05+00:00 10.1.4.12 haproxy[53]: 172.16.16.59:35802 [26/Jul/2023:15:39:05.666] proxy-http/3: SSL handshake failure (error:0A0000C1:SSL routines::no shared cipher)

And here’s output from ssldump again (just the same client hello repeated several times):

New TCP connection #1: 172.16.16.59(36002) <-> 10.1.4.10(7547)
1 1  0.0922 (0.0922)  C>S V3.0(105)  Handshake
      ClientHello
        Version 3.3
        random[32]=
          64 c1 13 8f 13 4c 24 18 2e 79 14 08 1d 0a cf 68
          e5 7d 1d ed b1 5f 00 81 3e 72 b2 9d 50 28 d6 0c
        cipher suites
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_RC4_128_SHA
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_EMPTY_RENEGOTIATION_INFO_SCSV
        compression methods
                  NULL
        extensions
          signature_algorithms
        ja3 string: 771,51-53-57-60-61-103-107-5-49171-49172-49187-49188-49195-49196-49199-49200-255,13,,
        ja3 fingerprint: 79b54cb52a6a294636100dc777d48821
1 2  0.0927 (0.0005)  S>C V3.3(2)  Alert
    level           fatal
    value           handshake_failure
1    0.0931 (0.0003)  S>C  TCP FIN
1    0.0975 (0.0044)  C>S  TCP FIN
New TCP connection #2: 172.16.16.59(42623) <-> 10.1.4.10(7547)
2 1  0.0045 (0.0045)  C>S V3.0(105)  Handshake
      ClientHello
        Version 3.3
        random[32]=
          64 c1 13 96 d8 1a df c5 c8 c2 a8 b8 42 61 18 4e
          5a aa 55 d5 31 0b 61 01 8b 86 ad ee 3f 33 79 0c
        cipher suites
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_RC4_128_SHA
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_EMPTY_RENEGOTIATION_INFO_SCSV
        compression methods
                  NULL
        extensions
          signature_algorithms
        ja3 string: 771,51-53-57-60-61-103-107-5-49171-49172-49187-49188-49195-49196-49199-49200-255,13,,
        ja3 fingerprint: 79b54cb52a6a294636100dc777d48821
2 2  0.0053 (0.0008)  S>C V3.3(2)  Alert
    level           fatal
    value           handshake_failure
2    0.0056 (0.0003)  S>C  TCP FIN
2    0.0091 (0.0034)  C>S  TCP FIN
New TCP connection #3: 172.16.16.59(33512) <-> 10.1.4.10(7547)
3 1  0.0043 (0.0043)  C>S V3.0(105)  Handshake
      ClientHello
        Version 3.3
        random[32]=
          64 c1 13 aa a4 44 76 0d 55 b6 5f 47 f6 d1 1b 3c
          47 9f 21 2b 1a 9a 7e e4 ff 4b 3e 6a e1 b4 24 d7
        cipher suites
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_RC4_128_SHA
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_EMPTY_RENEGOTIATION_INFO_SCSV
        compression methods
                  NULL
        extensions
          signature_algorithms
        ja3 string: 771,51-53-57-60-61-103-107-5-49171-49172-49187-49188-49195-49196-49199-49200-255,13,,
        ja3 fingerprint: 79b54cb52a6a294636100dc777d48821
3 2  0.0047 (0.0004)  S>C V3.3(2)  Alert
    level           fatal
    value           handshake_failure
3    0.0053 (0.0005)  S>C  TCP FIN
3    0.0085 (0.0031)  C>S  TCP FIN
New TCP connection #4: 172.16.16.59(44100) <-> 10.1.4.10(7547)
4 1  0.0069 (0.0069)  C>S V3.0(105)  Handshake
      ClientHello
        Version 3.3
        random[32]=
          64 c1 13 c0 76 ae 31 7c 17 f9 b4 19 cb d0 76 1a
          df a6 96 d2 82 86 78 6b 36 e2 3f cb fa 70 ba fd
        cipher suites
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_RC4_128_SHA
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_EMPTY_RENEGOTIATION_INFO_SCSV
        compression methods
                  NULL
        extensions
          signature_algorithms
        ja3 string: 771,51-53-57-60-61-103-107-5-49171-49172-49187-49188-49195-49196-49199-49200-255,13,,
        ja3 fingerprint: 79b54cb52a6a294636100dc777d48821
4 2  0.0074 (0.0004)  S>C V3.3(2)  Alert
    level           fatal
    value           handshake_failure
4    0.0078 (0.0004)  S>C  TCP FIN
4    0.0113 (0.0034)  C>S  TCP FIN
New TCP connection #5: 172.16.16.59(35802) <-> 10.1.4.10(7547)
5 1  0.0054 (0.0054)  C>S V3.0(105)  Handshake
      ClientHello
        Version 3.3
        random[32]=
          64 c1 13 e8 53 47 a4 28 34 95 f5 ea c6 c0 7c d0
          00 11 84 bb e9 09 c7 cf 34 d1 e0 dd fd 33 0c a0
        cipher suites
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_RC4_128_SHA
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_EMPTY_RENEGOTIATION_INFO_SCSV
        compression methods
                  NULL
        extensions
          signature_algorithms
        ja3 string: 771,51-53-57-60-61-103-107-5-49171-49172-49187-49188-49195-49196-49199-49200-255,13,,
        ja3 fingerprint: 79b54cb52a6a294636100dc777d48821
5 2  0.0059 (0.0004)  S>C V3.3(2)  Alert
    level           fatal
    value           handshake_failure
5    0.0065 (0.0006)  S>C  TCP FIN
5    0.0094 (0.0029)  C>S  TCP FIN
New TCP connection #6: 172.16.16.59(38142) <-> 10.1.4.10(7547)
6 1  0.0044 (0.0044)  C>S V3.0(105)  Handshake
      ClientHello
        Version 3.3
        random[32]=
          64 c1 14 42 0c eb 59 8e 02 02 ba 9e eb 2c b0 57
          84 7a 5d 99 aa 7e fb 4b b9 2d ce 86 10 aa ca eb
        cipher suites
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA256
        TLS_RSA_WITH_AES_256_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA256
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA256
        TLS_RSA_WITH_RC4_128_SHA
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
        TLS_EMPTY_RENEGOTIATION_INFO_SCSV
        compression methods
                  NULL
        extensions
          signature_algorithms
        ja3 string: 771,51-53-57-60-61-103-107-5-49171-49172-49187-49188-49195-49196-49199-49200-255,13,,
        ja3 fingerprint: 79b54cb52a6a294636100dc777d48821
6 2  0.0074 (0.0030)  S>C V3.3(2)  Alert
    level           fatal
    value           handshake_failure
6    0.0077 (0.0002)  S>C  TCP FIN
6    0.0111 (0.0033)  C>S  TCP FIN
8

Something is wrong here.

The log lines your shared come from 3 different PID’s.

Triple check that you don’t have multiple old haproxy running in the background, because they could be running with an old configuration.

Stop haproxy, make sure no haproxy processes are left running and start it again.

If that is not it, I’m sending you an upload form for the pcap file.

9

No, the pids are to be expected. We are not running a conventional setup. We have 2 Haproxy instances running in 2 different containers and on 2 different hosts. We’ve set them up in a HA active/passive setup with keepalived. Rsyslog is receiving logs from both, because I’ve been restarting them a few times and the VIP gets assigned to a different one each time I restart.

10

What I’m noticing is that the SSL client hello handshake version (the “envelope” one if you will) has SSLv3.0 in it (the real client hello is TLSv1.2, but there are 2 versions present in each hello and the former is often older for backwards compatibility).

I assume the ssl library on the client is not openssl based, because openssl 0.9.8 had SSLv2 and 1.0.0 already used TLSv1.0 here, so I cannot reproduce the client_hello of your client easily (I would have to replay the packet from tcpdump with something like tcpreplay, which is quite complicated).

Considering that you are using bleeding edge openssl 3.1 which again raised security default, I suggest to drop the seclevel:

https://www.openssl.org/docs/man3.1/man3/SSL_CTX_set_security_level.html

To do this, edit the ssl-default-bind-ciphers parameter in haproxy by appending :@SECLEVEL=0.

11

I just changed the security level based on your suggestion. Still getting the same result.

12

Ok, I spend a lot more time on this than I wanted to, but at least I learned a lot things myself.

So first of all, to best replay the client hello of the ONT I just used python (I will note this here because I will need the snippet at some point, so at least I don’t have to research it all over again - should be easily portable to python3):

$ python2

import socket
import time

b = \
"\x16\x03\x00\x00\x69\x01\x00\x00\x65\x03\x03\x64\xc2\x0e\x8c\xd1" \
"\x66\xdb\x75\x22\x97\xc5\x4a\x87\x8d\x06\x33\xaf\xe8\xfc\x8d\xd3" \
"\x0e\x68\x43\xc4\x63\x64\xe7\xd0\xa9\xd2\x87\x00\x00\x22\x00\x33" \
"\x00\x35\x00\x39\x00\x3c\x00\x3d\x00\x67\x00\x6b\x00\x05\xc0\x13" \
"\xc0\x14\xc0\x23\xc0\x24\xc0\x2b\xc0\x2c\xc0\x2f\xc0\x30\x00\xff" \
"\x01\x00\x00\x1a\x00\x0d\x00\x16\x00\x14\x06\x03\x06\x01\x05\x03" \
"\x05\x01\x04\x03\x04\x01\x03\x03\x03\x01\x02\x03\x02\x01"

s = socket.socket()
s.connect(("127.0.0.1", 443))
s.send(b)
time.sleep(3)
del(s)

Should work in python3 like this:

python3

import socket
import time

b = \
b"\x16\x03\x00\x00\x69\x01\x00\x00\x65\x03\x03\x64\xc2\x0e\x8c\xd1" \
b"\x66\xdb\x75\x22\x97\xc5\x4a\x87\x8d\x06\x33\xaf\xe8\xfc\x8d\xd3" \
b"\x0e\x68\x43\xc4\x63\x64\xe7\xd0\xa9\xd2\x87\x00\x00\x22\x00\x33" \
b"\x00\x35\x00\x39\x00\x3c\x00\x3d\x00\x67\x00\x6b\x00\x05\xc0\x13" \
b"\xc0\x14\xc0\x23\xc0\x24\xc0\x2b\xc0\x2c\xc0\x2f\xc0\x30\x00\xff" \
b"\x01\x00\x00\x1a\x00\x0d\x00\x16\x00\x14\x06\x03\x06\x01\x05\x03" \
b"\x05\x01\x04\x03\x04\x01\x03\x03\x03\x01\x02\x03\x02\x01"

s = socket.socket()
s.connect(("127.0.0.1", 443))
s.send(b)
time.sleep(3)
del(s)

I noticed that when forcing ECDHE ciphers, openssl just rejects the handshake with (no shared cipher), although the same ECDHE ciphers are supported on both client and server.

I’m assuming some extensions are missing in the client hello, so the client really doesn’t support the ECDHE ciphers it announces.

To workaround this problem on the server side, you need to add at least one non-ECDHE ciphers actually supported by the client.

The following one should work:

IANA -> openssl cipher
TLS_DHE_RSA_WITH_AES_128_CBC_SHA -> DHE-RSA-AES128-SHA
TLS_RSA_WITH_AES_256_CBC_SHA -> AES256-SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA -> DHE-RSA-AES256-SHA
TLS_RSA_WITH_AES_128_CBC_SHA256 -> AES128-SHA256
TLS_RSA_WITH_AES_256_CBC_SHA256 -> AES256-SHA256
TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 -> DHE-RSA-AES128-SHA256
TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 -> DHE-RSA-AES256-SHA256

So I suggest to append your cipher configuration with one of those.

For example:

ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:AES128-SHA256

This will allow TLS_RSA_WITH_AES_128_CBC_SHA256 (AES128-SHA256).

Or, if you prefer a forward secrecy cipher:

ssl-default-bind-ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA256

This will allow TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 (DHE-RSA-AES128-SHA256).

Of course by adding older ciphers you are lowering security here.

This way you should be able to proceed. I hope you will face less complicated TR069 issues with this ONT, compared to the TLS issue :slight_smile:

13

Thanks man. Unfortunately I’ve tried every single one of those ciphers now, and it still fails. Now I will probably need to take this up with the device vendor.

14

Try combing those new ciphers with the :@SECLEVEL=0 at the end, maybe in openssl 3.1 you need both (I used 3.0 here).

If it still doesn’t work, can you check if you still see the same no shared cipher error in haproxy?

15

Tried it. Still no shared cipher

16

Can you send another tcpdump trace with those ciphers in it? The upload link you used last time is still valid. I really don’t see how this can still happen.

17

I’ll share the trace tomorrow.