HAProxy community

Client sends “FIN, ACK” instead of “Client Hello” | Connection closed during SSL handshake

I am trying to establish SSL connection between the .NET WebSocket Client and my server by calling “wss://domain.xyz”. Haproxy is handling the SSL handshake and once that is done it connect to a NodeJs server running on the same server.
Currently, I am facing an issue where after TCP handshake is made, client sends ACK and FIN, ACK packets instead of “Client Hello” to server. And getting below error in haproxy log.

secured/1: Connection closed during SSL handshake

Normally, that would indicate that this is a client code issue. The confusing part is that the client is able to do ssl handshake when calling “wss://echo.websocket.org”.
But also the Haproxy configuration looks like its working fine since I am able to do ssl handshake using browser (Chrome, IE11, Edge, Firefox) and “WebSocket Test Client” chrome add-on. Ssllabs confirms that part as well. Client is also able to open non-secure websocket connection.

Haproxy -vv

    HA-Proxy version 2.0.18-1ppa1~bionic 2020/09/30 - https://haproxy.org/
    Build options :
      TARGET  = linux-glibc
      CPU     = generic
      CC      = gcc
      CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-VtLTwE/haproxy-2.0.18=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
      OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1 USE_SYSTEMD=1
    
    Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED +REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL +SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
    
    Default settings :
      bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
    
    Built with multi-threading support (MAX_THREADS=64, default=1).
    Built with OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
    Running on OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
    OpenSSL library supports TLS extensions : yes
    OpenSSL library supports SNI : yes
    OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
    Built with Lua version : Lua 5.3.3
    Built with network namespace support.
    Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
    Built with zlib version : 1.2.11
    Running on zlib version : 1.2.11
    Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
    Built with PCRE2 version : 10.31 2018-02-12
    PCRE2 library supports JIT : yes
    Encrypted password support via crypt(3): yes
    Built with the Prometheus exporter as a service
    
    Available polling systems :
          epoll : pref=300,  test result OK
           poll : pref=200,  test result OK
         select : pref=150,  test result OK
    Total: 3 (3 usable), will use epoll.
    
    Available multiplexer protocols :
    (protocols marked as <default> cannot be specified using 'proto' keyword)
                  h2 : mode=HTX        side=FE|BE     mux=H2
                  h2 : mode=HTTP       side=FE        mux=H2
           <default> : mode=HTX        side=FE|BE     mux=H1
           <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS
    
    Available services :
            prometheus-exporter
    
    Available filters :
            [SPOE] spoe
            [COMP] compression
            [CACHE] cache
            [TRACE] trace

Below is my HaProxy configuration. I am suspecting I am doing something wrong in it.

Haproxy configuration

    global
      log /dev/log local0
      maxconn 4096
      user haproxy
      group haproxy
      daemon
      ca-base /etc/haproxy/certs
      crt-base /etc/haproxy/certs
      tune.ssl.default-dh-param 2048
      tune.ssl.cachesize 100000
      tune.ssl.lifetime 6000
      #ssl-default-bind-options no-sslv3
      ssl-default-server-options force-tlsv12
      ssl-default-bind-options force-tlsv12 no-tls-tickets
      ssl-default-bind-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP:TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384:TLS_DHE_RSA_WITH_AES_128_GCM_SHA256:TLS_DHE_RSA_WITH_AES_256_GCM_SHA384:TLS_DHE_RSA_WITH_AES_128_CBC_SHA:TLS_DHE_RSA_WITH_AES_256_CBC_SHA:TLS_DHE_RSA_WITH_AES_128_CBC_SHA256:TLS_DHE_RSA_WITH_AES_256_CBC_SHA256:TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256:TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305:TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256:TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305:TLS_DHE_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_128_CBC_SHA:TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA:TLS_RSA_WITH_CAMELLIA_128_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_128_CBC_SHA:TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA:TLS_EMPTY_RENEGOTIATION_INFO_SCSV:ECDHE-RSA-AES128-SHA256:TLS_RSA_WITH_RC4_128_SHA:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:TLS_RSA_WITH_AES_128_GCM_SHA256:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS:DES-CBC3-SHA:RSA+3DES
      ssl-default-server-ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:ECDHE-RSA-DES-CBC3-SHA:ECDHE-ECDSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:DES-CBC3-SHA:HIGH:SEED:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!RSAPSK:!aDH:!aECDH:!EDH-DSS-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA:!SRP:TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384:TLS_DHE_RSA_WITH_AES_128_GCM_SHA256:TLS_DHE_RSA_WITH_AES_256_GCM_SHA384:TLS_DHE_RSA_WITH_AES_128_CBC_SHA:TLS_DHE_RSA_WITH_AES_256_CBC_SHA:TLS_DHE_RSA_WITH_AES_128_CBC_SHA256:TLS_DHE_RSA_WITH_AES_256_CBC_SHA256:TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256:TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305:TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256:TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305:TLS_DHE_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_128_CBC_SHA:TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA:TLS_RSA_WITH_CAMELLIA_128_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256:TLS_RSA_WITH_AES_128_CBC_SHA256:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA:TLS_RSA_WITH_AES_128_CBC_SHA:TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA:TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA:TLS_EMPTY_RENEGOTIATION_INFO_SCSV:ECDHE-RSA-AES128-SHA256:TLS_RSA_WITH_RC4_128_SHA:TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256:TLS_RSA_WITH_AES_128_GCM_SHA256:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS:DES-CBC3-SHA:RSA+3DES 
    
    defaults
      mode http
      log global
      maxconn 4096
      option forwardfor
      option http-server-close
      retries 3      
      timeout connect 30s
      timeout client 30s
      timeout server 30s
      # Long timeout for WebSocket connections.
      timeout tunnel 1h
      timeout http-keep-alive 10s
      timeout check 10s
    
    frontend secured
      bind :443 ssl crt /etc/haproxy/certs/domain.com.pem alpn http/1.1 ca-file /etc/haproxy/certs/domain.com.ca
      #reqadd X-Forwarded-Proto:\ https
      http-request set-header X-Forwarded-Proto https
      http-response set-header Content-Security-Policy "frame-ancestors domain.com"
      http-request add-header X-Custom-SSL-Cipher %sslc
      http-request add-header X-Custom-SSL-Version %sslv
      acl is_websocket hdr(Upgrade) -i WebSocket
      use_backend node if is_websocket
      mode http
      default_backend node 
    
    #frontend unsecured
      mode http
      bind *:80
      acl http      ssl_fc,not
      http-request redirect scheme https if http
    
      http-request set-header X-Forwarded-Proto https
      http-response set-header Content-Security-Policy "frame-ancestors domain.com"
      bind *:443 ssl crt hostname.com.pem alpn http/1.1
    
      acl is_websocket hdr_end(host) -i WebSocket
      use_backend node if is_websocket
      default_backend node
    
    backend node
      mode http
      balance roundrobin
      timeout check 50000ms
      http-request set-header Host domain.com
      server node1 127.0.0.1:8080 check inter 500ms

Wireshark pcap

    34	2.285439	192.168.10.19	178.XX.XX.XXX	TCP	66	55514 → 443 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 WS=256 SACK_PERM=1
    37	2.373956	178.XX.XX.XXX	192.168.10.19	TCP	66	443 → 55514 [SYN, ACK] Seq=0 Ack=1 Win=64240 Len=0 MSS=1452 SACK_PERM=1 WS=128
    38	2.374042	192.168.10.19	178.XX.XX.XXX	TCP	54	55514 → 443 [ACK] Seq=1 Ack=1 Win=132096 Len=0
    39	2.406464	192.168.10.19	178.XX.XX.XXX	TCP	54	55514 → 443 [FIN, ACK] Seq=1 Ack=1 Win=132096 Len=0
    45	2.495315	178.XX.XX.XXX	192.168.10.19	TCP	60	443 → 55514 [FIN, ACK] Seq=1 Ack=2 Win=64256 Len=0
    46	2.495364	192.168.10.19	178.XX.XX.XXX	TCP	54	55514 → 443 [ACK] Seq=2 Ack=2 Win=132096 Len=0

openssl s_client -connect domain.zyx:443

    root@ubuntu-s-1vcpu-1gb-ams3-01:/etc/haproxy# openssl s_client -connect domain.xyz:443
    CONNECTED(00000005)
    depth=2 C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
    verify return:1
    depth=1 C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
    verify return:1
    depth=0 CN = domain.xyz
    verify return:1
    ---
    Certificate chain
     0 s:CN = domain.xyz
       i:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
     1 s:C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
       i:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
     2 s:C = US, ST = New Jersey, L = Jersey City, O = The USERTRUST Network, CN = USERTrust RSA Certification Authority
       i:C = GB, ST = Greater Manchester, L = Salford, O = Comodo CA Limited, CN = AAA Certificate Services
    ---
    Server certificate
    -----BEGIN CERTIFICATE-----
    (Certificate)
    -----END CERTIFICATE-----
    subject=CN = domain.xyz
    
    issuer=C = GB, ST = Greater Manchester, L = Salford, O = Sectigo Limited, CN = Sectigo RSA Domain Validation Secure Server CA
    
    ---
    No client certificate CA names sent
    Peer signing digest: SHA256
    Peer signature type: RSA-PSS
    Server Temp Key: X25519, 253 bits
    ---
    SSL handshake has read 4951 bytes and written 409 bytes
    Verification: OK
    ---
    New, TLSv1.2, Cipher is ECDHE-RSA-CHACHA20-POLY1305
    Server public key is 2048 bit
    Secure Renegotiation IS supported
    Compression: NONE
    Expansion: NONE
    No ALPN negotiated
    SSL-Session:
        Protocol  : TLSv1.2
        Cipher    : ECDHE-RSA-CHACHA20-POLY1305
        Session-ID: ***************************************************************E
        Session-ID-ctx:
        Master-Key: *************************************************************************************2
        PSK identity: None
        PSK identity hint: None
        SRP username: None
        Start Time: 1601745936
        Timeout   : 7200 (sec)
        Verify return code: 0 (ok)
        Extended master secret: yes
    ---
    HTTP/1.1 408 Request Time-out
    content-length: 110
    cache-control: no-cache
    content-type: text/html
    connection: close
    
    <html><body><h1>408 Request Time-out</h1>
    Your browser didn't send a complete request in time.
    </body></html>
    closed

I tried using all tls versions, verified cipher suites even though it didn’t get to that point. In addition, I checked all client debugs but there were no errors. I wonder if calling wss:// uri from .net application needs to be handled differently in haproxy or If I need to configure anything on Domain website provider or hosting provider website. Machine is running Ubuntu 18.04 and client is running on .NET 2.0 (also tried with .NET3.5 & .NET 4.x). I am confused to which end is having the issue here since both work fine but not working together.

Would really appreciate it if you could help me debug this. If I missed any information please let me know. Thank you in advance.

There is nothing you can do on the haproxy side.

If your client doesn’t even send client_hello, everything modifying haproxy configuration is useless. SSL ciphers and other settings only come into play once the SSL handshake actually starts.

You need to look at your client instead.