Websockets - 1006 Random Disconnect

I’m running a .Net 5 application using SignalR for websockets with a Vue JS app.

Recently, my client seems to continually disconnect randomly. I am able to reproduce the issue more consistently by opening another browser/device and establishing a new WSS connection.

Traffic:

  1. Last WS traffic at 51:39.929 (Type 6/KeepAlive)
  2. Client SignalR Logs (Debug) at 51:39.999Z
  3. New websocket connections being continually created
  4. .Net App Traffic
2021-01-23T04:51:39.122251204Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.HttpConnectionManager[1]
2021-01-23T04:51:39.122285493Z       New connection SBCK7IOpjJgPDcCq7n8zEQ created.
2021-01-23T04:51:39.122289621Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.HttpConnectionDispatcher[10]
2021-01-23T04:51:39.122293416Z       Sending negotiation response.
2021-01-23T04:51:39.328029116Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.HttpConnectionDispatcher[4]
2021-01-23T04:51:39.328054094Z       Establishing new connection.
2021-01-23T04:51:39.328067732Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsTransport[1]
2021-01-23T04:51:39.328071260Z       Socket opened using Sub-Protocol: '(null)'.
2021-01-23T04:51:39.328074252Z dbug: Microsoft.AspNetCore.SignalR.HubConnectionHandler[5]
2021-01-23T04:51:39.328077311Z       OnConnectedAsync started.
2021-01-23T04:51:39.385290276Z dbug: Microsoft.AspNetCore.SignalR.Internal.DefaultHubProtocolResolver[2]
2021-01-23T04:51:39.385315351Z       Found protocol implementation for requested protocol: json.
2021-01-23T04:51:39.385319091Z dbug: Microsoft.AspNetCore.SignalR.HubConnectionContext[1]
2021-01-23T04:51:39.385336045Z       Completed connection handshake. Using HubProtocol 'json'.

2021-01-23T04:51:40.028338267Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsTransport[14]
2021-01-23T04:51:40.028355133Z       Socket connection closed prematurely.
2021-01-23T04:51:40.028358571Z       System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
2021-01-23T04:51:40.028363128Z          at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2021-01-23T04:51:40.028368417Z          at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2021-01-23T04:51:40.028373089Z          at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory`1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)
2021-01-23T04:51:40.028377818Z          at Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsServerTransport.StartReceiving(WebSocket socket)
2021-01-23T04:51:40.028382630Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsTransport[4]
2021-01-23T04:51:40.028387671Z       Waiting for the application to finish sending data.
2021-01-23T04:51:40.028391779Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsTransport[2]
2021-01-23T04:51:40.028408655Z       Socket closed.
2021-01-23T04:51:40.028413197Z dbug: Microsoft.AspNetCore.SignalR.HubConnectionHandler[6]
2021-01-23T04:51:40.028416471Z       OnConnectedAsync ending.

I’m unable to reproduce this issue at all locally with multiple browsers. This only seems to happen on production: docker, HAProxy, and NGINX.

The .Net app is running in a docker container. HAProxy forwards all requests to this docker container.
The VueJS app is running on NGINX in a docker container.
Everything is also running behind Cloudflare.

If it was a keepalive/timeout with HAProxy, I would assume these disconnects would be at a consistent interval. If it was rate-limiting with HAProxy, the client networking area would show some 429 related errors.

Test:
I setup three devices on my home network.

  1. Laptop - home wifi - VPN
  2. Mobile device - home wifi
  3. Desktop - home wifi

Once two sessions were made on the phone and desktop, those two got the WSS disconnects. However, the device on my network using a VPN seemed to have no issues. This seems to be something specific to my home IP being rate-limited by HAProxy or something…

Here is the HAProxy Config

global                                                                                                                         
        maxconn                 30000                                                                                          
        maxcompcpuusage         90                                                                                             
        spread-checks           5                                                                                              
                                                                                                                               
        # node example.com                                                                                             
                                                                                                                             
        tune.ssl.default-dh-param 2048                                                                                         
        ssl-default-bind-ciphers 12334:ECDHE-
        ssl-default-bind-options no-sslv3 no-tls-tickets                                                                       
        ssl-default-server-ciphers 12345:ECDH
        ssl-default-server-options no-sslv3 no-tls-tickets                                                                     
                                                                                                                               
defaults                                                                                                                       
        mode                    http                                                                                           
        log                     /dev/log local0                                                                                
        option                  httplog                                                                                        
        option                  dontlognull                                                                                    
        option                  redispatch                                                                                     
        option                  tcp-smart-accept                                                                               
        option                  tcp-smart-connect                                                                              
        option                  http-server-close                                                                              
        option                  splice-response                                                                                
        option                  http-keep-alive                                                                                
        option                  clitcpka                                                                                       
        option                  srvtcpka                                                                                       
        option                  contstats                                                                                      
        retries                 3                                                                                              
        timeout http-request    5s   
        timeout queue           10s                                                                                            
        timeout connect         10s                                                                                            
        timeout client          1m                                                                                             
        timeout client-fin      1m                                                                                             
        timeout server          2m                                                                                             
        timeout tunnel          40m                                                                                            
        timeout http-keep-alive 5s                                                                                             
        timeout check           10s                                                                                            
        timeout tarpit          15s                                                                                            
        default-server          init-addr none                                                                                 
                                                                                                                               
resolvers docker                                                                                                               
        nameserver dns 127.0.0.11:53                                                                                                                                                                                                                                                                            
                                                                                                                               
frontend internal-connect                                                                                                      
        bind :444 ssl crt /usr/local/etc/haproxy/ssl/                                                                          
        option forwardfor                                                                                                      
        use_backend stats if { hdr_dom(host) -i haproxy.zabbix_zbx_net_backend }                                               
                                                                                                                               
frontend http                                                                                                                  
        bind :80                                                                                                               
        bind :443 ssl crt /usr/local/etc/haproxy/ssl/     
        log global                                                                                                             
                                                                                                                               
        # option forwardfor                                                                                                    
                                                                                                                               
        http-request set-header X-Forwarded-Port %[dst_port]                                                                   
        http-request add-header X-Forwarded-Proto https if { ssl_fc }                                                          
                                                                                                                               
        http-request redirect scheme https if !{ ssl_fc }                                                                      
                                                                                                                               
        acl from_cf    src -f /usr/local/etc/haproxy/cloudflare_ips.lst                                                        
        acl cf_ip_hdr  req.hdr(CF-Connecting-IP) -m found                                                                      
                                                                                                                               
        http-request set-header X-Forwarded-For %[req.hdr(CF-Connecting-IP)] if from_cf cf_ip_hdr                              
                                                                                                                               
        acl control_auth http_auth(control)                                                                                    
        # acl control_auth_dev http_auth_group(control) dev-access                                                             
        # acl control_auth_full http_auth_group(control) full-access                                                           
                                                                                                                               
        http-request auth realm Protected if { hdr_dom(host) -i stats.example.com } !control_auth                      
        # use_backend frontend if { hdr_dom(host) -i control.example.com } control_auth_full # or control_auth_dev     
                                                                                                                               
        use_backend sonoransupport if { hdr_dom(host) -i support.example.com }                                         
        use_backend sonoransupport-backend if { hdr_dom(host) -i api.example.com }                                                          
        use_backend frontend if { hdr_dom(host) -i example.example }                                                          
        use_backend frontend if { hdr_dom(host) -m str -i example.com }                                                
        default_backend frontend                                                                                               
                                                                                                                               
backend frontend                                                                                                               
        option httpchk HEAD / HTTP/1.1\r\nHost:\ example.com                                                           
                                                                                                                               
        http-response del-header ^Server:.*$                                                                                   
                                                                                                                               
        server nginx nginx.frontend_default:80 check send-proxy resolvers docker resolve-prefer ipv4                           
                                                                                                                                                      
backend sonoransupport                                                                                                         
        option httpchk HEAD / HTTP/1.1\r\nHost:\ support.example.com                                                   
                                                                                                                               
        # http-response del-header ^Server:.*$                                                                                 
                                                                                                                               
        server support_frontend sonoransupport.frontend_default:80 check resolvers docker resolve-prefer ipv4                  
                                                                                                                               
backend sonoransupport-backend                                                                                                 
        option httpchk HEAD / HTTP/1.1\r\nHost:\ api.example.com
       # http-response del-header ^Server:.*$                                                                                 
                                                                                                                               
        server support_backend sonoransupport.backend_default:80 check resolvers docker resolve-prefer ipv4 

Any idea what could be causing these websockets to be closed? It seems to be an issue specifically with HAProxy. Almost like a “max websockets per client” or something.

Update:

The issue seems to be between Cloudflare and HAProxy. If we disable Cloudflare’s proxy, the disconnect issue no longer perrsists. However, with Cloudflare proxy enabled the websockets randomly disconnect every ~20-60 seconds as shown.

You are troubleshooting a Cloudflare issue (Error code 1006):

Check your account and logs or ask Cloudflare for help.

edit: also see:

A support tech confirmed to me via email that Cloudflare automatically disconnects websocket connections that remain dormant for 100 seconds.