I’m running a .Net 5 application using SignalR for websockets with a Vue JS app.
Recently, my client seems to continually disconnect randomly. I am able to reproduce the issue more consistently by opening another browser/device and establishing a new WSS connection.
Traffic:
- Last WS traffic at
51:39.929(Type 6/KeepAlive)
- Client SignalR Logs (Debug) at
51:39.999Z
- New websocket connections being continually created
- .Net App Traffic
2021-01-23T04:51:39.122251204Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.HttpConnectionManager[1]
2021-01-23T04:51:39.122285493Z New connection SBCK7IOpjJgPDcCq7n8zEQ created.
2021-01-23T04:51:39.122289621Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.HttpConnectionDispatcher[10]
2021-01-23T04:51:39.122293416Z Sending negotiation response.
2021-01-23T04:51:39.328029116Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.HttpConnectionDispatcher[4]
2021-01-23T04:51:39.328054094Z Establishing new connection.
2021-01-23T04:51:39.328067732Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsTransport[1]
2021-01-23T04:51:39.328071260Z Socket opened using Sub-Protocol: '(null)'.
2021-01-23T04:51:39.328074252Z dbug: Microsoft.AspNetCore.SignalR.HubConnectionHandler[5]
2021-01-23T04:51:39.328077311Z OnConnectedAsync started.
2021-01-23T04:51:39.385290276Z dbug: Microsoft.AspNetCore.SignalR.Internal.DefaultHubProtocolResolver[2]
2021-01-23T04:51:39.385315351Z Found protocol implementation for requested protocol: json.
2021-01-23T04:51:39.385319091Z dbug: Microsoft.AspNetCore.SignalR.HubConnectionContext[1]
2021-01-23T04:51:39.385336045Z Completed connection handshake. Using HubProtocol 'json'.
2021-01-23T04:51:40.028338267Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsTransport[14]
2021-01-23T04:51:40.028355133Z Socket connection closed prematurely.
2021-01-23T04:51:40.028358571Z System.Net.WebSockets.WebSocketException (0x80004005): The remote party closed the WebSocket connection without completing the close handshake.
2021-01-23T04:51:40.028363128Z at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2021-01-23T04:51:40.028368417Z at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2021-01-23T04:51:40.028373089Z at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TWebSocketReceiveResultGetter,TWebSocketReceiveResult](Memory`1 payloadBuffer, CancellationToken cancellationToken, TWebSocketReceiveResultGetter resultGetter)
2021-01-23T04:51:40.028377818Z at Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsServerTransport.StartReceiving(WebSocket socket)
2021-01-23T04:51:40.028382630Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsTransport[4]
2021-01-23T04:51:40.028387671Z Waiting for the application to finish sending data.
2021-01-23T04:51:40.028391779Z dbug: Microsoft.AspNetCore.Http.Connections.Internal.Transports.WebSocketsTransport[2]
2021-01-23T04:51:40.028408655Z Socket closed.
2021-01-23T04:51:40.028413197Z dbug: Microsoft.AspNetCore.SignalR.HubConnectionHandler[6]
2021-01-23T04:51:40.028416471Z OnConnectedAsync ending.
I’m unable to reproduce this issue at all locally with multiple browsers. This only seems to happen on production: docker, HAProxy, and NGINX.
The .Net app is running in a docker container. HAProxy forwards all requests to this docker container.
The VueJS app is running on NGINX in a docker container.
Everything is also running behind Cloudflare.
If it was a keepalive/timeout with HAProxy, I would assume these disconnects would be at a consistent interval. If it was rate-limiting with HAProxy, the client networking area would show some 429 related errors.
Test:
I setup three devices on my home network.
- Laptop - home wifi - VPN
- Mobile device - home wifi
- Desktop - home wifi
Once two sessions were made on the phone and desktop, those two got the WSS disconnects. However, the device on my network using a VPN seemed to have no issues. This seems to be something specific to my home IP being rate-limited by HAProxy or something…
Here is the HAProxy Config
global
maxconn 30000
maxcompcpuusage 90
spread-checks 5
# node example.com
tune.ssl.default-dh-param 2048
ssl-default-bind-ciphers 12334:ECDHE-
ssl-default-bind-options no-sslv3 no-tls-tickets
ssl-default-server-ciphers 12345:ECDH
ssl-default-server-options no-sslv3 no-tls-tickets
defaults
mode http
log /dev/log local0
option httplog
option dontlognull
option redispatch
option tcp-smart-accept
option tcp-smart-connect
option http-server-close
option splice-response
option http-keep-alive
option clitcpka
option srvtcpka
option contstats
retries 3
timeout http-request 5s
timeout queue 10s
timeout connect 10s
timeout client 1m
timeout client-fin 1m
timeout server 2m
timeout tunnel 40m
timeout http-keep-alive 5s
timeout check 10s
timeout tarpit 15s
default-server init-addr none
resolvers docker
nameserver dns 127.0.0.11:53
frontend internal-connect
bind :444 ssl crt /usr/local/etc/haproxy/ssl/
option forwardfor
use_backend stats if { hdr_dom(host) -i haproxy.zabbix_zbx_net_backend }
frontend http
bind :80
bind :443 ssl crt /usr/local/etc/haproxy/ssl/
log global
# option forwardfor
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
http-request redirect scheme https if !{ ssl_fc }
acl from_cf src -f /usr/local/etc/haproxy/cloudflare_ips.lst
acl cf_ip_hdr req.hdr(CF-Connecting-IP) -m found
http-request set-header X-Forwarded-For %[req.hdr(CF-Connecting-IP)] if from_cf cf_ip_hdr
acl control_auth http_auth(control)
# acl control_auth_dev http_auth_group(control) dev-access
# acl control_auth_full http_auth_group(control) full-access
http-request auth realm Protected if { hdr_dom(host) -i stats.example.com } !control_auth
# use_backend frontend if { hdr_dom(host) -i control.example.com } control_auth_full # or control_auth_dev
use_backend sonoransupport if { hdr_dom(host) -i support.example.com }
use_backend sonoransupport-backend if { hdr_dom(host) -i api.example.com }
use_backend frontend if { hdr_dom(host) -i example.example }
use_backend frontend if { hdr_dom(host) -m str -i example.com }
default_backend frontend
backend frontend
option httpchk HEAD / HTTP/1.1\r\nHost:\ example.com
http-response del-header ^Server:.*$
server nginx nginx.frontend_default:80 check send-proxy resolvers docker resolve-prefer ipv4
backend sonoransupport
option httpchk HEAD / HTTP/1.1\r\nHost:\ support.example.com
# http-response del-header ^Server:.*$
server support_frontend sonoransupport.frontend_default:80 check resolvers docker resolve-prefer ipv4
backend sonoransupport-backend
option httpchk HEAD / HTTP/1.1\r\nHost:\ api.example.com
# http-response del-header ^Server:.*$
server support_backend sonoransupport.backend_default:80 check resolvers docker resolve-prefer ipv4
Any idea what could be causing these websockets to be closed? It seems to be an issue specifically with HAProxy. Almost like a “max websockets per client” or something.