This channel can no longer be used to send message output session was auto-closed due to a server-initiated shutdown

Help!
1

Hello,

I am using HAProxy to load balance a .NET / Net.TCP application.
The application works but if there is an idle of 1 minute or more, an exception is created by the caller -

This channel can no longer be used to send messages as the output session was auto-closed due to a server-initiated shutdown. Either disable auto-close by setting the DispatchRuntime.AutomaticInputSessionShutdown to false, or consider modifying the shutdown protocol with the remote server.

If we remove HAProxy from the middle, wait a minute or more between requests, this message never appears.

What is going on here? More importantly how can I stop this from happening?

It seems like the client (HAProxy in this case) is initiating the close.

haproxy.cfg -

global
        log 127.0.0.1   local3
        log /dev/log    local0
        log /dev/log    local0 debug
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd lis$
        stats socket ipv4@0.0.0.0:1979 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        nbproc 2
        cpu-map 1 1
        cpu-map 2 2
        maxconn 100000

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private


    # Default ciphers to use on SSL-enabled listening sockets.
    # For more information, see ciphers(1SSL). This list is from:
    #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
    # An alternative list with additional directives can be obtained from
    #  https://mozilla.github.io/server-side-tls/ssl-config-generator/?serv$
    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:EC$
    ssl-default-bind-options no-sslv3

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http


listen stats # Define a listen section called "stats"
  bind *:80 # Listen on localhost:80
  mode http
  stats enable  # Enable stats page
  stats hide-version  # Hide HAProxy version
  stats realm Haproxy\ Statistics  # Title text for popup window
  stats uri /haproxy_stats  # Stats URI
  #stats auth Username:Password  # Authentication credentials

dotnetconfg -

frontend lt_dotnet_7078_frontend
    bind *:7078
    mode tcp
    option clitcpka
    default_backend lt_dotnet_backend

backend lt_dotnet_backend
        timeout connect 5000
        option tcp-check
        option tcplog
        option srvtcpka
        mode tcp
        balance roundrobin
        server server02_10.11.20.32_7078 10.11.20.32:7078 check port 7078 inter 200

Help is appreciated.

2

Sounds like timeouts are intervening.

Share your complete configuration and haproxy logs.

3

You were right!

Timeouts. We are also making some internal code changes to handle this a bit better.

Question: I expected the option clitcpka and option srvtcpka to have some sort of affect here, but it does not appear that they make any difference. The timeout setting is the winner.

It seems with no timeouts set, the default is 60 seconds.

4

No, clitcpka and srvtcpka enable TCP keepalives on the socket (setting SO_KEEPALIVE). What that means is that the OS (Linux) is instructed to send periodic keepalives on this TCP connection, to avoid hitting timeouts on external devices like firewalls or NAT gateways.

However that doesn’t mean that this option circumvents haproxy’s own timeouts, that would not be beneficial.

I don’t think there is any default in haproxy regarding the timeouts, are you sure you don’t have a default section where those timeouts are defined?

5

I updated the original post with the HAProxy.cfg … Again, you were right. There were timeouts set in the defaults section.

What is the expected behavior with no timeouts set?

Also, why does the stats site keep looking for favicon ?
heh

6

When no timeouts sessions will remain active possibly forever. When maxconn is reached, not new connections will be accepted and the service will stop working.

That’s why it’s important to set at least the three main timeouts connect/client/server, so that haproxy can clear those sessions.

Haproxy will also warn strongly if a config does not contain those timeouts.

Regarding the favicon, it’s the browser that is looking for it.

7

OK thank you.
Today we are performing a small load test.
The Stats page shows what I seem to think is a large number of errors in the response column… Hovering over the number says: Coonnection rests during transfers 1105 client, 71 server.
Incoming about 55 - 100 hits/sec. Everything else seems to look ok.