Contents load after "timeout server" hit

Help!
1

I’m totally new in the world of haproxy. I’m using it as loadbalancer with the following setup:

  • loadablanacer (haproxy 2.2.9, deb11)
  • 2x webserver with php-fpm and the Roundcube app (a webmail client, deb11)
  • db server (MariaDB, deb11)

I managed to set up all what I need for my test but when I open the website (Roundcube) using the loadbalancer some content remains in pending until the timeout server setting hit. After that, the content continue to load correctly (it could happens for more than one file, tipically .js).

The log, after the timeout, return the following lines:

Dec 30 15:54:27 lb haproxy\[513\]: [my.public.ip.addr:61687](https://my.public.ip.addr:61687) \[30/Dec/2022:15:54:21.210\] webmail-fe webmail-be/webmail1 0/0/1/3/6005 200 7804 - - sDVN 6/6/4/4/0 0/0 "GET /plugins/jqueryui/themes/elastic/jquery-ui.min.css?s=1658607433 HTTP/1.1"

That sDVN, if I got it right, means that the webserver didnt’ reply with any data and then it got killed (HAProxy version 1.8.30 - Configuration Manual):

s : the server-side timeout expired while waiting for the server to send or receive data.
D : the session was killed by haproxy because the server was detected as down and was configured to kill all connections when going down.

On the webserver, the respective HTTP request:

webmail1.local - - [30/Dec/2022:15:54:21 +0100] "GET /plugins/jqueryui/themes/elastic/jquery-ui.min.css?s=1658607433 HTTP/1.1" 200 7848 "http://loadbalancer.local/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"

If I visit the website directly on the webserver everything works fine without waiting time.
All the server are connected via LAN and there are no latency or packet loss.

I’ll appreciate any hint or help. If need more info I’ll gather’em asap.

Here my configuration:

global
        maxconn         5000
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 12s
        timeout client  12s
        timeout server  12s
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend webmail-fe
        bind *:80
        mode http
        default_backend webmail-be

backend webmail-be
    balance leastconn
    option forwardfor
    cookie SERVERUSED insert indirect nocache
    # http-request cache-use webmail-cache
    # http-response cache-store webmail-cache
    # http-request set-header X-Forwarded-Port %[dst_port]
    # http-request add-header X-Forwarded-Proto https if { ssl_fc }
    # option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    option httpchk HEAD /
    default-server check maxconn 50
    server webmail1 webmail1.local:8080 cookie webmail1
#    server webmail2 192.168.0.53:8080 cookie webmail2

# listen stats
#   bind :32700
#    stats enable
#    stats uri /
#    stats hide-version
#    stats auth someuser:password

#cache webmail-cache
#   total-max-size 128
#   max-object-size 1000
#   max-age 14

No error log from webserver or any strange log on haproxy other than the line already posted.
No hanging connection using ss or any clear problem seen using tcpdump

I expect that, with so few connection (it’s a test env) and small webapp all the resources are loaded fast as directly visited skipping the loadbalancer.

2

First, be careful using documentation from different versions of HAProxy. You indicated that you are running 2.2.9 but are looking at documentation for 1.8.30. For the most part, the configuration is the same, however there are some significant differences in configuration between major versions. Please use HAProxy version 2.2.26-3 - Configuration Manual - Session state at disconnection for your version.

To clarify, the termination state sDVN means

  • On the first character, the server-side timeout expired while waiting for the server to send or receive data.
  • On the second character, the session was in the DATA phase.
  • On the third character, the client provided a VALID cookie, and was sent to the associated server.
  • On the last character, NO cookie was provided by the server, and none was inserted either.

Also from the termination state documentation about the first two characters being sD:

 sD   The server did not send nor acknowledge any data for as long as the
      "timeout server" setting during the data phase. This is often caused
      by too short timeouts on L4 equipment before the server (firewalls,
      load-balancers, ...), as well as keep-alive sessions maintained
      between the client and the server expiring first on haproxy.

Though 12 seconds seems like plenty, you might try increasing your timeouts in the defaults (or in the backend if you plan to have more backends with different configs) to something like 20s and see if it persists.

Side note: It’s very odd to me that the log entry from HAProxy shows a 200 response which indicates “OK” but then also has termination state information. When a request is processed without errors, the termination state should look like ---- to also indicate everything from HAProxy’s standpoint is normal. That’s probably not important, but if you continue to see errors after increasing the timeouts, maybe that’s worth digging into.

1 Like
3

Hi,

I played with different timeout valuse and, no matter what the contents load after it hit.

KeepAlive and other timeout (php fpm) on the webserver side seems ok and definetly above 12 seconds.

I tried the TCP mode instead of HTTP and it works like a charm, but I need sticky session so I guess I need to stick with HTTP mode.

Thank you for your reply, I’ll dig in more!

4

If possible, try upgrading to 2.2.26. After looking at this and noticing that little bit of weirdness in the logs, you might have run across an old bug that was fixed in a later version.

5

Aaaand upgrading to 2.4 fix the problem, everything works fine.

Thank you very much for the tip!

1 Like