Server Disconnects / TCP Window Full

[MaEh86]

Hello together,

we are using HAproxy widely, but we’re actually facing a strange problem, I can’t get behind and hope, someone can help

Our constellation: Client => HAproxy => Dockerhost with Camunda-Container

To classify the IPs in the attached config, here are some Infos:

Client: 10.8.11.150
HAproxy: 10.1.2.180
Backend (Docker with Camunda-Container): 10.1.3.103

If the Client sends small POST-Requests, all works fine. When the request are getting bigger, you see the following in the HAproxy-Log:

Apr 28 11:59:26 kag-lx-pxy-t1 haproxy[6979]: 10.8.11.150:58270 [28/Apr/2023:11:59:26.804] haproxy-intern-test-frontend-http backend_camunda-dev/kag-docker-03 0/0/0/2/101 401 149 - - SD-- 12/1/0/0/0 0/0 "POST /engine-rest/message HTTP/1.1"

First, I thought, that https in the frontend is the problem, so I enabled http for the frontend. Unfortunately, this doesn’t help. We made further tests with the client communicating directly to the Docker-Host and the problem disappeared. So, I think, the problem is HAproxy related.

In the next step, I made tcpdumps on frontend- and backend-side. Here the TCP window size messages in the backend communication were strange, while the client continued to talk diligently with the frontend. I think this is where the problem lies, but I don’t know, how to resolve this issue.

Any help on this topic? What else can I do to narrow down the problem?

Attached, some usefull informations (config, tcpdump-screenshots)

thanks and best regards,
markus

root@kag-lx-pxy-t1:~# haproxy -vv
HAProxy version 2.6.12-1ppa1~bionic 2023/04/01 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2027.
Known bugs: http://www.haproxy.org/bugs/bugs-2.6.12.html
Running on: Linux 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -O2 -fdebug-prefix-map=/build/haproxy-rnXPxH/haproxy-2.6.12=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_PROMEX=1
  DEBUG   = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE +LIBCRYPT +LINUX_SPLICE +LINUX_TPROXY +LUA -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -QUIC +RT +SLZ -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with the Prometheus exporter as a service
Built with network namespace support.
Support for malloc_trim() is enabled.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.31 2018-02-12
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 7.5.0

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
         h2 : mode=HTTP  side=FE|BE  mux=H2    flags=HTX|HOL_RISK|NO_UPG
       fcgi : mode=HTTP  side=BE     mux=FCGI  flags=HTX|HOL_RISK|NO_UPG
  <default> : mode=HTTP  side=FE|BE  mux=H1    flags=HTX
         h1 : mode=HTTP  side=FE|BE  mux=H1    flags=HTX|NO_UPG
  <default> : mode=TCP   side=FE|BE  mux=PASS  flags=
       none : mode=TCP   side=FE|BE  mux=PASS  flags=NO_UPG

Available services : prometheus-exporter
Available filters :
        [CACHE] cache
        [COMP] compression
        [FCGI] fcgi-app
        [SPOE] spoe
        [TRACE] trace

global
        log /dev/log    local0 info
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
        stats socket /var/lib/haproxy/stats
        stats timeout 30s
        user haproxy
        group haproxy
        daemon
        maxconn 60000

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
        ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
        ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
        ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets

        tune.ssl.default-dh-param 2048

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen stats
        bind *:8000
        mode http
        stats enable
        stats realm Haproxy\ Statistics
        stats uri /
        stats refresh 5s

peers haproxy-intern-test-peers
        peer kag-lx-pxy-t1 10.1.2.180:50000

######################
#                    #
#  Frontend - HTTP   #
#                    #
######################

frontend haproxy-intern-test-frontend-http
        bind *:80

        acl valid_http_method method GET HEAD POST PUT DELETE OPTIONS
        http-request deny if ! valid_http_method

        # Umleitung von http zu https (für bestimmte Webs unterdrücken, da sie auch per http erreichbar sein müssen)
        redirect scheme https code 301 if !{ ssl_fc } !{ hdr_beg(host) -i camunda-test } !{ hdr_beg(host) -i camunda-dev }

        option forwardfor
        default_backend haproxy-intern-test-backend_no_match

######################
#                    #
# Backend-Zuordnung  #
#        HTTP        #
#                    #
######################

        use_backend backend_camunda-test                if { hdr_beg(host) -i camunda-test }
        use_backend backend_camunda-dev                 if { hdr_beg(host) -i camunda-dev }

######################
#                    #
#  Frontend - HTTPS  #
#                    #
######################

frontend haproxy-intern-test-frontend-https
        bind *:443 ssl crt /etc/haproxy/ssl/ alpn h2,http/1.1

        acl valid_http_method method GET HEAD POST PUT DELETE OPTIONS
        http-request deny if ! valid_http_method

        http-request set-header X-Forwarded-Host %[req.hdr(Host)]
        http-request set-header X-Forwarded-Proto https

        http-response set-header Strict-Transport-Security max-age=15552000;\ includeSubdomains

        option forwardfor

        default_backend haproxy-intern-test-backend_no_match

######################
#                    #
# Backend-Zuordnung  #
#       HTTPS        #
#                    #
######################

        use_backend backend_camunda-dev                 if { hdr_beg(host) -i camunda-dev }
        use_backend backend_camunda-test                if { hdr_beg(host) -i camunda-test }

######################
#                    #
#      Backends      #
#                    #
######################

#Default Backend -> 403 ausgeben!
backend haproxy-intern-test-backend_no_match
        http-request deny deny_status 403

#########################################################

backend backend_camunda-dev
        stick-table type ip size 200k expire 30m peers haproxy-intern-test-peers
        stick on src
        server docker-03 10.1.3.103:9002 check inter 10000
        server docker-04 10.1.3.104:9002 check inter 10000 backup

        option httpchk GET /
               http-check expect status 200
               http-check send-state

        option redispatch
        option forwardfor

        http-response del-header Server
        http-response del-header X-Powered-By

#########################################################

backend backend_camunda-test
        stick-table type ip size 200k expire 30m peers haproxy-intern-test-peers
        stick on src
        server docker-03 10.1.3.103:9102 check inter 10000
        server docker-04 10.1.3.104:9102 check inter 10000 backup

        option httpchk GET /
               http-check expect status 200
               http-check send-state

        option redispatch
        option forwardfor

        http-response del-header Server
        http-response del-header X-Powered-By

[IgorC]

The SD-- in the haproxy log indicates the connection was closed by the backend while haproxy was in the process of sending data. This is also confirmed by the tcp dump where we can see camunda sending [RST, ACK] packet.

Seems like camunda can not keep up with the data sending rate (receive buffer full), sets a window size of 0 and then resets the connection upon receiving TCP keep-alive packet (common procedure to probe the chocked side if ready to start receiving again) from haproxy. Why is this happening only when there is haproxy in the middle I have no idea. What does camunda container log say, any errors? What about the http 401 in the dump? Does it happen without haproxy in the middle? BTW did you mention what kind of boxes are these, all linux, some linux some windows, or …)? Any insight on the haproxy and camunda (host and container) network settings would be good here. In case of linux probably most interesting would be:

net.ipv4.tcp_window_scaling
net.ipv4.tcp_rmem
net.ipv4.tcp_wmem
net.core.rmem_max
net.core.wmem_max

Apart from that I really have no real suggestion about what might help in terms of haproxy parameters tuning.

[MaEh86]

Thanks for your answer. I wanted to think about everything, but forgot to tell something about the machines - my fault, sorry

Client: Windows 10 (but shouldn’t play any role)
HAproxy: Ubuntu 18.04 LTS (VM / 2 Cores / 4GB Ram)
Docker-Host: Ubuntu 20.04 LTS (VM / 4 Cores / 32GB Ram)

In the Camunda-Logs, nothing helpfull could be found, but I think, it’s OS-related now

Your requested sysctl-Values are the same on both machines:

net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_rmem = 4096        131072  6291456
net.ipv4.tcp_wmem = 4096        16384   4194304
net.core.rmem_max = 212992
net.core.wmem_max = 212992

I compared the sysctl-Output on both machines, and what is different for example, is net.ipv4.tcp_mem (382716/510290/765432 on Docker-Host & 45354/60473/90708 on HAproxy), but this seems Ram-related.

I currently tested the Request with net.ipv4.tcp_window_scaling = 0 on the HAproxy-Host, and the error the error did not occur, so the direction seems right. But I don’t think, that this is a good solution for production environment.

thanks and best regards,
Markus

btw.: the 401 comes from Camunda. It’s a basic-authentication.

[IgorC]

Similar issue seems has been reported here Occasional TCP connection timeout/TCP zerowindow to backend- HTTP2 only · Issue #1468 · haproxy/haproxy · GitHub BUT in that case it was haproxy sending zero-window and in your case is the other way around. And apparently that was fixed and should be applied to the 2.6 version.

As something to try, maybe increase the network buffers on the camunda side (you can do it on the fly):

sysctl -w net.core.rmem_max=4194304
sysctl -w net.core.wmem_max=4194304

and see if that makes any diff.

[MaEh86]

Hello Igor and thanks for your reply. The increased network-buffers on camunda-side didn’t help.
I did much research in the meantime and tested a little more, but I think, the problem is related to the basic-auth with code 401 sended from camunda.

I made a fresh tcp-dump:

You can see, that the HAproxy (10.1.2.181) is sequentially sending data and on packet 60, the camunda-Server sends the basic-auth-request with http code 401. At this point, the HAproxy diligently sends on his packets until the TCP Window gets full and here the problem starts. I can’t say, why the HAproxy doesn’t stop sending, but we will try to deactivate the basic-auth for testing - when I’m right, the request should pass without problems.

I also looked, why direct traffic without HAproxy (directly from client to the camunda-server) passes and there, I have noticed, that Window Scaling was used so there could be enough space, that the client sends the complete request without waiting for an ACK. So, Window Scaling would help in this case, but if the request gets much bigger, the same behaviour should occur with much bigger requests and direct communication.

So, 2 tasks, to work on

• Why isn’t window scaling working on the communication between HAproxy and Docker-Host/camunda-server?
• Will the request over HAproxy work without basic-out?

thanks and best regards,
Markus

[MaEh86]

Short Update:

• HAproxy-Host is using Window Scaling, but uses very small window-size:

• Camunda Rest-API without Authentication isn’t possible (by design)
  Maybe, there is a possibility, that HAproxy directly answers the first request without authentication to the frontend directly (without the “detour” over the backend) , as long, as the Header “Authentication” isn’t included in the request?

best regards,
Markus

[MaEh86]

Update 2:

found a working config for intercepting the basic-auth at the HAproxy

http-request auth realm authentication_for_camunda if !{ req.hdr(authorization) -m found } { path_beg /engine-rest/ }

best regards,
Markus