2.8.x to 2.9 Sudden 411 issues

totallyrandomname · December 9, 2023, 5:28am

Hi,

I’m a very new user of HAProxy and am running into an issue when trying to upgrade from 2.8.x to 2.9.

We are running HAProxy on Kubernetes and over the past few weeks we’ve run a few billion messages through our setup (load testing is fun!).

When we try to upgrade to 2.9 we start to get 411’s (the server refuses to accept the request without a defined Content-Length header) in places where we were getting 200’s on 2.8.3 and 2.8.4.

I looked through the release notes and didn’t see anything between 2.8.4 and 2.9 that implied a difference with how those are being handled, but does anyone have ideas? I know there was CVE-2023-40225, but I thought the fix for that was in 2.8.4. When I log the Content-Length header I get 0 though, so maybe that’s it? I’m just confused why it would be an issue for 2.9 but not 2.8.4.

Has anyone else run into this or have ideas?

lukastribus · December 11, 2023, 10:16am

Are you saying that Haproxy is emitting the 411 or your backend server?

In that case, provide the output of show errors on the admin socket.

The fix for CVE-2023-40225 is in haproxy 2.8.2.

totallyrandomname · December 11, 2023, 4:58pm

Hi, thanks for the response.

We are getting these errors from the backend servers but only after going up to 2.9. 2.8.3 and 2.8.4 are returning 200.

The odd part is “show errors” does not show any errors with malformed HTTP calls on 2.9 or 2.8.x. We sent some purposefully malformed calls (two host headers) and “show errors” caught those.

After updating our logging it looks like only HTTP Posts with 0 or Undefined Content Lengths are being blocked in 2.9. Which is good, but now I’m curious why that isn’t happening in 2.8.3/4. Also curious if there are any configuration doodads we can change to make 2.9 behave like 2.8 while we update our clients and servers.

Thanks!

lukastribus · December 11, 2023, 5:38pm

Can you test 2.8.5 ? This could be a bug. You may have to file an issue on github:

totallyrandomname · December 11, 2023, 7:52pm

Sure thing, just tested on 2.8.5 and could not repro the issue.

We then tested on the different 2.9dev builds and found that the problem shows up between 2.9dev6 and 2.9dev7.

Thanks, so it sounds like a bug so I should report on Github instead of here. Appreciate the help!

lukastribus · December 18, 2023, 3:32pm

Fixed in 2.9.1 as per:

github.com/haproxy/haproxy

411's On 2.9-dev7 and above

opened 10:00PM - 11 Dec 23 UTC

closed 08:21AM - 15 Dec 23 UTC

picoseconddillo

type: bug status: fixed

### Detailed Description of the Problem Hi, Fairly recent user of HAProxy …here, loving it so far, thanks! We are running into an issue when trying to upgrade from 2.8.x to 2.9. When we try to upgrade to 2.9 we start to get 411’s from some of our backends on some calls where we were getting 200's. We tested on 2.8.3, 2.8.4, and 2.8.5 and all exhibited the same issue. Then we tested the various dev tags of 2.9 and got 200's as expected up until the 2.9-dev7. The 411's are only showing up on our backends that are Windows servers behind AppGW's. ### Expected Behavior Routes that 200 in 2.8.x should return 200 in 2.9.x rather than 411. ### Steps to Reproduce the Behavior 1. Curl endpoint behind HAProxy 2.8.x, get a 200 2. Deploy HAProxy 2.9.0, get 411 ### Do you have any idea what may have caused this? Not really, I have a suspicion it's something with the series of Content Length commits between 2.9-dev6 and 2.9-dev7 though. ### Do you have an idea how to solve the issue? Not yet. We're new to HAProxy. We've tried various adjustments on the Content-Length headers in configuration but no dice so far. ### What is your configuration? ```haproxy This is just one of our test configs global log stdout format raw local0 info maxconn 200 daemon nbthread 3 defaults log-format '{"level":"info","client":"%ci:%cp","time":"%tr","frontend_name":"%ft","frontend_backend_server":"%b/%s","TR":"%TR","Tw":"%Tw","Tc":"%Tc","Tr":"%Tr","Ta":"%Ta","status_code":"%ST","bytes_read":"%B","captured_request_cookie":"%CC","captured_response_cookie":"%CS","termination_state_with_cookie":"%tsc","connection_count":"%ac/%fc/%bc/%sc/%rc","queue_length":"%sq/%bq","host_header":"%hrl","response_headers":"%hs","request":"%r"}' error-log-format '{"level":"error","client":"%ci:%cp","time":"[%tr]","frontend_name":"%ft","connection_count":"%ac/%fc","message":"%[fc_err_str]/%[bc_err_str]","error_codes":"%[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed]","certificate_details":"%[ssl_fc_sni]/%sslv/%sslc","host_header":"%hr"}' log global option splice-auto option redispatch timeout client 60s timeout connect 60s timeout server 60s frontend tenants http-request capture req.hdr(X-Forwarded-For) len 64 http-request capture req.hdr(host) len 30 mode http bind *:443 ssl crt /usr/local/etc/ssl/tls.pem no-sslv3 alpn h2,http/1.1 bind *:80 option http-server-close option forwardfor http-request add-header X-Forwarded-Proto https http-request add-header X-Forwarded-Port 443 http-response add-header Strict-Transport-Security max-age=15768000 acl reviewservice_acl path_beg -i use_backend reviewservice if reviewservice_acl acl aqua-hosting_acl path_beg -i /api/review/viewer/ use_backend aqua-hosting if aqua-hosting_acl acl test-api_acl path_beg -i use_backend test-api if test-api_acl acl identity_acl path_beg -i use_backend identity if identity_acl acl smoketest_acl path_beg -i /istio-smoke-test/ use_backend smoketest if smoketest_acl acl healthcheck_acl path_beg -i /healthcheck/readyz use_backend healthcheck if healthcheck_acl acl loadtest_acl path_beg -i /loadtest use_backend loadtest if loadtest_acl backend reviewservice mode http server reviewservice ssl verify none sni req.hdr(host) backend test-api mode http server test-api ssl verify none sni req.hdr(host) backend identity mode http server identity ssl verify none sni req.hdr(host) backend smoketest mode http server smoketest ssl verify none sni req.hdr(host) backend healthcheck mode http http-request return status 299 backend loadtest mode http http-request return status 299 ``` ### Output of `haproxy -vv` ```plain HAProxy version 2.9.0-fddb8c1 2023/12/05 - https://haproxy.org/ Status: development branch - not safe for use in production. Known bugs: http://www.haproxy.org/bugs/bugs-2.9.0.html Running on: Linux 5.15.0-1041-azure #48-Ubuntu SMP Tue Jun 20 20:34:08 UTC 2023 x86_64 Build options : TARGET = linux-musl CPU = generic CC = cc CFLAGS = -O2 -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment OPTIONS = USE_PTHREAD_EMULATION=1 USE_LINUX_TPROXY=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_TFO=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1 DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS Feature list : -51DEGREES +ACCEPT4 -BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX +PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB Default settings : bufsize = 16384, maxrewrite = 1024, maxpollevents = 200 Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=32). Built with OpenSSL version : OpenSSL 3.1.4 24 Oct 2023 Running on OpenSSL version : OpenSSL 3.1.4 24 Oct 2023 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3 OpenSSL providers loaded : default Built with Lua version : Lua 5.4.6 Built with the Prometheus exporter as a service Built with network namespace support. Built with libslz for stateless compression. Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip") Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Built with PCRE2 version : 10.42 2022-12-11 PCRE2 library supports JIT : yes Encrypted password support via crypt(3): yes Built with gcc compiler version 12.2.1 20220924 Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. Available multiplexer protocols : (protocols marked as <default> cannot be specified using 'proto' keyword) quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG <default> : mode=HTTP side=FE|BE mux=H1 flags=HTX h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG <default> : mode=TCP side=FE|BE mux=PASS flags= none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG Available services : prometheus-exporter Available filters : [BWLIM] bwlim-in [BWLIM] bwlim-out [CACHE] cache [COMP] compression [FCGI] fcgi-app [SPOE] spoe [TRACE] trace ``` ### Last Outputs and Backtraces _No response_ ### Additional Information _No response_

Topic		Replies	Views
HTX (http-use-htx) and represention of HTTP headers Help!	4	7564	March 22, 2022
Mutating response headers mystery Help!	6	34	August 26, 2024
Moving from HAPROXY 1.8 to 2.4 and a backend service is returning a HTTP 400 Help!	2	454	January 6, 2024
Help interpreting Statistics Report data Help!	1	392	July 1, 2020
Randomly get 502 errors Help!	1	1499	November 28, 2022

2.8.x to 2.9 Sudden 411 issues

Related topics