Hi!
We have a problem with sending large http headers through haproxy 3.0.7-1ppa1~noble.
The backend server is a Ubuntu 22.04 with a nginx 1.18.0-6ubuntu14.5.
Few weeks ago the website was directly delivered from nginx on the backend server without any problems. To be able to put a maintenance site in front of the nginx while installing OS patches/reboot, we installed a new VM with haproxy in front of the nginx.
But now we have problems with http requests with large headers.
First we started with haproxy version 2.8.5-1ubuntu3.2 from ubuntu 22.04 repo. With this version we got only connection reset and no entry/message in haproxy.log. After upgrade to haproxy version 3.0.7-1ppa1~noble, we got 502 Bad Gateway response from haproxy and log entires in haproxy.log. And also the large http request was reaching the backend server (nginx).
The config:
haproxy -vv
HAProxy version 3.0.7-1ppa1~noble 2024/12/14 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2029.
Known bugs: http://www.haproxy.org/bugs/bugs-3.0.7.html
Running on: Linux 6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec 5 13:09:44 UTC 2024 x86_64
Build options :
TARGET = linux-glibc
CC = x86_64-linux-gnu-gcc
CFLAGS = -O2 -g -fwrapv -g -O2 -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -flto=auto -ffat-lto-objects -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -fdebug-prefix-map=/build/haproxy-kwUUeo/haproxy-3.0.7=/usr/src/haproxy-3.0.7-1ppa1~noble -Wdate-time -D_FORTIFY_SOURCE=3
OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_SYSTEMD=1 USE_OT=1 USE_QUIC=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1 USE_QUIC_OPENSSL_COMPAT=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL +OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION +QUIC +QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=4).
Built with OpenSSL version : OpenSSL 3.0.13 30 Jan 2024
Running on OpenSSL version : OpenSSL 3.0.13 30 Jan 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
OpenSSL providers loaded : default
Built with Lua version : Lua 5.4.6
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with OpenTracing support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 13.3.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
quic : mode=HTTP side=FE mux=QUIC flags=HTX|NO_UPG|FRAMED
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[ OT] opentracing
[SPOE] spoe
[TRACE] trace
cat /etc/haproxy/haproxy.cfg
global
log /dev/log local0
# log /dev/log local1 notice
log /dev/log local1 debug
chroot /var/lib/haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
stats timeout 30s
user haproxy
group haproxy
daemon
# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
# See: https://ssl-config.mozilla.org/#server=haproxy&server-version=2.0.3&config=intermediate
ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
tune.maxrewrite 8192
tune.bufsize 65535
defaults
log global
mode http
option httplog
option dontlognull
timeout connect 5000
timeout client 50000
timeout server 50000
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http
# haproxy statistic page
stats auth XXXX:XXXX
stats uri /haproxy?stats
stats realm Strictly\ Private
default-server check inter 2000 rise 2 fall 5
frontend site-xxx_at_fe
bind 10.x.x.x:80
bind 10.x.x.x:443 ssl crt /etc/haproxy/ssl/site-xxx.at.pem crt /etc/haproxy/ssl/ alpn h2,http/1.1
# sorry page to return when worst case happens and also backup backend is down
errorfile 503 /etc/haproxy/errors/503.http
acl vhost_site-xxx.at hdr(host) -i "stg1.site-xxx.at"
acl MAIN_backend_down nbsrv(site-xxx_at_be) eq 0
# https://www.iops.tech/blog/haproxy-dynamic-maintenance-page-using-unix-socket/
# empty regex rule; will be filled on maintenance
acl maintenance path -i -u 9699 -m reg
acl typo3_backend path_beg /typo3/
acl internal_ip src 10.0.0.0/8 # VPN
# drop connections without vhost set
http-request silent-drop if !vhost_site-xxx.at
http-request deny if typo3_backend !internal_ip
# redirect to https
redirect scheme https code 301 if !{ ssl_fc }
use_backend maintenance_be if MAIN_backend_down
use_backend maintenance_be if maintenance
default_backend site-xxx_at_be
backend site-xxx_at_be
# check with http OPTION request and ssl connect the /health urlpath
option httpchk OPTIONS /health
http-check connect ssl alpn h2
http-check send meth HEAD uri /health ver HTTP/2 hdr Host stg1.site-xxx.at
# add client IP header X-Forwarded-For
option forwardfor
default-server ssl verify none alpn h2,http/1.1
# backend server
server wstg1 10.x.x.x:443 check
backend maintenance_be
option httpchk OPTIONS /health
server wmain1 10.x.x.x:80 check
# we do not want a 200 in case of maintenance
http-after-response set-status 503
The large request with curl:
curl 'https://stg1.site-xxx.at/hotel/203106/?ddate=2025-01-08&rdate=2025-03-24&dur=7' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Accept-Language: de-AT,en-US;q=0.7,en;q=0.3' -H 'Accept-Encoding: gzip, deflate, br, zstd' -H 'Referer: https://www.site-xxx.com/hotels/r/353/?term=dubai&ddate=2025-01-08&rdate=2025-03-24&dur=7' -H 'DNT: 1' -H 'Sec-GPC: 1' -H 'Connection: keep-alive' -H 'Cookie: _ga_R6NBEK0SEH=GS1.1.1736259320.1.1.1736259560.0.0.1145470110; _ga=GA1.1.1752673086.1736259320; FPID=FPID2.2.djMpER1CAak43qrG8QmGBVJjKCil%2F28HgrGbT7zM61M%3D.1736259320; FPLC=N%2FZGw2Uv1nv6DMmmCTadQwis3TjLfvMHuju6DgImTwPJNVJoXpBNYpLg1CdqPgc9wVE4d6NMxOv8VfnubOy9nsm4D50YsPWZC7sZbvPvKCoUl8xpDSmdPn3PB1Dmgw%3D%3D; FPAU=1.2.1126819039.1736259320; FPGSID=1.1736259320.1736259517.G-R6NBEK0SEH.4kWwjpsJzcf2uM63MSE5Jw; favorites_RESTPLATZBOERSE-COM=%7B%22count%22:0,%22iff%22:%5B%5D,%22hotels%22:%5B%7B%22IffCode%22:174923,%22Name%22:%22Dukes%20The%20Palm,%20a%20Royal%20Hideaway%20Hotel%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:237868,%22Name%22:%22Radisson%20Beach%20Resort%20Palm%20Jumeirah%22,%22Region%22:%22Dubai%22,%22Category%22:4%7D,%7B%22IffCode%22:109967,%22Name%22:%22Sofitel%20Dubai%20The%20Palm%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:218746,%22Name%22:%22Hotel%20Riu%20Dubai%22,%22Region%22:%22Dubai%22,%22Category%22:4%7D,%7B%22IffCode%22:236999,%22Name%22:%22Marriott%20Resort%20Palm%20Jumeirah,%20Dubai%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:187970,%22Name%22:%22Rixos%20Premium%20Dubai%20JBR%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:189028,%22Name%22:%22The%20Retreat%20Palm%20Dubai%20MGallery%20by%20Sofitel%22,%22Region%22:%22Dubai%22,%22Category%22:4.5%7D,%7B%22IffCode%22:230352,%22Name%22:%22Taj%20Exotica%20Resort%20&%20Spa,%20The%20Palm%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:202978,%22Name%22:%22Royal%20Central%20Hotel%20-%20The%20Palm%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:91560,%22Name%22:%22Two%20Seasons%20Hotel%20&%20Apartments%22,%22Region%22:%22Dubai%22,%22Category%22:4%7D,%7B%22IffCode%22:61957,%22Name%22:%22The%20Westin%20Dubai%20Mina%20Seyahi%20Beach%20Resort%20&%20Marina%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:66289,%22Name%22:%22Atlantis,%20The%20Palm%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:228534,%22Name%22:%22Centara%20Mirage%20Beach%20Resort%20Dubai%22,%22Region%22:%22Dubai%22,%22Category%22:4%7D,%7B%22IffCode%22:196322,%22Name%22:%22W%20Dubai%20-%20The%20Palm%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:237993,%22Name%22:%22voco%20Dubai%20The%20Palm%22,%22Region%22:%22Dubai%22,%22Category%22:4%7D,%7B%22IffCode%22:76300,%22Name%22:%22Anantara%20The%20Palm%20Dubai%20Resort%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:229446,%22Name%22:%22NH%20Collection%20Dubai%20The%20Palm%22,%22Region%22:%22Dubai%22,%22Category%22:4%7D,%7B%22IffCode%22:210437,%22Name%22:%22Andaz%20Dubai%20The%20Palm%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:191742,%22Name%22:%22Raffles%20The%20Palm%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:191,%22Name%22:%22Le%20Royal%20Meridien%20Beach%20Resort%20&%20Spa%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:75093,%22Name%22:%22Rose%20Rayhaan%20by%20Rotana%22,%22Region%22:%22Dubai%22,%22Category%22:4%7D,%7B%22IffCode%22:22159,%22Name%22:%22Hilton%20Dubai%20Jumeirah%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:61820,%22Name%22:%22Amwaj%20Rotana%20-%20Jumeirah%20Beach%20Residence%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:195,%22Name%22:%22Al%20Habtoor%20Grand%20Resort,%20Autograph%20Collection%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D,%7B%22IffCode%22:203106,%22Name%22:%22Canal%20Central%20Hotel%20%E2%80%93%20Business%20Bay%22,%22Region%22:%22Dubai%22,%22Category%22:5%7D%5D%7D; rpb_lastSeenHotels=[{%22id%22:%2275093%22%2C%22name%22:%22Rose%20Rayhaan%20by%20Rotana%22%2C%22region%22:%22Dubai%22%2C%22category%22:4%2C%22urlparams%22:%22ddate=2025-01-08&rdate=2025-03-24&dur=7%22}%2C{%22id%22:%22191%22%2C%22name%22:%22Le%20Royal%20Meridien%20Beach%20Resort%20&%20Spa%22%2C%22region%22:%22Dubai%22%2C%22category%22:5%2C%22urlparams%22:%22ddate=2025-01-08&rdate=2025-03-24&dur=7%22}%2C{%22id%22:%22218746%22%2C%22name%22:%22Hotel%20Riu%20Dubai%22%2C%22region%22:%22Dubai%22%2C%22category%22:4%2C%22urlparams%22:%22ddate=2025-01-08&rdate=2025-03-24&dur=7%22}]' -H 'Upgrade-Insecure-Requests: 1' -H 'Sec-Fetch-Dest: document' -H 'Sec-Fetch-Mode: navigate' -H 'Sec-Fetch-Site: same-origin' -H 'Sec-Fetch-User: ?1' -H 'Priority: u=0, i' -H 'TE: trailers'
<html><body><h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
</body></html>
haproxy.log:
2025-01-29T12:20:58.347887+01:00 wsprox1 haproxy[1552886]: 10.x.x.x:46762 [29/Jan/2025:12:20:58.343] site-xxx_at_fe~ site-xxx_at_be/wstg1 0/0/2/-1/3 502 208 - - SH-- 1/1/0/0/0 0/0 "GET https://stg1.site-xxx.at/hotel/203106/?ddate=2025-01-08&rdate=2025-03-24&dur=7 HTTP/2.0"
nginx-access.log:
10.x.x.x - - [29/Jan/2025:12:26:24 +0100] "-" 000 0 "https://www.site-xxx.com/hotels/r/353/?term=dubai&ddate=2025-01-08&rdate=2025-03-24&dur=7" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0" -
As written, the large requests are working on nginx direct without haproxy.
I was playing around with these parameters in the haproxy.cfg:
tune.maxrewrite 8192
tune.bufsize 65535
but without success.
Is there any other setting in haproxy which is missing to get these large http header running through haproxy?
br, Marco