The haproxy configuration is exactly the same as we copied it from the v1.8 box.
Post migration, we are seeing a lot of 502 bad gateway errors in our analytics tool (datadog). Strangely, we are not seeing any 502 in the haproxy logs in itself.
The v1.8 box was running Centos 7.9 and the v3.0.3 is running RL 8.10. Everything else (CPU, memory, configuration etc. everything is the same)
haproxy -vv output:
HAProxy version 3.0.3-95a607c 2024/07/11 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2029.
Known bugs: http://www.haproxy.org/bugs/bugs-3.0.3.html
Running on: Linux 4.18.0-553.44.1.el8_10.x86_64 #1 SMP Mon Mar 10 11:32:40 UTC 2025 x86_64
Build options :
TARGET = linux-glibc
CC = cc
CFLAGS = -O2 -g -fwrapv
OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1
DEBUG =
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT +PCRE -PCRE2 -PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 +SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=8).
Built with OpenSSL version : OpenSSL 1.1.1k FIPS 25 Mar 2021
Running on OpenSSL version : OpenSSL 1.1.1k FIPS 25 Mar 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.5
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE version : 8.42 2018-03-20
Running on PCRE version : 8.42 2018-03-20
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): yes
Built with gcc compiler version 8.5.0 20210514 (Red Hat 8.5.0-20)
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : none
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
We are using haproxy only for load balancing and not for SSL termination.
The only anomaly I am seeing in the logs is as below, however I am not sure if this has anything to do with the issue at hand:
Jun 13 04:12:15 localhost haproxy[529908]: xx.xx.xx.xx:62134 [13/Jun/2025:04:12:15.839] FE01 xxxx/app02_8022 0/6/-1/-1/8 503 217 - - SCNN 55/55/0/0/+3 0/0 {xxxx.com} "POST /apps/PortalController HTTP/1.1"
Jun 13 04:12:16 localhost haproxy[529908]: xx.xx.xx.xx:53429 [13/Jun/2025:04:12:16.383] FE01 xxxx/app02_8024 0/4/-1/-1/5 503 217 - - SCNN 54/54/0/0/+3 0/0 {xxxx.com} "POST /apps/PortalController HTTP/1.1"
Jun 13 04:31:37 localhost haproxy[720655]: xx.xx.xx.xx:60610 [13/Jun/2025:04:31:37.129] FE01 xxxx/app02_8022 0/4/-1/-1/6 503 217 - - SCNN 47/47/0/0/+3 0/0 {xxxx.com} "POST /apps/PortalController HTTP/1.1"
Jun 13 04:31:37 localhost haproxy[720655]: xx.xx.xx.xx:31864 [13/Jun/2025:04:31:37.267] FE01 xxxx/app02_8024 0/4/-1/-1/6 503 217 - - SCNN 49/49/0/0/+3 0/0 {xxxx.com} "POST /apps/PortalController HTTP/1.1"
Jun 13 04:36:24 localhost haproxy[530912]: xx.xx.xx.xx:30526 [13/Jun/2025:04:36:24.319] FE01 xxxx/app02_8022 0/5/-1/-1/6 503 217 - - SCNN 49/49/0/0/+3 0/0 {xxxx.com} "POST /apps/PortalController HTTP/1.1"
Jun 13 04:36:24 localhost haproxy[530912]: xx.xx.xx.xx:58905 [13/Jun/2025:04:36:24.555] FE01 xxxx/app02_8024 0/4/-1/-1/5 503 217 - - SCNN 50/50/0/0/+3 0/0 {xxxx.com} "POST /apps/PortalController HTTP/1.1"
Jun 13 04:48:00 localhost haproxy[534050]: xx.xx.xx.xx:38169 [13/Jun/2025:04:48:00.790] FE01 xxxx/app02_8022 0/3/-1/-1/5 503 217 - - SCNN 54/54/0/0/+3 0/0 {xxxx.com} "POST /apps/PortalController HTTP/1.1"
For some reason, we are seeing a latency increase of 10% in the URLs that are hosted in the new box. Please help and let me know if any more information is needed.
