opened 03:55PM - 30 Aug 24 UTC
status: works as designed
status: future fix
### Detailed Description of the Problem
We use HAProxy as our load balancer and… as a stickiness mechanism to direct requests to specific backend servers. We recently added a new endpoint to our backend service and so we need to set the back port dynamically with set-dst-port depending on acl evaluation. We set the server using the HAProxy runtime API and update list of backend servers with backend server ip and port 0. Previously we just used whatever port value was stored statically with the address. This change caused port numbers to start getting doubled in requests to backend. So for the scenarios where we wanted to set the port to 19443, the port was always getting set to 38886, for all requests (causing a total outage). Is there any scenario where port numbers that were previous set in dynamo db would be added with ports set with set-dst-port ? or any known bugs where port numbers get doubled ? It seems too coincidental to have 38886 as the port used when we wanted port 19443.
### Expected Behavior
Whatever port is loaded to haproxy, it gets overwritten by the command `http-request set-dst-port int(19443) if is_ws_req`
### Steps to Reproduce the Behavior
1. Load the provided HAProxy configuration on an AWS EC2 instance
2. Perform a refresh on that instance
3. Serve traffic
### Do you have any idea what may have caused this?
Starting to use dynamic ports
### Do you have an idea how to solve the issue?
_No response_
### What is your configuration?
```haproxy
`global
daemon
maxconn 50200
# Stop HAProxy 60s after having been attempted to gracefully shut down.
hard-stop-after 600s
nbthread 15
cpu-map auto:1/1-15 0-14
stats socket /var/run/haproxy-1.sock mode 600 expose-fd listeners level admin
server-state-file var/run/haproxy_server_state
# syslog not is not running in RDE, so this will produce errors when
# starting this in a container.
# for containers can use : log stdout len 4096 local0 debug
log ring@localring local0 debug
tune.ssl.default-dh-param 2048
defaults
option log-health-checks
option log-separate-errors
load-server-state-from-file global
# documentation: https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#3.9-timeout%20connect
# times out after 1s
timeout connect 600ms
# The timeout client and timeout server measure time of inactivity after connection has been established
# in an on-going request (that's what differentiates it from an idle timeout). It is reset any time it sees any data
timeout client 900ms
timeout server 900ms
# See https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#timeout%20tunnel
# This is particularly applicable to websocket connections and we disconnect after 1m of being idle (typically won't happen due to ping pong)
timeout tunnel 60s
timeout client-fin 10s
log global
option httplog
option logasap
option contstats
mode http
option forwardfor
option dontlognull
ring localring
description "Local log buffer"
format rfc3164
maxlen 1500
size 131056
timeout connect 5s
timeout client 5s
timeout server 10s
server localsyslogsrv 127.0.0.1:6514 log-proto octet-count
peers haproxy
peer peer1 10.1.114.170:18080
peer peer2 10.2.114.229:18080
peer peer3 10.3.169.209:18080
peer peer4 10.4.159.55:18080
frontend health_check
maxconn 100
bind 0.0.0.0:8080
monitor-uri /ping
frontend stats
maxconn 100
bind *:8404
http-request use-service prometheus-exporter if { path /metrics }
frontend ws_frontend
acl is_ws_req path /ws
acl is_delete_req path /v1/delete
maxconn 50000
bind 0.0.0.0:8443 ssl crt var/etc/tls.pem
option http-keep-alive
# used for logging purpose
http-request capture req.hdr(identifier5) len 15 if is_ws_req
http-request capture req.hdr(identifier6) len 15 if is_delete_req
http-request capture req.hdr(identifier1) len 50
http-request capture req.hdr(identifier2) len 50
http-request capture req.hdr(identifier3) len 15
http-request capture req.hdr(identifier4) len 20
# https://docs.haproxy.org/2.8/configuration.html#log-format.
# https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#8.2.4
log-format "%ci:%cp [%tr] %ft %b/%s %TR %Tw %Tc %Tr %Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hs %{+Q}r %bi %bp %si %sp %[capture.req.hdr(0)] %[capture.req.hdr(1)] %[capture.req.hdr(2)] %[capture.req.hdr(3)] %[capture.req.hdr(4)]"
http-request deny if !is_ws_req !is_delete_req
# use_backend ws_backend if is_ws_req is_delete_req
default_backend ws_backend
backend ws_backend
acl is_ws_req path /ws
acl is_delete_req path /v1/delete
# set destination port based on requested endpoint
# delete endpoint port
http-request set-dst-port int(8083) if is_delete_req
# port for ws
http-request set-dst-port int(19443) if is_ws_req
stick-table type string len 48 size 1m expire 1200s peers haproxy srvkey addr
# match and add to sticky table only for playback requests
stick on req.hdr(identifier5) if is_ws_req
# only use sticky table for matching, don't add for delete requests
stick match req.hdr(identifier6) if is_delete_req
balance roundrobin
option http-keep-alive
# https://docs.haproxy.org/2.8/configuration.html#retries
retries 1
# https://docs.haproxy.org/2.8/configuration.html#option%20redispatch
# 1 indicates a redispatch on every retry
option redispatch 1
retry-on empty-response 0rtt-rejected conn-failure response-timeout 503 500
http-reuse always
option httpchk
option httpchk GET /ping HTTP/1.1
http-check send hdr Host localhost
http-check expect status 200
default-server inter 2s fall 3 rise 1 weight 100 init-addr last,libc,none
# setting server port to 0, as that is dynamically set based on endpoint
server-template server 4000 localhost:0 disabled ssl ca-file /etc/cacerts/amazon-root-ca-rsa-2k-1.pem check port 8084 check-ssl`
```
### Output of `haproxy -vv`
```plain
++ export AWS_REGION=us-east-1
++ AWS_REGION=us-east-1
++ export HAPROXY_MAX_BACKENDS=4000
++ HAPROXY_MAX_BACKENDS=4000
+ unset APOLLO_ENV_INITIALIZING
+ echo_verbose
+ '[' 0 '!=' 0 ']'
+ echo_verbose '(4) Executing '\''haproxy'\'''
+ '[' 0 '!=' 0 ']'
+ '[' -f /haproxy ']'
+ echo_verbose 'Path '\''haproxy'\'' does not exist relative to environment root '
+ '[' 0 '!=' 0 ']'
+ echo_verbose 'Executing '\''haproxy'\'' plainly from the command line'
+ '[' 0 '!=' 0 ']'
+ COMMAND=haproxy
+ echo_verbose '$' 'exec' haproxy '-vv'
+ '[' 0 '!=' 0 ']'
+ echo_verbose
+ '[' 0 '!=' 0 ']'
+ exec haproxy -vv
HAProxy version 2.8.9 2024/04/05 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2028.
Known bugs: http://www.haproxy.org/bugs/bugs-2.8.9.html
Running on: Linux 4.14.348-265.562.amzn2.x86_64 #1 SMP Fri Jun 14 09:53:28 UTC 2024 x86_64
Build options :
TARGET = linux-glibc
CPU = generic
CC = x86_64-pc-linux-gnu-gcc
CFLAGS = -Wchar-subscripts -Wcomment -Wformat -Winit-self -Wmain -Wmissing-braces -Wno-pragmas -Wparentheses -Wreturn-type -Wsequence-point -Wstrict-aliasing -Wswitch -Wtrigraphs -Wuninitialized -Wunknown-pragmas -Wunused-label -Wunused-variable -Wunused-value -Wno-error=missing-include-dirs -Wno-error=narrowing -Wpointer-sign -Wimplicit -pthread -fdiagnostics-color=auto -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -O3 -msse -mfpmath=sse -march=core2 -g -fPIC -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS
Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=16).
Built with OpenSSL version : OpenSSL 1.1.1y 04 JUN 2024
Running on OpenSSL version : OpenSSL 1.1.1y 04 JUN 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
<default> : mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
<default> : mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG
Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace
```
### Last Outputs and Backtraces
_No response_
### Additional Information
_No response_