Port doubling with set-dst-port

Help!
1

Hi,
We use HAProxy as our load balancer and as a stickiness mechanism to direct requests to specific backend servers. We recently added a new endpoint to our backend service and so we need to set the back port dynamically with set-dst-port depending on acl evaluation. We set the server using the HAProxy runtime API and update list of backend servers with backend server ip and port 0. Previously we just used whatever port value was stored statically with the address. This change caused port numbers to start getting doubled in requests to backend. So for the scenarios where we wanted to set the port to 19443, the port was always getting set to 38886, for all requests (causing a total outage). Is there any scenario where port numbers that were previous set in dynamo db would be added with ports set with set-dst-port ? or any known bugs where port numbers get doubled ? It seems too coincidental to have 38886 as the port used when we wanted port 19443. I have listed our haproxy config file and the haproxy -vv output.

global
daemon
maxconn 50200

# Stop HAProxy 60s after having been attempted to gracefully shut down.
hard-stop-after 600s

nbthread 15
cpu-map auto:1/1-15 0-14

stats socket /var/run/haproxy-1.sock mode 600 expose-fd listeners level admin
server-state-file var/run/haproxy_server_state

# syslog not is not running in RDE, so this will produce errors when
# starting this in a container.
# for containers can use : log stdout len 4096 local0  debug
log ring@localring local0 debug
tune.ssl.default-dh-param 2048

defaults
option log-health-checks
option log-separate-errors
load-server-state-from-file global

# documentation: https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#3.9-timeout%20connect
# times out after 1s
timeout connect 600ms
# The timeout client and timeout server measure time of inactivity after connection has been established
# in an on-going request (that's what differentiates it from an idle timeout). It is reset any time it sees any data
timeout client 900ms
timeout server 900ms
# See https://cbonte.github.io/haproxy-dconv/2.4/configuration.html#timeout%20tunnel
# This is particularly applicable to websocket connections and we disconnect after 1m of being idle (typically won't happen due to ping pong)
timeout tunnel 60s
timeout client-fin 10s
log global
option  httplog
option  logasap
option  contstats
mode http
option forwardfor
option dontlognull

ring localring
description “Local log buffer”
format rfc3164
maxlen 1500
size 131056
timeout connect 5s
timeout client 5s
timeout server 10s
server localsyslogsrv 127.0.0.1:6514 log-proto octet-count

peers haproxy
peer peer1 10.1.114.170:18080
peer peer2 10.2.114.229:18080
peer peer3 10.3.169.209:18080
peer peer4 10.4.159.55:18080

frontend health_check
maxconn 100
bind 0.0.0.0:8080
monitor-uri /ping

frontend stats
maxconn 100
bind *:8404
http-request use-service prometheus-exporter if { path /metrics }

frontend rms_frontend

acl is_ws_req path /ws
acl is_delete_req path /v1/delete

maxconn 50000
bind 0.0.0.0:8443 ssl crt var/etc/tls.pem
option http-keep-alive

# used for logging purpose
http-request capture req.hdr(identifier5) len 15 if is_ws_req
http-request capture req.hdr(identifier6) len 15 if is_delete_req
http-request capture req.hdr(identifier1) len 50
http-request capture req.hdr(identifier2) len 50
http-request capture req.hdr(identifier3) len 15
http-request capture req.hdr(identifier4) len 20
# https://docs.haproxy.org/2.8/configuration.html#log-format.
# https://www.haproxy.com/documentation/haproxy-configuration-manual/latest/#8.2.4
log-format "%ci:%cp [%tr] %ft %b/%s %TR %Tw %Tc %Tr %Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hs %{+Q}r %bi %bp %si %sp %[capture.req.hdr(0)] %[capture.req.hdr(1)] %[capture.req.hdr(2)] %[capture.req.hdr(3)] %[capture.req.hdr(4)]"

http-request deny if !is_ws_req !is_delete_req
# use_backend rms_backend if is_ws_req is_delete_req
default_backend ws_backend

backend ws_backend

acl is_ws_req path /ws
acl is_delete_req path /v1/delete

# set destination port based on requested endpoint
# delete endpoint port
http-request set-dst-port int(8083) if is_delete_req
# port for ws
http-request set-dst-port int(19443) if is_ws_req

stick-table type string len 48 size 1m expire 1200s peers haproxy srvkey addr
# match and add to sticky table only for playback requests
stick on req.hdr(identifier5) if is_ws_req
# only use sticky table for matching, don't add for delete requests
stick match req.hdr(identifier6) if is_delete_req

balance roundrobin
option http-keep-alive
# https://docs.haproxy.org/2.8/configuration.html#retries
retries 1
# https://docs.haproxy.org/2.8/configuration.html#option%20redispatch
# 1 indicates a redispatch on every retry
option redispatch 1
retry-on empty-response 0rtt-rejected conn-failure response-timeout 503 500
http-reuse always
option httpchk
option httpchk GET /ping HTTP/1.1
http-check send hdr Host localhost
http-check expect status 200
default-server inter 2s fall 3 rise 1 weight 100 init-addr last,libc,none

# setting server port to 0, as that is dynamically set based on endpoint
server-template server 4000 localhost:0 disabled ssl ca-file /etc/cacerts/amazon-root-ca-rsa-2k-1.pem check port 8084 check-ssl

++ export AWS_REGION=us-east-1
++ AWS_REGION=us-east-1
++ export HAPROXY_MAX_BACKENDS=4000
++ HAPROXY_MAX_BACKENDS=4000

  • unset APOLLO_ENV_INITIALIZING
  • echo_verbose
  • ‘[’ 0 ‘!=’ 0 ‘]’
  • echo_verbose ‘(4) Executing ‘'‘haproxy’'’’
  • ‘[’ 0 ‘!=’ 0 ‘]’
  • ‘[’ -f /haproxy ‘]’
  • echo_verbose 'Path ‘'‘haproxy’'’ does not exist relative to environment root ’
  • ‘[’ 0 ‘!=’ 0 ‘]’
  • echo_verbose ‘Executing ‘'‘haproxy’'’ plainly from the command line’
  • ‘[’ 0 ‘!=’ 0 ‘]’
  • COMMAND=haproxy
  • echo_verbose ‘$’ ‘exec’ haproxy ‘-vv’
  • ‘[’ 0 ‘!=’ 0 ‘]’
  • echo_verbose
  • ‘[’ 0 ‘!=’ 0 ‘]’
  • exec haproxy -vv
    HAProxy version 2.8.9 2024/04/05 - https://haproxy.org/
    Status: long-term supported branch - will stop receiving fixes around Q2 2028.
    Known bugs: http://www.haproxy.org/bugs/bugs-2.8.9.html
    Running on: Linux 4.14.348-265.562.amzn2.x86_64 #1 SMP Fri Jun 14 09:53:28 UTC 2024 x86_64
    Build options :
    TARGET = linux-glibc
    CPU = generic
    CC = x86_64-pc-linux-gnu-gcc
    CFLAGS = -Wchar-subscripts -Wcomment -Wformat -Winit-self -Wmain -Wmissing-braces -Wno-pragmas -Wparentheses -Wreturn-type -Wsequence-point -Wstrict-aliasing -Wswitch -Wtrigraphs -Wuninitialized -Wunknown-pragmas -Wunused-label -Wunused-variable -Wunused-value -Wno-error=missing-include-dirs -Wno-error=narrowing -Wpointer-sign -Wimplicit -pthread -fdiagnostics-color=auto -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -O3 -msse -mfpmath=sse -march=core2 -g -fPIC -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -Wfatal-errors -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wno-atomic-alignment
    OPTIONS = USE_OPENSSL=1 USE_LUA=1 USE_SLZ=1 USE_PROMEX=1 USE_PCRE2=1 USE_PCRE2_JIT=1
    DEBUG = -DDEBUG_STRICT -DDEBUG_MEMORY_POOLS

Feature list : -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS +DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT +LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER +NS -OBSOLETE_LINKER +OPENSSL -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 +PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL +PROMEX -PTHREAD_EMULATION -QUIC -QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ +SSL -STATIC_PCRE -STATIC_PCRE2 -SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB

Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, default=16).
Built with OpenSSL version : OpenSSL 1.1.1y 04 JUN 2024
Running on OpenSSL version : OpenSSL 1.1.1y 04 JUN 2024
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.4.4
Built with the Prometheus exporter as a service
Built with network namespace support.
Built with libslz for stateless compression.
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with PCRE2 version : 10.42 2022-12-11
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with gcc compiler version 11.4.0

Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTTP side=FE|BE mux=H2 flags=HTX|HOL_RISK|NO_UPG
fcgi : mode=HTTP side=BE mux=FCGI flags=HTX|HOL_RISK|NO_UPG
: mode=HTTP side=FE|BE mux=H1 flags=HTX
h1 : mode=HTTP side=FE|BE mux=H1 flags=HTX|NO_UPG
: mode=TCP side=FE|BE mux=PASS flags=
none : mode=TCP side=FE|BE mux=PASS flags=NO_UPG

Available services : prometheus-exporter
Available filters :
[BWLIM] bwlim-in
[BWLIM] bwlim-out
[CACHE] cache
[COMP] compression
[FCGI] fcgi-app
[SPOE] spoe
[TRACE] trace

2

Discussed in: