HAProxy Logs for Redirects

Help!
1

We have some redirects set up in HAProxy and when the redirection happens, we can see it in the logs as expected, but sometimes it reports the correct status_code, and other times it doesn’t. It’s almost like something it overwriting the %ST log variable so that when the redirect happens it logs that value instead of the value sent to the client. I should note that the operation of the redirects is working as intended–this is purely a problem with what we are seeing in the logs.

Given the following log settings:

global
  nbthread 4
  log 127.0.0.1:9515 local0
  log-send-hostname

defaults
  log global
  option dontlognull
  mode http

frontend http
  option httplog
  log-format %ci:%cp\ %T\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %H\ %{+Q}r
  redirect scheme https code 301

frontend default
  option httplog
  log-format %ci:%cp\ %T\ %ft\ %b/%s\ %Tq/%Tw/%Tc/%Tr/%Tt\ %ST\ %B\ %CC\ %CS\ %tsc\ %ac/%fc/%bc/%sc/%rc\ %sq/%bq\ %hr\ %H\ %{+Q}r
  redirect prefix https://location1 code 301 if is_acl_1 !is_acl_2

So when we send in an HTTP request, we correctly get a 301 to the client with the correct location and the logs show this as well

x.x.x.x:42973 06/Dec/2019:17:45:59 +0000 http http/<NOSRV> 0/-1/-1/-1/0 301 95 - - LR-- 7108/16/0/0/0 0/0 <hostname> "GET / HTTP/1.1"

When we send in an HTTPS request, we correctly get the 301 status_code to the client with the correct location, but our log looks like this

x.x.x.x:26080 06/Dec/2019:17:45:59 +0000 default default/<NOSRV> 60/-1/-1/-1/60 503 343 - - LR-- 7107/7090/0/0/0 0/0 {<headers>} <hostname> "GET <path> HTTP/1.1"

Please note that I have inserted the x.x.x.x, <headers>, <hostname>, and <path> strings to mask potentially sensitive data.

We do have a lot of other configuration values in there that I have left out, but we can start with this and I can add more configuration if needed to debug this further.

Haproxy version output

HA-Proxy version 2.0.7 2019/09/27 - https://haproxy.org/
Build options :
  TARGET  = linux-glibc
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-format-truncation -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wno-implicit-fallthrough -Wno-stringop-overflow -Wno-cast-function-type -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_PCRE2=1 USE_PCRE2_JIT=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_LUA=1 USE_ZLIB=1

Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT +PCRE2 +PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL +LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=8).
Built with OpenSSL version : OpenSSL 1.1.1c  28 May 2019
Running on OpenSSL version : OpenSSL 1.1.1c  28 May 2019
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with Lua version : Lua 5.3.3
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with PCRE2 version : 10.32 2018-09-10
PCRE2 library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with the Prometheus exporter as a service

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTX        side=FE|BE     mux=H2
              h2 : mode=HTTP       side=FE        mux=H2
       <default> : mode=HTX        side=FE|BE     mux=H1
       <default> : mode=TCP|HTTP   side=FE|BE     mux=PASS

Available services :
        prometheus-exporter

Available filters :
        [SPOE] spoe
        [COMP] compression
        [CACHE] cache
        [TRACE] trace