How to debug term code CD?

Help!
1

For downloading certain files via haproxy I’m often, but not always, getting a term code CD. Asking the backend directly for this file, always works. It’s just an image of ~300kb and the response time is never high. So I’m wondering what’s going on here, how can I debug this?

In browsers it results in an image only partially shown. And using curl I get: curl: (18) transfer closed with 62685 bytes remaining to read.

Feb  9 14:58:54 haproxy haproxy[6447]: 10.0.0.100:61330 [09/Feb/2022:14:58:54.516] www XXXX/XXXX 0/0/0/3/6 200 325957 - - ---- 7/3/0/0/0 0/0 {XXXX|curl/7.80.0||} {|} "GET /images/temp/bg-00.jpg HTTP/1.1"
Feb  9 14:58:54 haproxy haproxy[6447]: 10.0.0.100:61332 [09/Feb/2022:14:58:54.795] www XXXX/XXXX 0/0/0/4/7 200 325957 - - ---- 7/3/0/0/0 0/0 {XXXX|curl/7.80.0||} {|} "GET /images/temp/bg-00.jpg HTTP/1.1"
Feb  9 14:58:55 haproxy haproxy[6447]: 10.0.0.100:61334 [09/Feb/2022:14:58:55.080] www XXXX/XXXX 0/0/1/3/8 200 325957 - - ---- 7/3/0/0/0 0/0 {XXXX|curl/7.80.0||} {|} "GET /images/temp/bg-00.jpg HTTP/1.1"
Feb  9 14:58:55 haproxy haproxy[6447]: 10.0.0.100:61336 [09/Feb/2022:14:58:55.345] www XXXX/XXXX 0/0/1/3/7 200 325957 - - CD-- 7/3/0/0/0 0/0 {XXXX|curl/7.80.0||} {|} "GET /images/temp/bg-00.jpg HTTP/1.1"
Feb  9 14:58:55 haproxy haproxy[6447]: 10.0.0.100:61339 [09/Feb/2022:14:58:55.606] www XXXX/XXXX 0/0/0/3/6 200 325957 - - CD-- 7/3/0/0/0 0/0 {XXXX|curl/7.80.0||} {|} "GET /images/temp/bg-00.jpg HTTP/1.1"
Feb  9 14:58:55 haproxy haproxy[6447]: 10.0.0.100:61341 [09/Feb/2022:14:58:55.959] www XXXX/XXXX 0/0/1/4/8 200 325957 - - ---- 7/3/0/0/0 0/0 {XXXX|curl/7.80.0||} {|} "GET /images/temp/bg-00.jpg HTTP/1.1"
2

Please provide the configuration and the output of haproxy -vv.

3 
HAProxy version 2.4.11-1e183b8 2022/01/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.11.html
Running on: FreeBSD 13.0-RELEASE-p4 FreeBSD 13.0-RELEASE-p4 #0: Tue Aug 24 07:33:27 UTC 2021     root@amd64-builder.daemonology.net:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64
Build options :
  TARGET  = freebsd
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -pipe -fstack-protector-strong -fno-strict-aliasing -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-missing-field-initializers -Wno-string-plus-int -Wno-atomic-alignment -Wtype-limits -Wshift-negative-value -Wnull-dereference -DFREEBSD_PORTS
  OPTIONS = USE_PCRE=1 USE_PCRE_JIT=1 USE_STATIC_PCRE=1 USE_GETADDRINFO=1 USE_OPENSSL=1 USE_ACCEPT4=1 USE_ZLIB=1 USE_CPU_AFFINITY=1
  DEBUG   =

Feature list : -EPOLL +KQUEUE -NETFILTER +PCRE +PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -BACKTRACE +STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE +LIBCRYPT -CRYPT_H +GETADDRINFO +OPENSSL -LUA -FUTEX +ACCEPT4 +CLOSEFROM +ZLIB -SLZ +CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER -PRCTL +PROCCTL -THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=4).
Built with OpenSSL version : OpenSSL 1.1.1k-freebsd  24 Aug 2021
Running on OpenSSL version : OpenSSL 1.1.1k-freebsd  24 Aug 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY
Built with PCRE version : 8.45 2021-06-15
Running on PCRE version : 8.45 2021-06-15
PCRE library supports JIT : yes
Encrypted password support via crypt(3): yes
Built with clang compiler version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe)

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=

Available services : none

Available filters :
        [SPOE] spoe
        [CACHE] cache
        [FCGI] fcgi-app
        [COMP] compression
        [TRACE] trace

And the config:

global
  ssl-default-bind-options ssl-min-ver TLSv1.2 prefer-client-ciphers
  ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-bind-ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1:!AESCCM:@SECLEVEL=2

  ssl-default-server-options ssl-min-ver TLSv1.2
  ssl-default-server-ciphersuites TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256
  ssl-default-server-ciphers ECDH+AESGCM:ECDH+CHACHA20:ECDH+AES256:ECDH+AES128:!aNULL:!SHA1:!AESCCM:@SECLEVEL=2

  ssl-dh-param-file /usr/local/etc/haproxy/dhparams.pem
  tune.ssl.default-dh-param  2048

  log /var/run/log local0
  maxconn 4096
  user www
  group www
  daemon
  stats socket /var/run/haproxy.socket mode 600 expose-fd listeners level user

defaults
  log global
  mode http

  compression algo gzip
  compression type text/html text/plain text/css application/vnd.api+json application/json text/javascript

  option httpchk GET /

  option dontlognull
  option forwardfor
  option http-ignore-probes
  option http-server-close
  option httplog
  option redispatch

  timeout connect 5000
  timeout client  50000
  timeout server  50000


frontend www-ssl
  bind 10.0.0.100:443 ssl strict-sni crt /usr/local/etc/haproxy/certs alpn h2,http/1.1

  # Certbot
  acl letsencrypt-acl path_beg /.well-known/acme-challenge

  capture request header Host len 32
  capture request header User-agent len 128
  capture request header Content-Length len 100
  capture request header Referer len 200
  capture response header Cache-Control len 80
  capture response header Location len 200

  http-response add-header Strict-Transport-Security max-age=31536000
  http-response add-header X-Content-Type-Options nosniff
  http-response add-header Referrer-Policy strict-origin
  http-response add-header X-Frame-Options SAMEORIGIN
  http-response del-header Server

  http-request redirect location https://%[hdr(host),map(/usr/local/etc/haproxy/rewrite_domains.map)]%[capture.req.uri] code 301 if ! letsencrypt-acl { hdr(host),map(/usr/local/etc/haproxy/rewrite_domains.map) -i -m found }
  http-request add-header X-Forwarded-Proto https

  use_backend letsencrypt-backend if letsencrypt-acl
  use_backend %[ssl_fc_sni,lower,map_dom(/usr/local/etc/haproxy/https_domains.map)]

frontend www
  bind 10.0.0.100:80

  # Certbot
  acl letsencrypt-acl path_beg /.well-known/acme-challenge

  capture request header Host len 32  # idx 0
  capture request header User-agent len 128
  capture request header Content-Length len 100
  capture request header Referer len 200
  capture response header Cache-Control len 80
  capture response header Location len 200

  http-response add-header X-Content-Type-Options nosniff
  http-response add-header Referrer-Policy strict-origin
  http-response add-header X-Frame-Options SAMEORIGIN
  http-response del-header Server

  http-request redirect location http://%[hdr(host),map(/usr/local/etc/haproxy/rewrite_domains.map)]%[capture.req.uri] code 301 if ! letsencrypt-acl { req.hdr(host),map(/usr/local/etc/haproxy/rewrite_domains.map) -i -m found }

  http-request redirect scheme https code 301 if !{ ssl_fc } ! letsencrypt-acl { req.hdr(host),lower,map_dom(/usr/local/etc/haproxy/https_domains.map) -m found }

  use_backend letsencrypt-backend if letsencrypt-acl
  use_backend %[req.hdr(host),lower,map_dom(/usr/local/etc/haproxy/http_domains.map)]

backend XXXX
  server client-XXXX 10.0.0.13:4002


backend client_YYYYY
  option httpchk GET /test
  server client_YYYY 10.0.0.12:4000 check

backend varnish
  server varnish 10.0.0.104:8080 send-proxy-v2

backend letsencrypt-backend
  server letsencrypt 127.0.0.1:8000

Thanks!

4

Yeah, that’s a major bug in 2.4.11, 2.4.12 was released 4 days after the 2.4.11 release to fix this. You should upgrade to 2.4.12 as soon as possible.

5

Oh wow, I did not notice that release. Works great after the upgrade! Thanks.

1 Like