1.7.11 Compression Issue? Parsing errors on response


#1

Hey

We recently upgraded from 1.7.10 to 1.7.11 and we started to notice some sporadic issues in some of the responses in our application. Specifically some searches via Solr started to fail with what looks to be somewhat garbled responses.

We were at a loss at what was causing this until we took HAProxy out of the equation and issue immediately disappeared. Introducing it back in and then switching off compression also sorted the issue.

So I was just wondering if there was any changes to the compression behavior in 1.7.11? We just use gzip compression and not changed anything on our end. Until this is sorted we’ll revert back to 1.7.10.

Cheers,
R


#2

Can you provide the output of haproxy -vv from both 1.7.10 and the failing 1.7.11, as well as the configuration (without confidential data).


#3

Hey @lukastribus. Here’s the outputs and config:

1.7.11:

HA-Proxy version 1.7.11 2018/04/30
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>
Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -fno-strict-overflow
  OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
	  epoll : pref=300,  test result OK
	   poll : pref=200,  test result OK
	 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
		[COMP] compression
		[TRACE] trace
		[SPOE] spoe

1.7.10

HA-Proxy version 1.7.10-a7dcc3b 2018/01/02
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv
  OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips  26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.32 2012-11-30
Running on PCRE version : 8.32 2012-11-30
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.4
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
	  epoll : pref=300,  test result OK
	   poll : pref=200,  test result OK
	 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
		[COMP] compression
		[TRACE] trace
		[SPOE] spoe

Sanitized Config:

global
    log 127.0.0.1 local2 info
    maxconn 2000
    tune.ssl.default-dh-param 2048
    tune.maxrewrite 16384
    tune.bufsize 32768
    user admin
    group admin
    stats socket /etc/haproxy/haproxysock level admin

defaults
    log global
    mode http
    compression algo gzip
    compression type text/html text/plain text/javascript application/javascript application/xml text/css image/png font/woff font/woff2
    default-server inter 10s init-addr last,libc,none resolve-prefer ipv4
    option httplog
    option dontlognull
    option redispatch
    option forceclose
    option forwardfor
    retries 3
    timeout connect  5000
    timeout client  160000
    timeout server  160000

resolvers serverdns
    nameserver dnsmasq 127.0.0.1:53

frontend em
    bind *:80
    bind *:443 ssl crt /etc/haproxy/server.lab.local.pem

    acl https ssl_fc
    reqadd X-Forwarded-Proto:\ https if https
    redirect scheme https if !https

    default_backend server_backend

backend server_backend
    mode http
    cookie EXTERNALSERVERID insert indirect nocache
    option httpchk /health
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request set-header X-Forwarded-Proto https if { ssl_fc }

    server server1 server.lab.local:8280 cookie server1 check resolvers serverdns ssl ca-file /etc/haproxy/chain.crt

listen stats
    bind *:1936
    mode http
    balance
    timeout client 160000
    timeout connect 5000
    timeout server 160000
    stats uri /
    stats realm HAProxy\ Statistics
    stats admin if TRUE