Everything working, but one page returns 502

Hi,

we have a haproxy instance handling multiple domains and getting the requests to the correct servers.

We have a nginx server, which runs fine without haproxy and mostly runs fine with haproxy. But one single page returns an 502 error:

Sep 18 15:00:28 dktig-proxy haproxy[2367]: <IP>:41656 [18/Sep/2018:15:00:28.473] ft_https~ bk_dkv/dkv 4/0/0/-1/6 502 16189 - - PH-- 1/1/0/0/0 0/0 "GET /de/suche/comparison/compare.html HTTP/1.1"

nginx has not much to say about this:

172.16.3.252 - - [18/Sep/2018:15:00:33 +0200] "GET /de/suche/comparison/compare.html HTTP/1.1" 499 0 "https://<domain>/de/suche/search/memo/show/1/asc/name.html" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36"

I already tried setting “accept-invalid-http-response”, but no change.

Here are now coming (relevant parts of) the config and the “show errors” output.

global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Descrease the size of the rewrite buffer so that
        # there is more room to handle large (>8K) headers. See the HAProxy
        # manual around tune.bufsize and tune.maxrewrite.
        tune.maxrewrite 4096

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/haproxy/cert

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3
        tune.ssl.default-dh-param 2048

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 60s
        timeout client  300s
        timeout server  300s
        timeout http-request 120s
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend ft_http
        bind :80
        mode http
        option httplog
        redirect scheme https code 301 if !{ ssl_fc }

frontend ft_https
        bind :443 ssl crt <alot of certs here> strict-sni
        tcp-request inspect-delay 5s
        tcp-request content accept if { req_ssl_hello_type 1 }
        mode http
        reqadd X-Forwarded-Proto:\ https
        option httplog
        acl is_gitlab hdr(host) -i <DOMAIN>
        acl is_europ hdr(host) -i <DOMAIN>
        acl is_ip hdr(host) -i <IP>
        use_backend bk_gitlab if is_gitlab
        use_backend bk_europ if is_europ
        tcp-request connection reject if is_ip
        default_backend bk_dkv

backend bk_europ
        mode http
        balance roundrobin
        server gitlab 172.16.3.50:80

backend bk_gitlab
        mode http
        balance roundrobin
        server gitlab 172.16.3.50:80

backend bk_dkv
        mode http
        balance roundrobin
        option accept-invalid-http-response
        server dkv 172.16.3.45:443 ssl verify none
Total events captured on [18/Sep/2018:14:58:38.821] : 2
 
[18/Sep/2018:14:58:36.027] backend bk_dkv (#7): invalid response
  frontend ft_https (#3), server dkv (#1), event #1
  src <IP>:41546, session #83, session flags 0x002004ce
  HTTP msg state 26, msg flags 0x00000000, tx flags 0xa8000000
  HTTP chunk len 0 bytes, HTTP body len 0 bytes
  buffer flags 0x80048002, out 0 bytes, total 15984 bytes
  pending 4088 bytes, wrapping at 16392, error at position 0:
 
  00000  n_menu_toolbar.css":1,"modules\\/shortcut\\/shortcut.css":1,"sites\\/a
  00067+ ll\\/modules\\/ctools\\/css\\/ctools.css":1,"sites\\/all\\/libraries\\
  00130+ /fancybox\\/source\\/jquery.fancybox.css":1,"sites\\/all\\/modules\\/p
  00195+ anels\\/css\\/panels.css":1,"sites\\/all\\/modules\\/devel\\/devel_kru
  00259+ mo.css":1,"0":1,"public:\\/\\/pr_suche_sitesallmodulesproxyreset.css":
  00327+ 1,"public:\\/\\/pr_suche_sitesallmodulesproxydefault.css":1,"sites\\/a
  00394+ ll\\/themes\\/dkg\\/bootstrap\\/dist\\/css\\/bootstrap.min.css":1,"sit
  00458+ es\\/all\\/themes\\/dkg\\/bootstrap\\/dist\\/css\\/bootstrap-theme.min
  00521+ .css":1,"sites\\/all\\/themes\\/promato\\/smartmenus\\/addons\\/bootst
  00585+ rap\\/jquery.smartmenus.bootstrap.css":1,"sites\\/all\\/themes\\/proma
  00651+ to\\/css\\/nf-main.css":1,"sites\\/all\\/themes\\/promato\\/css\\/nf-w
  00714+ ebform.css":1,"sites\\/all\\/themes\\/promato\\/css\\/nf-news.css":1,"
  00779+ sites\\/all\\/themes\\/promato\\/css\\/nf-icons.css":1,"sites\\/all\\/
  00842+ themes\\/promato\\/css\\/nf-gallery.css":1,"sites\\/all\\/themes\\/pro
  00906+ mato\\/font-awesome\\/css\\/font-awesome.min.css":1,"sites\\/all\\/the
  00971+ mes\\/dkv\\/css\\/disable-responsive.css":1,"sites\\/all\\/themes\\/dk
  01035+ v\\/css\\/dkv.css":1,"sites\\/all\\/themes\\/dkg\\/css\\/override.css"
  01098+ :1}},"fancybox":[],"overlay":{"paths":{"admin":"node\\/*\\/webform\\nn
  01165+ ode\\/*\\/webform\\/*\\nnode\\/*\\/webform-results\\nnode\\/*\\/webfor
  01226+ m-results\\/*\\nnode\\/*\\/submission\\/*\\nnode\\/*\\/edit\\nnode\\/*
  01286+ \\/delete\\nnode\\/*\\/revisions\\nnode\\/*\\/revisions\\/*\\/revert\\
  01346+ nnode\\/*\\/revisions\\/*\\/delete\\nnode\\/add\\nnode\\/add\\/*\\nove
  01406+ rlay\\/dismiss-message\\nuser\\/*\\/shortcuts\\nadmin\\nadmin\\/*\\nba
  01468+ tch\\ntaxonomy\\/term\\/*\\/edit\\nnode\\/*\\/translate\\nuser\\/*\\/c
  01528+ ancel\\nuser\\/*\\/edit\\nuser\\/*\\/edit\\/*\\ntaxonomy\\/*\\/transla
  01588+ te\\ntaxonomy\\/*\\/translate\\/*\\ndevel\\/*\\nnode\\/*\\/devel\\nnod
  01648+ e\\/*\\/devel\\/*\\ncomment\\/*\\/devel\\ncomment\\/*\\/devel\\/*\\nus
  01707+ er\\/*\\/devel\\nuser\\/*\\/devel\\/*\\ntaxonomy\\/term\\/*\\/devel\\n
  01766+ taxonomy\\/term\\/*\\/devel\\/*\\nnode\\/*\\/revisions\\/view\\/*\\/*"
  01826+ ,"non_admin":"admin\\/structure\\/block\\/demo\\/*\\nadmin\\/reports\\
  01889+ /status\\/php"},"pathPrefixes":["de","en"],"ajaxCallback":"overlay-aja
  01958+ x"},"tableHeaderOffset":"Drupal.admin.height","admin_menu":{"destinati
  02028+ on":"destination=suche\\/search\\/memo.html","hash":"b2f84c002aa2c65b2
  02096+ 56f81e58c48466e","basePath":"\\/de\\/admin_menu","margin_top":1,"posit
  02164+ ion_fixed":1,"toolbar":[]},"bootstrap":{"anchorsFix":"1","anchorsSmoot
  02234+ hScrolling":"1","formHasError":1,"popoverEnabled":"1","popoverOptions"
  02304+ :{"animation":1,"html":0,"placement":"right","selector":"","trigger":"
  02374+ click","triggerAutoclose":1,"title":"","content":"","delay":0,"contain
  02444+ er":"body"},"tooltipEnabled":"1","tooltipOptions":{"animation":1,"html
  02514+ ":0,"placement":"auto left","selector":"","trigger":"hover focus","del
  02584+ ay":0,"container":"body"}}});</script>\n
  02623  </head>\n
  02631  <body class="html not-front logged-in no-sidebars page-suche page-such
  02701+ e-search page-suche-search-memohtml i18n-de" >\n
  02748    <div id="skip-link">\n
  02771      <a href="#main-content" class="element-invisible element-focusable
  02841+ ">Skip to main content</a>\n
  02868    </div>\n
  02877      <div class="region region-page-top">\n
  02918      <div id="overlay-disable-message" class="clearfix"><h3 class="elem
  02988+ ent-invisible">Options for the administrative overlay</h3><a href="/de
  03058+ /user/1/edit?destination=suche/search/memo.html#edit-overlay-control" 
  03128+ id="overlay-profile-link" class="overlay-exclude element-invisible">If
  03198+  you have problems accessing administrative pages on this site, disabl
  03268+ e the overlay on your profile page.</a> <a href="/de/overlay/dismiss-m
  03338+ essage?destination=suche/search/memo.html&amp;token=jfmKm3UoGUx7x-ruMN
  03408+ JGetOoc5_T3hq7BnZhCERKsPw" id="overlay-dismiss-message" class="overlay
  03478+ -exclude element-invisible">Dismiss this message.</a></div>  </div>\n
  03546    <div class="container header-image">\n
  03585  \t      <!--<a class="logo navbar-btn pull-left" href="/de" title="Hom
  03654+ e">\n
  03658          <img src="https://<DOMAIN>/sites/all/themes/dkg/logo.png
  03728+ " alt="Home" />\n
  03744        </a>-->\n
  03758              <div class="site_name"><h1 style="color:#11574c;">DEUTSCHE
  03828+ S KRANKENHAUSVERZEICHNIS</h1></div>\n
  03864        <!--<div class="site_slogan"><span style="font-size:85%;">Servic
  03934+ e of the hospitals in berlin and the berlin hospital association in co
  04004+ nnection with the german hospital directory</span></div>-->\n
  04064        <div class="langua
HA-Proxy version 1.6.3 2015/12/25
Copyright 2000-2015 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with OpenSSL version : OpenSSL 1.0.2g-fips  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 8.38 2015-11-23
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Does someone sees anything odd? What can I try to resolve this problem? Any help is much appreciated.

The same problem persists with haproxy 1.8, but the show errors is slightly different.

HA-Proxy version 1.8.13-1ppa1~xenial 2018/08/01
Copyright 2000-2018 Willy Tarreau <willy@haproxy.org>

Build options :
  TARGET  = linux2628
  CPU     = generic
  CC      = gcc
  CFLAGS  = -g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
Running on OpenSSL version : OpenSSL 1.0.2g  1 Mar 2016
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.38 2015-11-23
Running on PCRE version : 8.38 2015-11-23
PCRE library supports JIT : yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
      epoll : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
        [SPOE] spoe
        [COMP] compression
        [TRACE] trace
Total events captured on [18/Sep/2018:15:21:50.767] : 1
 
[18/Sep/2018:15:21:38.581] backend bk_dkv (#7): invalid response
  frontend ft_https (#3), server dkv (#1), event #0
  src <IP>:43082, session #59, session flags 0x001004ce
  HTTP msg state MSG_RPBEFORE(8), msg flags 0x00000000, tx flags 0xa8003000
  HTTP chunk len 0 bytes, HTTP body len 0 bytes
  buffer flags 0x80008002, out 0 bytes, total 24576 bytes
  pending 12288 bytes, wrapping at 16384, error at position 0:
 
  00000  /all\\/themes\\/dkg\\/bootstrap\\/dist\\/css\\/bootstrap.min.css":1,"s
  00064+ ites\\/all\\/themes\\/dkg\\/bootstrap\\/dist\\/css\\/bootstrap-theme.m
  00127+ in.css":1,"sites\\/all\\/themes\\/promato\\/smartmenus\\/addons\\/boot
  00191+ strap\\/jquery.smartmenus.bootstrap.css":1,"sites\\/all\\/themes\\/pro
  00257+ mato\\/css\\/nf-main.css":1,"sites\\/all\\/themes\\/promato\\/css\\/nf
  00320+ -webform.css":1,"sites\\/all\\/themes\\/promato\\/css\\/nf-news.css":1
  00385+ ,"sites\\/all\\/themes\\/promato\\/css\\/nf-icons.css":1,"sites\\/all
  00448+ \\/themes\\/promato\\/css\\/nf-gallery.css":1,"sites\\/all\\/themes\\/
  00511+ promato\\/font-awesome\\/css\\/font-awesome.min.css":1,"sites\\/all\\/
  00576+ themes\\/dkv\\/css\\/disable-responsive.css":1,"sites\\/all\\/themes\\
  00640+ /dkv\\/css\\/dkv.css":1,"sites\\/all\\/themes\\/dkg\\/css\\/override.c
  00703+ ss":1}},"fancybox":[],"overlay":{"paths":{"admin":"node\\/*\\/webform
  00770+ \\nnode\\/*\\/webform\\/*\\nnode\\/*\\/webform-results\\nnode\\/*\\/we
  00830+ bform-results\\/*\\nnode\\/*\\/submission\\/*\\nnode\\/*\\/edit\\nnode
  00891+ \\/*\\/delete\\nnode\\/*\\/revisions\\nnode\\/*\\/revisions\\/*\\/reve
  00951+ rt\\nnode\\/*\\/revisions\\/*\\/delete\\nnode\\/add\\nnode\\/add\\/*\\
  01010+ noverlay\\/dismiss-message\\nuser\\/*\\/shortcuts\\nadmin\\nadmin\\/*
  01072+ \\nbatch\\ntaxonomy\\/term\\/*\\/edit\\nnode\\/*\\/translate\\nuser\\/
  01132+ *\\/cancel\\nuser\\/*\\/edit\\nuser\\/*\\/edit\\/*\\ntaxonomy\\/*\\/tr
  01191+ anslate\\ntaxonomy\\/*\\/translate\\/*\\ndevel\\/*\\nnode\\/*\\/devel
  01251+ \\nnode\\/*\\/devel\\/*\\ncomment\\/*\\/devel\\ncomment\\/*\\/devel\\/
  01310+ *\\nuser\\/*\\/devel\\nuser\\/*\\/devel\\/*\\ntaxonomy\\/term\\/*\\/de
  01369+ vel\\ntaxonomy\\/term\\/*\\/devel\\/*\\nnode\\/*\\/revisions\\/view\\/
  01429+ *\\/*","non_admin":"admin\\/structure\\/block\\/demo\\/*\\nadmin\\/rep
  01492+ orts\\/status\\/php"},"pathPrefixes":["de","en"],"ajaxCallback":"overl
  01560+ ay-ajax"},"tableHeaderOffset":"Drupal.admin.height","admin_menu":{"des
  01630+ tination":"destination=suche\\/search\\/memo.html","hash":"b2f84c002aa
  01698+ 2c65b256f81e58c48466e","basePath":"\\/de\\/admin_menu","margin_top":1,
  01766+ "position_fixed":1,"toolbar":[]},"bootstrap":{"anchorsFix":"1","anchor
  01836+ sSmoothScrolling":"1","formHasError":1,"popoverEnabled":"1","popoverOp
  01906+ tions":{"animation":1,"html":0,"placement":"right","selector":"","trig
  01976+ ger":"click","triggerAutoclose":1,"title":"","content":"","delay":0,"c
  02046+ ontainer":"body"},"tooltipEnabled":"1","tooltipOptions":{"animation":1
  02116+ ,"html":0,"placement":"auto left","selector":"","trigger":"hover focus
  02186+ ","delay":0,"container":"body"}}});</script>\n
  02231  </head>\n
  02239  <body class="html not-front logged-in no-sidebars page-suche page-such
  02309+ e-search page-suche-search-memohtml i18n-de" >\n
  02356    <div id="skip-link">\n
  02379      <a href="#main-content" class="element-invisible element-focusable
  02449+ ">Skip to main content</a>\n
  02476    </div>\n
  02485      <div class="region region-page-top">\n
  02526      <div id="overlay-disable-message" class="clearfix"><h3 class="elem
  02596+ ent-invisible">Options for the administrative overlay</h3><a href="/de
  02666+ /user/1/edit?destination=suche/search/memo.html#edit-overlay-control" 
  02736+ id="overlay-profile-link" class="overlay-exclude element-invisible">If
  02806+  you have problems accessing administrative pages on this site, disabl
  02876+ e the overlay on your profile page.</a> <a href="/de/overlay/dismiss-m
  02946+ essage?destination=suche/search/memo.html&amp;token=jfmKm3UoGUx7x-ruMN
  03016+ JGetOoc5_T3hq7BnZhCERKsPw" id="overlay-dismiss-message" class="overlay
  03086+ -exclude element-invisible">Dismiss this message.</a></div>  </div>\n
  03154    <div class="container header-image">\n
  03193  \t      <!--<a class="logo navbar-btn pull-left" href="/de" title="Hom
  03262+ e">\n
  03266          <img src="https://<DOMAIN>/sites/all/themes/dkg/logo.png
  03336+ " alt="Home" />\n
  03352        </a>-->\n
  03366              <div class="site_name"><h1 style="color:#11574c;">DEUTSCHE
  03436+ S KRANKENHAUSVERZEICHNIS</h1></div>\n
  03472        <!--<div class="site_slogan"><span style="font-size:85%;">Servic
  03542+ e of the hospitals in berlin and the berlin hospital association in co
  03612+ nnection with the german hospital directory</span></div>-->\n
  03672        <div class="language-switch">\n
  03708          <a class="switch-ger" href="/de/suche" target="_self">\n
  03771                  <img height="15" src="/sites/all/themes/dkv/images/de.
  03841+ gif" alt="de"/>\n
  03857          </a>\n
  03870          <a class="switch-eng" href="/en/search" target="_self">\n
  03934                  <img height="15" src="/sites/all/themes/dkv/images/uk.
  04004+ gif" alt="en"/>\n
  04020          </a>\n
  04033  </div>\n
  04040    </div>\n
  04049  \n
  04050  <header id="navbar" role="banner" class="navbar container navbar-defau
  04120+ lt">\n
  04125    \n
  04128  <!--   <div class="container"> -->\n
  04163      <!--<div class="navbar-header">\n
  04199        \n
  04206  \n
  04207              <a class="name navbar-brand" href="/de" title="Home">Deuts
  04277+ ches Krankenhaus Verzeichnis</a>\n
  04310        -->\n
  04320  \n
  04321        <!-- .btn-navbar is used as the toggle for collapsed navbar cont
  04391+ ent -->\n
  04399        <!--<button type="button" class="navbar-toggle" data-toggle="col
  04469+ lapse" data-target=".navbar-collapse">\n
  04508          <span class="sr-only">Toggle navigation</span>\n
  04563          <span class="icon-bar"></span>\n
  04602          <span class="icon-bar"></span>\n
  04641          <span class="icon-bar"></span>\n
  04680        </button>-->\n
  04699   <!--   </div>-->\n
  04717  \n
  04718      <!--       <div class="navbar"> -->\n
  04758          <nav role="navigation">\n
  04790                        <ul class="menu nav navbar-nav"><li class="first
  04860+  expanded dropdown"><a title="" data-target="#" class="dropdown-toggle
  04930+  nolink">Suche</a><ul class="dropdown-menu"><li class="first leaf"><a 
  05000+ href="/de/suche/search/reset.html" title="">Neue Suche</a></li>\n
  05064  <li class="last leaf"><a href="/de/suche/dkv/search/continue.html" tit
  05134+ le="">Suche fortsetzen</a></li>\n
  05166  </ul></li>\n
  05177  <li class="expanded dropdown"><a title="" data-target="#" class="dropd
  05247+ own-toggle nolink">Hilfe zur Suche</a><ul class="dropdown-menu"><li cl
  05317+ ass="first leaf"><a href="/de/hilfe/allgemeines">Allgemeines</a></li>
  05386+ \n
  05387  <li class="leaf"><a href="/de/hilfe/regional">Regional</a></li>\n
  05451  <li class="leaf"><a href="/de/hilfe/krankheitsbilder-behandlungen">Kra
  05521+ nkheitsbilder / Behandlungen</a></li>\n
  05559  <li class="leaf"><a href="/de/hilfe/qualitaet">Qualit\xC3\xA4t</a></li
  05623+ >\n
  05625  <li class="leaf"><a href="/de/hilfe/struktur-leistungen">Struktur / Le
  05695+ istungen</a></li>\n
  05713  <li class="leaf"><a href="/de/hilfe/haeufige-krankheiten">H\xC3\xA4ufi
  05777+ ge Krankheiten</a></li>\n
  05801  <li class="leaf"><a href="/de/hilfe/suchergebnisse">Suchergebnisse</a>
  05871+ </li>\n
  05877  <li class="last leaf"><a href="/de/hilfe/merkliste">Merkliste</a></li>
  05947+ \n
  05948  </ul></li>\n
  05959  <li class="expanded dropdown"><a title="" data-target="#" class="dropd
  06029+ own-toggle nolink">Informationen</a><ul class="dropdown-menu"><li clas
  06099+ s="first last leaf"><a href="/de/ueber-das-dkv">\xC3\x9Cber das DKV</a
  06163+ ></li>\n
  06170  </ul></li>\n
  06181  <li class="expanded dropdown"><a title="" data-target="#" class="dropd
  06251+ own-toggle nolink">Dienste</a><ul class="dropdown-menu"><li class="fir
  06321+ st leaf"><a href="/de/stellenboerse">Stellenb\xC3\xB6rse</a></li>\n
  06381  <li class="last collapsed"><a href="/de/geraeteboerse" title="">Ger
  06448+ \xC3\xA4teb\xC3\xB6rse</a></li>\n
  06468  </ul></li>\n
  06479  <li class="last leaf"><a href="http://www.dkgev.de/" title="Zur DKG">Z
  06549+ ur DKG</a></li>\n
  06565  </ul>                                      </nav>\n
  06615  <!--       </div> -->\n
  06637      <!--   </div> -->\n
  06659  </header>\n
  06669  \n
  06670  <div class="main-container container">\n
  06709  \n
  06710    <header role="banner" id="page-header">\n
  06752      \n
  06757        </header> <!-- /#page-header -->\n
  06796  \n
  06797    <div class="row">\n
  06817  \n
  06818      \n
  06823      <section class="col-sm-12">\n
  06855                    <a id="main-content"></a>\n
  06899                                                                  <div c
  06969+ lass="region region-content">\n
  06999      <section id="block-system-main" class="block block-system clearfix
  07069+ ">\n
  07072  \n
  07073        \n
  07080    <div class="view view-dkv-proxy view-id-dkv_proxy view-display-id-pa
  07150+ ge view-dom-id-2843e18d24129a508bebec5dc8693200">\n
  07200              <div class="view-header">\n
  07238        <div id="dkv_2"><div id="dkv_1"><div id="dkv" style="position: r
  07308+ elative;">\n
  07319  </div></div></div>    </div>\n
  07348    \n
  07351    \n
  07354    \n
  07357    \n
  07360    \n
  07363    \n
  07366    \n
  07369    \n
  07372    \n
  07375  </div>\n
  07382  </section>\n
  07393  <section id="block-views-news-footer-block" class="block block-views c
  07463+ learfix">\n
  07473  \n
  07474          <h2 class="block-title">Nachrichten</h2>\n
  07523      \n
  07528    <div class="view view-news-footer view-id-news_footer view-display-i
  07598+ d-block view-dom-id-fd719de56456119ccdd1e36200102c69">\n
  07653          \n
  07662    \n
  07665    \n
  07668        <div class="view-content">\n
  07701        \t<div class="row">\n
  07726  \t\t\t<div class="col-xs-4">\n
  07752  \t\t\t\n
  07756  \t\t\t<!-- Bild -->\n
  07773  \t\t\t\n
  07777  \t\t\t\t\t\t\t\t<!-- Body -->\n
  07799  \t\t\t\t<div>\n
  07809  \t\t\t\t\t<strong>Tagesaktuelle Daten und optimierte Suchoptionen</str
  07874+ ong>\n
  07879  \t\t\t\t\t<br/>\n
  07890  \t\t\t\t\t<span>- <i class="fa fa-calendar"></i> 12.05.2016 -</span>\n
  07954  \t\t\t\t\t<br/>\n
  07965  \t\t\t\t\t<p><strong>Deutsches Krankenhausverzeichnis aktualisiert</st
  08030+ rong></p>\n
  08040  <p><strong>Berlin, 12. Mai\xC2\xA0 2016</strong> \xE2\x80\x93\xC2\xA0 
  08089+ Das Deutsche Krankenhausverzeichnis (DKV) wurde aktualisiert und nutze
  08159+ rfreundlicher gestaltet. Es bietet Patienten und einweisenden \xC3\x84
  08223+ rzten umfassende Informationen \xC3\xBCber die Leistungen der Krankenh
  08287+ \xC3\xA4user in den Regionen und deutschlandweit. Fast 4 Millionen Mal
  08351+  wurde das Verzeichnis j\xC3\xA4hrlich aufgerufen.</p>\t\t\t\t\t<span>
  08410+ \n
  08411  \t\t\t\t\t\t...\n
  08421  \t\t\t\t\t</span>\n
  08434  \t\t\t\t</div>\n
  08445  \t\t\t\t\n
  08450  \t\t\t\n
  08454  \t\t\t\n
  08458  \t\t\t\t<a href="/de/news/1463047312/tagesaktuelle-daten-und-optimiert
  08524+ e-suchoptionen">mehr ...</a>\n
  08553  \t\t\t\n
  08557  \t\t</div>\n
  08566  \t\t\t<div class="col-xs-4">\n
  08592  \t\t\t\n
  08596  \t\t\t<!-- Bild -->\n
  08613  \t\t\t\n
  08617  \t\t\t\t\t\t\t\t<!-- Body -->\n
  08639  \t\t\t\t<div>\n
  08649  \t\t\t\t\t<strong>QB-IPQ-2013 zur Erstellung des Qualit\xC3\xA4tsberic
  08708+ htes 2013 seit 18.08.2014 verf\xC3\xBCgbar</strong>\n
  08754  \t\t\t\t\t<br/>\n
  08765  \t\t\t\t\t<span>- <i class="fa fa-calendar"></i> 12.08.2014 -</span>\n
  08829  \t\t\t\t\t<br/>\n
  08840  \t\t\t\t\t<p>Die Erfassungssoftware Internet Portal Qualit\xC3\xA4tsbe
  08899+ richt (IPQ) f\xC3\xBCr die Erstellung des Qualit\xC3\xA4tsberichtes 20
  08957+ 13 steht seit 18.08.2014 zur Verf\xC3\xBCgung und kann seit diesem Zei
  09021+ tpunkt von den Krankenh\xC3\xA4usern kostenfrei genutzt werden.</p>\n
  09083  <p>Bei der Erstellung des Qualit\xC3\xA4tsberichtes 2013 steht eine ko
  09147+ mfortable \xC3\x9Cbernahmem\xC3\xB6glichkeit der DKV-Daten oder der Da
  09205+ ten aus dem Qualit\xC3\xA4tsbericht 2012 zur Verf\xC3\xBCgung. Diese D
  09263+ aten k\xC3\xB6nnen als Grundlage f\xC3\xBCr die weitere Bearbeitung un
  09321+ d Erstellung des Qualit\xC3\xA4tsberichtes 2013 genutzt werden.</p>\n
  09383  <p>Krankenh\xC3\xA4user, die bereits als Nutzer von IPQ registriert si
  09447+ nd, k\xC3\xB6nnen sich mit ihren Zugangsdaten am System anmelden.</p>
  09510+ \n
  09511  <p>Krankenh\xC3\xA4user, die noch nicht als Nutzer von IPQ registriert
  09575+  und an einem Zugang interessiert sind, wenden sich bitte an ihre Land
  09645+ eskrankenhausgesellschaft.</p>\n
  09676  \t\t\t\t\t<span>\n
  09688  \t\t\t\t\t\t...\n
  09698  \t\t\t\t\t</span>\n
  09711  \t\t\t\t</div>\n
  09722  \t\t\t\t\n
  09727  \t\t\t\n
  09731  \t\t\t\n
  09735  \t\t\t\t<a href="/de/news/1431697417/qb-ipq-2013-zur-erstellung-des-qu
  09801+ alit%C3%A4tsberichtes-2013-seit-18082014-verf%C3%BCgbar">mehr ...</a>
  09870+ \n
  09871  \t\t\t\n
  09875  \t\t</div>\n
  09884  \t\t\t<div class="col-xs-4">\n
  09910  \t\t\t\n
  09914  \t\t\t<!-- Bild -->\n
  09931  \t\t\t\n
  09935  \t\t\t\t\t\t\t\t<!-- Body -->\n
  09957  \t\t\t\t<div>\n
  09967  \t\t\t\t\t<strong>Datenbasis des Deutschen Krankenhaus Verzeichnisses 
  10032+ (DKV) aktualisiert</strong>\n
  10060  \t\t\t\t\t<br/>\n
  10071  \t\t\t\t\t<span>- <i class="fa fa-calendar"></i> 17.04.2014 -</span>\n
  10135  \t\t\t\t\t<br/>\n
  10146  \t\t\t\t\t<p>Seit dem 16. April 2014 stehen im DKV, dem Krankenhausver
  10211+ zeichnis der Deutschen Krankenhausgesellschaft und der 16 Landeskranke
  10281+ nhausgesellschaften, aktualisierte Informationen aus den neuesten Qual
  10351+ it\xC3\xA4tsberichten der Krankenh\xC3\xA4user (Berichtsjahr 2012) in 
  10409+ allgemeinverst\xC3\xA4ndlicher Sprache und mit umfangreichen Suchfunkt
  10473+ ionen zur Verf\xC3\xBCgung. . Grunds\xC3\xA4tzlich basiert das DKV 
  10528+ \xE2\x80\x93 wie alle anderen Krankenhaus-Portale \xE2\x80\x93 auf den
  10580+  "Strukturierten Qualit\xC3\xA4tsberichten", die die Krankenh\xC3\xA4u
  10638+ ser j\xC3\xA4hrlich erstellen m\xC3\xBCssen. Im Gegensatz zu anderen P
  10696+ ortal bietet das DKV allerdings die M\xC3\xB6glichkeit f\xC3\xBCr die 
  10754+ Krankenh\xC3\xA4user, ihre Daten jederzeit zu aktualisieren. Der Nutze
  10818+ r profitiert von tagesaktuellen Informationen.</p>\n
  10869  <p><strong>Suchm\xC3\xB6glichkeiten</strong></p>\n
  10912  <p>Der Einstieg in die Suche wurde mit der Aktualisierung des DKV noch
  10982+  \xC3\xBCbersichtlicher gestaltet. Bei der Suche nach einem geeigneten
  11046+  Krankenhaus erfolgt eine Unterst\xC3\xBCtzung durch die strukturierte
  11110+  Vorgabe von sechs verschiedenen Suchm\xC3\xB6glichkeiten:</p>\t\t\t\t
  11170+ \t<span>\n
  11178  \t\t\t\t\t\t...\n
  11188  \t\t\t\t\t</span>\n
  11201  \t\t\t\t</div>\n
  11212  \t\t\t\t\n
  11217  \t\t\t\n
  11221  \t\t\t\n
  11225  \t\t\t\t<a href="/de/news/1397742077/datenbasis-des-deutschen-krankenh
  11291+ aus-verzeichnisses-dkv-aktualisiert">mehr ...</a>\n
  11341  \t\t\t\n
  11345  \t\t</div>\n
  11354  \t\t\t<div class="col-xs-4">\n
  11380  \t\t\t\n
  11384  \t\t\t<!-- Bild -->\n
  11401  \t\t\t\n
  11405  \t\t\t\t\t\t\t\t<!-- Body -->\n
  11427  \t\t\t\t<div>\n
  11437  \t\t\t\t\t<strong>Ausnahmeregelung f\xC3\xBCr die nachtr\xC3\xA4gliche
  11490+  Anmeldung (Nachregistrierung) und Nachlieferung der Qualit\xC3\xA4tsb
  11554+ erichte 2012</strong>\n
  11576  \t\t\t\t\t<br/>\n
  11587  \t\t\t\t\t<span>- <i class="fa fa-calendar"></i> 17.04.2014 -</span>\n
  11651  \t\t\t\t\t<br/>\n
  11662  \t\t\t\t\t<p>Der G-BA hat in seiner Sitzung am 20.03.2014 eine einmali
  11727+ ge Ausnahmeregelung f\xC3\xBCr die Anmeldung (Nachregistrierung) und N
  11791+ achlieferung der Qualit\xC3\xA4tsberichte des Berichtsjahres 2012 besc
  11855+ hlossen. Die Ausnahmeregelung erfolgt auf der Grundlage von \xC2\xA7 6
  11919+  Abs. 3a der Qb-R und stellt kein Pr\xC3\xA4judiz f\xC3\xBCr die folge
  11977+ nden Berichtsjahre dar.</p>\n
  12005  <p>Alle berichtspflichten Krankenh\xC3\xA4user k\xC3\xB6nnen durch die
  12063+ sen Beschluss erstmalig ihre Qualit\xC3\xA4tsberichte 2012 liefern ode
  12127+ r korrigierte Qualit\xC3\xA4tsberichte nachliefern.</p>\n
  12177  <p>Anmeldezeitraum: 19.05.2014 bis 26.05.2014<br />\n
  12229  Abgabezeitraum: 23.06.2014 bis 04.07.2014</p>\n
  12275  <p>Details zu

It looks to me like there is no HTTP header at all and the response begins somewhere in the middle of the actual content.

Is there any chance you could downgrade your backend traffic to clear text and capture the traffic (tcpdump) between haproxy and the backend. Or, configure a non-FS ciphers and decrypt the traffic with the private key?

1 Like

Here is the file: https://www.file-upload.net/download-13319679/haproxy.zip.html. Password is dellwing.

I already took a look with wireshark and spotted an red backgrounded package. But I have not really a clue of that.

The command I ran was sudo tcpdump -i ens18 -s 0 -q '(src 172.16.3.45) or (dst 172.16.3.45)' -w output.dump

Here is a HAR of the expected request result: https://www.file-upload.net/download-13319704/nohaproxy.har.html

After requesting /de/suche/search/reset.html, the backend servers responds with a 302 Moved Temporarily and Content-Length: 0. That’s perfectly fine, except that the server then begins to actual send HTTP payload (which is not allowed, since the server just declared in the headers that there will be no payload).

You can actually reproduce this issue with your live site (which does not use haproxy yet, I assume) and see the bogus data and subsequent error message in curl:

$ curl https://dkg.promato.de/de/suche/search/reset.html -v
*   Trying 213.135.13.68...
* Connected to dkg.promato.de (213.135.13.68) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: C:\Program Files (x86)\unixutils\ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* NPN, negotiated HTTP1.1
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Unknown (67):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=dkg.promato.de
*  start date: Jul 31 09:30:36 2018 GMT
*  expire date: Oct 29 09:30:36 2018 GMT
*  subjectAltName: host "dkg.promato.de" matched cert's "dkg.promato.de"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET /de/suche/search/reset.html HTTP/1.1
> Host: dkg.promato.de
> User-Agent: curl/7.48.0
> Accept: */*
>
< HTTP/1.1 302 Moved Temporarily
< Server: nginx
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: keep-alive
< Expires: Sun, 19 Nov 1978 05:00:00 GMT
< Cache-Control: no-cache, must-revalidate
< Set-Cookie: JSESSIONID=2C82032AB0C9C0978A983840E827747A; path=/de/suche; domain=dkg.promato.de
< location: https://dkg.promato.de/de/suche/Regional.html;jsessionid=2C82032AB0C9C0978A983840E827747A
< Set-Cookie: JSESSIONID=F7CFD893AAA84811346590D8971067A4; path=/de/suche; domain=dkg.promato.de
< date: Tue, 18 Sep 2018 14:33:15 GMT
< Content-Language: de
< X-Frame-Options: SAMEORIGIN
< X-Generator: Drupal 7 (http://drupal.org)
<
* Excess found in a non pipelined read: excess = 15746 url = /de/suche/search/reset.html (zero-length body)
* Connection #0 to host dkg.promato.de left intact
$

The Excess found in a non pipelined read: excess = 15746 url = /de/suche/search/reset.html (zero-length body)` at the end there, after a supposedly zero-length body is what explains the issue. It’s a miracle that this does not cause browsers to throw in the towel, but I guess they just ignore the bogus response payload and follow the redirect.

What happens in haproxy is that after parsing the 302 response header there is no reason to parse anything else (as per the Content-Length: 0 header the response does not contain payload), so after a while haproxy issues the next response, which happens to happen while the server is still sending response payload data. This HTTP session is completely messed up at this point, so that haproxy tries to begin parsing a HTTP header when actually the buffer is in the middle of a response:

1 Like

Any idea what happens where? I guess the problem is, that the main container of the site is loaded via Drupal modul from a backend Tomcat server. But I currently have no clue why he sends data after 302 status code…

HTTP payload after 302 is actually fine, as long as the Content-Length header does not mislead. It’s really the wrong Content-Length that is causing the issue here.

Not sure what the actual issue here is.

1 Like

Ok, thanks for your help. I “fixed” it (well not really, but its good enough for the moment).

I forced nginx into chunked transfer encoding by setting more_clear_headers -s 302 'Content-Length'; (requires nginx-extra or self compiled nginx) and it now works.

1 Like