We upgraded from an older version of haproxy1, to haproxy 2.0
We’'re using it as a “reverse proxy” to serve out content.
but now some of our content is being BLOCKED from being downloaded by customers.
Logs are giving a “PH” errork, which says basically that haproxy is choosing to block the context from passing through.
No further details are given.
How can we turn off this PH blocking?
Please troubleshoot the actual reason for this block.
You can use the show errors command on the admin socket to show the exact reasons for this error.
Once you know the actual reason for this, you can try option accept-invalid-http-response to ignore some of those issues.
Hmm.
well thanks for the pointer on that show errors thing.
It said:
"backend xxx_backend (#7): invalid response
frontend xxxx (#4), server xxx (#1), event #0, src x.x.x.x:35312
buffer starts at 0 (including 0 out), 64 free,
len 16320, wraps at 16336, error at position 176
H1 connection flags 0x00000020, H1 stream flags 0x00004044
H1 msg state MSG_HDR_L2_LWS(24), H1 msg flags 0x00001414
H1 chunk len 0 bytes, H1 body len 0 bytes :
… very long message details follows…
"
The position 176 seems to point to the middle of the following header:
“Content-Description: File Transfer\r\n”
doesnt seem to be any problem with that?
I tried adding that “option accept-invalid-http-response” you suggested, to the relevant back end, and restarted haproxy.
Still sending back errror 502 to the client.
So I’m back to asking, “how can I just disable this PH thing completely?”
We’re having to revert back to running haproxy v1.5.
That doesnt cause problems for us.
Post the complete output please.
I think you’re missing the point here.
im not interested in finding out the problem with our back end.
I dont have CONTROL over our back end.
I have control over haproxy.
so I want to get haproxy working again.
so how do I turn this new validation checking in haproxy v2, off???
You can probably revert to the old behavior in 2.0 by disabling htx mode (no option http-use-htx) in the default section.
Starting with 2.1, this is not a possibility anymore.
Please consider this a feature request to add a config option to disable PH checking in v2.1 then 
That’s not possible. The old H1 logic has been completely removed, there is no going back.
What is needed to understand this case better, that includes understanding if haproxy is legitimately dropping the request or whether this is a haproxy bug, is to analyze the root cause of the drop.
I guess it depends on your definition of “legitimate”.
I can concede that our backend headers violate some kind of spec, I’m sure.
but the point is, they WORK, for the range of browsers that our clients use.
I dont think haproxy should be in the business of protecting people that dont WANT to be protected.
We just want a passthrough, albeit with some smart redirecting.
We should be able to tell haproxy, “shut up and pass everything through”.
What haproxy is currently doing, is kind of equivalent to some browser saying, “wait… that HTML code violates HTML spec. Screw you, i refuse to render the entire page”.
Do you want your browser to behave like that? I dont.
I dont want our proxy behaving like that either 
ps:
its difficult to parse, but the header line it appears to be complaning about, is a header line that is duplicated.
there is a messy “Content-Description” header, and then a cleaner, second one.
haproxy is barfing at the “:” of the second one.
WIthout saying that the problem is a duplicate header.
So that implies two problems.
My point is: we can only improve if we understand the root cause.
You are saying that you don’t want haproxy to check any of those things - then you cannot use haproxy in HTTP mode, because haproxy will always check at least some basic HTTP syntax, and with HTX mode (default in 2.0 and the only http mode in 2.1) the requirements are even higher (because the entire transaction is translated internally to a version agnostic HTTP mode we call HTX).
Some checks can be disabled with accept-invalid-http-response, others cannot. Sometimes there are bugs in the parser that need fixing. Sometimes it simply cannot be done because the headers are so broken. Sometimes the checks are oversensitive, sometimes the opposite is the case.
If the general idea that haproxy performs some basic syntax checking, which cannot be fully disabled is not acceptable for your use-case, then you need to use TCP mode, not HTTP mode, and haproxy will become a pipe passing only TCP payload back and forth.
So let me know if you want me to look into whether haproxy rejects duplicate header names, the reasons for it and if there is room for improvement for this case, or whether you want to transition to a configuration that is passing traffic transparently between the frontend and the backend, without HTTP parsing.
I disagree, every browser rejects tons of invalid stuff for security and performance reasons. We would be in a world of hurt if browsers would still be accepting everything like 15, 20 years ago. Drive-by adware et all.
Our particular uses require examining the URL and doing redirects based on that. (and in some cases, URL rewriting).
So a pure TCP passthrough would not work for us.
Then troubleshooting the root cause is needed.
For the record, it’s not about duplicate header names, in my tests haproxy passes them without any issues (not even accept-invalid-http-response is set):
$ curl dev.lan.ltri.eu/duplicateheadername.php -v
* Trying 10.0.0.33...
* TCP_NODELAY set
* Connected to dev.lan.ltri.eu (10.0.0.33) port 80 (#0)
> GET /duplicateheadername.php HTTP/1.1
> Host: dev.lan.ltri.eu
> User-Agent: curl/7.55.1
> Accept: */*
>
< HTTP/1.1 200 OK
< server: nginx/1.14.0 (Ubuntu)
< date: Mon, 23 Dec 2019 20:59:16 GMT
< content-type: text/html; charset=UTF-8
< transfer-encoding: chunked
< content-description: asdsdadss
< content-description: File Transfer
< content-sadasd: ASDDS
<
asdasd
as
da
sd
asd* Connection #0 to host dev.lan.ltri.eu left intact
$
Can you:
haproxy -vv
Hmmm.
well, I can confirm that disabling htx mode makes things work again.
I did include the “show errors” output, in my second message in this thread.
I just snipped the actual headers.
I can potentially provide the headers logged there, after doing some sanitization, if desired.
Check if the previous and the next line contain any sensitive data, and if not, provide those 3 lines, however - very important - including the hex dump of said lines. The textual representation of the header line is actually not very helpful in this particular use-case, because we specifically need to know each and every single binary value.
Less data we have to look at, the less likely we are going to have find the root cause here.
Remember to provide the ouput of haproxy -vv (no sensitive data there) and an overview of the configurations (there is certainly sensitive data in there, but used features as well as general setting will certainly be helpful).
not sure where “the hex dump” is supposed to come from.
the socket thingie didnt provide one?
heres the headers, minus one small bit.
00000 HTTP/1.1 200 OK\r\n
00017 Server: Apache-Coyote/1.1\r\n
00044 Content-Description:\xC0[\x95\x02\x00\x00\x00\x00\xC0\xF4\x8B\x00\x00
00077+ \x00\x00\x00\xD0[\x95\x02\x00\x00\x00\x00 ~\x8B\x00\x00\x00\x00\x00
00096+ \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xFF\xFF\xFF\xFFHTTP/1
00118+ .1 200 OK\r\n
00129 Server: Apache-Coyote/1.1\r\n
00156 Content-Description: File Transfer\r\n
00192 Content-Disposition: attachment;filename=xxxxxx
00261+ \r\n
00263 Content-Length:: 137358\r\n
00288 Content-Transfer-Encoding: binary \r\n
00324 Expires: 0\r\n
00336 Content-Type: application/octet-stream\r\n
00376 Transfer-Encoding: chunked\r\n
00404 Date: Mon, 23 Dec 2019 17:59:55 GMT\r\n
00441 Connection: close\r\n
$ /usr/local/sbin/haproxy -vv
HA-Proxy version 2.0.8 2019/10/23 - https://haproxy.org/
Build options :
TARGET = linux-glibc
CPU = generic
CC = gcc
CFLAGS = -O2 -g -fno-strict-aliasing -Wdeclaration-after-statement -fwrapv -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-old-style-declaration -Wno-ignored-qualifiers -Wno-clobbered -Wno-missing-field-initializers -Wtype-limits
OPTIONS = USE_OPENSSL=1 USE_ZLIB=1
Feature list : +EPOLL -KQUEUE -MY_EPOLL -MY_SPLICE +NETFILTER -PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -REGPARM -STATIC_PCRE -STATIC_PCRE2 +TPROXY +LINUX_TPROXY +LINUX_SPLICE +LIBCRYPT +CRYPT_H -VSYSCALL +GETADDRINFO +OPENSSL -LUA +FUTEX +ACCEPT4 -MY_ACCEPT4 +ZLIB -SLZ +CPU_AFFINITY +TFO +NS +DL +RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER +PRCTL +THREAD_DUMP -EVPORTS
Default settings :
bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with multi-threading support (MAX_THREADS=64, default=2).
Built with OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
Running on OpenSSL version : OpenSSL 1.0.2k-fips 26 Jan 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with network namespace support.
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND
Built with zlib version : 1.2.7
Running on zlib version : 1.2.7
Compression algorithms supported : identity(“identity”), deflate(“deflate”), raw-deflate(“deflate”), gzip(“gzip”)
Built without PCRE or PCRE2 support (using libc’s regex instead)
Encrypted password support via crypt(3): yes
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available multiplexer protocols :
(protocols marked as cannot be specified using ‘proto’ keyword)
h2 : mode=HTX side=FE|BE mux=H2
h2 : mode=HTTP side=FE mux=H2
: mode=HTX side=FE|BE mux=H1
: mode=TCP|HTTP side=FE|BE mux=PASS
Available services : none
Available filters :
[SPOE] spoe
[COMP] compression
[CACHE] cache
[TRACE] trace
Yeah I did confuse the part about the hex dump. What is there is the offset, which is very helpful.
Can you tell which part of the dump you modified exactly, so I can replay this response as close to your real scenario as possible.
I just xx’d out the filename
I can see the issue while replaying the same response, but the error is at position 288, where Content-Transfer-Encoding begins and it seems to be triggered by the double colon after the previous Content-Length header, which appears to confuse haproxy.
When I replace the double colon with a single colon, the request actually get’s trough in my tests.
This doesn’t match with the error you are saying, because it fails earlier.
Can you double check that the error position is really from same exact request that the output is from and confirm it points to the beginning of the second Content-Description header?
oh! haha. didnt notice the double colon.
yes, its really the same error.
I saved the entire output to a file, after a fresh restart of things. there was no other error to get confused with.
possibly the position counter got screwd up because of the double content-description header, but the actual “fatal” error is with the content length header.