We are getting 502 errors from HAproxy sayingthe server returned an invalid or incomplete response. In the haproxy logs, it has code PH which when looked into, it said The proxy blocked the server’s response, because it was invalid,
incomplete, dangerous (cache control), or matched a security filter.
In any case, an HTTP 502 error is sent to the client. One possible
cause for this error is an invalid syntax in an HTTP header name
containing unauthorized characters.
We also noticed that the request didn’t get logged at the server end. Maybe we didn’t have enough logging implemented but we don’t know the issue. How to fix it? This has not happened for any of our other services onboarded to same Haproxy cluster. Please someone can guide on this.
I would suggest to enable the unix admin socket and use show errors on it, this will show you more details about the last response that has been blocked, see:
Where do we print show errors? on the host where we are deploying, will the logs be printed where other haproxy logs are printed or if not, at what path should I check?
The error is that this is random binary data, it does not even resemble a HTTP response.
You will have to tcpdump the traffic and analyze what happens here. Either the backend server randomly response with binary instead of HTTP or perhaps those are pipelined request with broken transaction delimitation.
We still could not figure out the way. Do we have any other similar example for this? Interesting part is we have other services of the same module which completely works fine but not this.