Disable PH checks

Help!
21

Well that’s still different behavior we are seeing, for you pointing to position 176, for me 288 - for the same response.

[24/Dec/2019:00:48:07.992] backend nc (#4): invalid response
  frontend myfrontend (#2), server nc (#1), event #0, src 10.0.0.4:52335
  buffer starts at 0 (including 0 out), 15879 free,
  len 505, wraps at 16336, error at position 288
  H1 connection flags 0x00000000, H1 stream flags 0x0000401c
  H1 msg state MSG_HDR_L2_LWS(24), H1 msg flags 0x00001414
  H1 chunk len 0 bytes, H1 body len 0 bytes :

  00000  HTTP/1.1 200 OK\r\n
  00017  Server: Apache-Coyote/1.1\r\n
  00044  Content-Description:\xC0[\x95\x02\x00\x00\x00\x00\xC0\xF4\x8B\x00\x00
  00077+ \x00\x00\x00\xD0[\x95\x02\x00\x00\x00\x00 ~\x8B\x00\x00\x00\x00\x00
  00096+ \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xFF\xFF\xFF\xFFHTTP/1
  00118+ .1 200 OK\r\n
  00129  Server: Apache-Coyote/1.1\r\n
  00156  Content-Description: File Transfer\r\n
  00192  Content-Disposition: attachment;filename=xxxxxxxxxxxxxxxxxxxxxxxxxxxx
  00261+ \r\n
  00263  Content-Length:: 137358\r\n
  00288  Content-Transfer-Encoding: binary \r\n
  00324  Expires: 0\r\n
  00336  Content-Type: application/octet-stream\r\n
  00376  Transfer-Encoding: chunked\r\n
  00404  Date: Mon, 23 Dec 2019 17:59:55 GMT\r\n
  00441  Connection: close\r\n
  00460  \r\n
  00462  4\r\n
  00465  Wiki\r\n
  00471  5\r\n
  00474  pedia\r\n
  00481  E\r\n
  00484   in\r\n
  00489  \r\n
  00491  chunks.\r\n
  00500  0\r\n
  00503  \r\n

I’m not sure we are looking at the same issue.

22

@willy can you take a look at this. In HTX mode we are generally stricter and the HTTP response has numerous issues, but can you tell:

  • why replaying the same response we get different error positions (maybe timing related)
  • it is unclear what error haproxy is seeing at position 176 in the OP’s output
  • it is unclear why the error position for my output points to 288 when really the double colon of the previous content-length header triggered the error

The OP’s error:

"backend xxx_backend (#7): invalid response
frontend xxxx (#4), server xxx (#1), event #0, src x.x.x.x:35312
buffer starts at 0 (including 0 out), 64 free,
len 16320, wraps at 16336, error at position 176
H1 connection flags 0x00000020, H1 stream flags 0x00004044
H1 msg state MSG_HDR_L2_LWS(24), H1 msg flags 0x00001414
H1 chunk len 0 bytes, H1 body len 0 bytes :

00000 HTTP/1.1 200 OK\r\n
00017 Server: Apache-Coyote/1.1\r\n
00044 Content-Description:\xC0[\x95\x02\x00\x00\x00\x00\xC0\xF4\x8B\x00\x00
00077+ \x00\x00\x00\xD0[\x95\x02\x00\x00\x00\x00 ~\x8B\x00\x00\x00\x00\x00
00096+ \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xFF\xFF\xFF\xFFHTTP/1
00118+ .1 200 OK\r\n
00129 Server: Apache-Coyote/1.1\r\n
00156 Content-Description: File Transfer\r\n
00192 Content-Disposition: attachment;filename=xxxxxx
00261+ \r\n
00263 Content-Length:: 137358\r\n
00288 Content-Transfer-Encoding: binary \r\n
00324 Expires: 0\r\n
00336 Content-Type: application/octet-stream\r\n
00376 Transfer-Encoding: chunked\r\n
00404 Date: Mon, 23 Dec 2019 17:59:55 GMT\r\n
00441 Connection: close\r\n

Note: the filename in the Content-Disposition header at offset 192 has been shortened for sensitivity.

My error (replaying the same response with netcat):

lukas@dev:~$ echo "show errors" | socat stdio /tmp/haproxy
Total events captured on [25/Dec/2019:20:19:45.737] : 1

[25/Dec/2019:20:19:43.901] backend nc (#3): invalid response
  frontend myfrontend (#2), server nc (#1), event #0, src 10.0.0.4:51124
  buffer starts at 0 (including 0 out), 15924 free,
  len 460, wraps at 16336, error at position 288
  H1 connection flags 0x00000000, H1 stream flags 0x0000401c
  H1 msg state MSG_HDR_L2_LWS(24), H1 msg flags 0x00001414
  H1 chunk len 0 bytes, H1 body len 0 bytes :

  00000  HTTP/1.1 200 OK\r\n
  00017  Server: Apache-Coyote/1.1\r\n
  00044  Content-Description:\xC0[\x95\x02\x00\x00\x00\x00\xC0\xF4\x8B\x00\x00
  00077+ \x00\x00\x00\xD0[\x95\x02\x00\x00\x00\x00 ~\x8B\x00\x00\x00\x00\x00
  00096+ \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xFF\xFF\xFF\xFFHTTP/1
  00118+ .1 200 OK\r\n
  00129  Server: Apache-Coyote/1.1\r\n
  00156  Content-Description: File Transfer\r\n
  00192  Content-Disposition: attachment;filename=xxxxxxxxxxxxxxxxxxxxxxxxxxxx
  00261+ \r\n
  00263  Content-Length:: 137358\r\n
  00288  Content-Transfer-Encoding: binary \r\n
  00324  Expires: 0\r\n
  00336  Content-Type: application/octet-stream\r\n
  00376  Transfer-Encoding: chunked\r\n
  00404  Date: Mon, 23 Dec 2019 17:59:55 GMT\r\n
  00441  Connection: close\r\n

lukas@dev:~$

Replaying response on port 8080:

echo -ne \
'HTTP/1.1 200 OK\r\nServer: Apache-Coyote/1.1\r\nContent-Description:\xC0[\x95\x02\x00\x00\x00\x00\xC0\xF4\x8B\x00\x00\x00\x00\x00\xD0[\x95\x02\x00\x00\x00\x00 ~\x8B\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xFF\xFF\xFF\xFFHTTP/1.1 200 OK\r\nServer: Apache-Coyote/1.1\r\nContent-Description: File Transfer\r\nContent-Disposition: attachment;filename=xxxxxxxxxxxxxxxxxxxxxxxxxxxx\r\nContent-Length:: 137358\r\nContent-Transfer-Encoding: binary \r\nExpires: 0\r\nContent-Type: application/octet-stream\r\nTransfer-Encoding: chunked\r\nDate: Mon, 23 Dec 2019 17:59:55 GMT\r\nConnection: close\r\n\r\n' \
> pbrownBogusOriginalResponse.http
while true; do cat pbrownBogusOriginalResponse.http | nc -l 8080; done

Config:

global
 log 10.0.0.4 syslog debug
 stats socket /tmp/haproxy mode 666 level admin

defaults
 #option accept-invalid-http-response
 option http-use-htx
 mode http
 timeout connect 3s
 timeout http-request 20s
 timeout http-keep-alive 30s
 timeout client 10s
 timeout server 70s

frontend myfrontend
 bind :80
 option httplog
 log global
 use_backend nc

backend nc
 server nc 127.0.0.1:8080
23

Thanks Lukas. I’ll have to see with @capflam why the capture offsets are wrong.

What I’m seeing that’s puzzling me in this trace is that a header field contains NUL characters, which are strictly forbiddeen by the spec and do not pass through most HTTP agents since they are difficult to deal with (i.e. indexing/regex/replacing/etc) and are essentially used for attacks in order to hide malicious contents from analysis.

I’m not that much surprized it blocks on HTX, especially since we made sure to block the potential attack reported by Tim consisting in using NUL chars in headers between H2 and H1. However I think that if technically possible and if this doesn’t re-introduce a security problem we should do our best to let them pass through when the option is set to accept invalid headers. In theory HTX can be binary-transparent.

However @pbrown you must really be aware that such an application is living its last moments if it’s not fixed quickly, because it will become increasingly difficult to find HTTP agents compatible with such dangerous violations and sooner or later this one will be abused to attack a browser and this will suddently stop working in the middle of a maintenance version without prior notification!

24

Thank you for that feedback

25

@lukastribus, @willy, in fact there are 2 bugs here. The first one about the error position. The position reported is the relative offset of the next unparsed byte in the message when the error occurs. Most of time it is accurate, because it is a syntax error. But some tests are performed on headers value depending on their name (content-length, transfer-encoding, host). When an error occurs at this step, the parser is already on the next header line because the current header was already fully parsed. To have the right position, the parser state must be updated to point on the invalid value instead. In @pbrown example, the problem is about the double colon of the content-length header. The parser tries to convert the string : 137358 in integer and it obviously fails.

The second bug is about the error capture. In fact the first 112 bytes in @pbrown capture is a junk part. It is not part of the message. The function responsible to capture the error is buggy. The bug happens when the buffer is full.

I fixed the both bugs. I’ll push them very soon. Thanks guys !

1 Like
26

Pointing to the second colon now exactly, great:

lukas@dev:~$ echo "show errors" | socat stdio /tmp/haproxy
Total events captured on [07/Jan/2020:21:20:16.588] : 1

[07/Jan/2020:21:20:12.290] backend nc (#3): invalid response
  frontend myfrontend (#2), server nc (#1), event #0, src 10.0.0.4:50300
  buffer starts at 0 (including 0 out), 15922 free,
  len 462, wraps at 16336, error at position 278
  H1 connection flags 0x00000000, H1 stream flags 0x00004014
  H1 msg state MSG_HDR_L2_LWS(24), H1 msg flags 0x00001414
  H1 chunk len 0 bytes, H1 body len 0 bytes :

  00000  HTTP/1.1 200 OK\r\n
  00017  Server: Apache-Coyote/1.1\r\n
  00044  Content-Description:\xC0[\x95\x02\x00\x00\x00\x00\xC0\xF4\x8B\x00\x00
  00077+ \x00\x00\x00\xD0[\x95\x02\x00\x00\x00\x00 ~\x8B\x00\x00\x00\x00\x00
  00096+ \x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xFF\xFF\xFF\xFFHTTP/1
  00118+ .1 200 OK\r\n
  00129  Server: Apache-Coyote/1.1\r\n
  00156  Content-Description: File Transfer\r\n
  00192  Content-Disposition: attachment;filename=xxxxxxxxxxxxxxxxxxxxxxxxxxxx
  00261+ \r\n
  00263  Content-Length:: 137358\r\n
  00288  Content-Transfer-Encoding: binary \r\n
  00324  Expires: 0\r\n
  00336  Content-Type: application/octet-stream\r\n
  00376  Transfer-Encoding: chunked\r\n
  00404  Date: Mon, 23 Dec 2019 17:59:55 GMT\r\n
  00441  Connection: close\r\n
  00460  \r\n

lukas@dev:~$