Extremely Large HTTP Responses with Chunked Encoding Blocked

Help!
1

I’m trying to figure out why haproxy is blocking server responses with the “PH” flag.

The responses are VERY large, we’re talking 3-5GB responses. This particular response is 3633514102 bytes (3.4GB).

I have tried a number of things such as setting tune.bufsize to 5GB in size. I’ve tried turning on option accept-invalid-http-response however, I’m kind of just fumbling around in the dark trying to guess what the problem might be.

I’m not sure what combination of settings I need to support such large responses, or if something else is going on here?

I’m on Version 2.2.6-1~bpo10+1

I can see that it’s complaining about the error at position 130, which is the hex representation of the chunk size (3633514102). But I can’t for the life of me figure out why it’s complaining at that location.

The “show error” output is here:

Total events captured on [22/Jan/2021:18:41:25.300] : 1
 
[22/Jan/2021:18:41:20.864] backend turbogeth (#11): invalid response
  frontend http-https-in (#2), server archive04 (#2), event #0, src <redacted>:39386
  buffer starts at 0 (including 0 out), 14936 free,
  len 1448, wraps at 16336, error at position 130
  H1 connection flags 0x00000000, H1 stream flags 0x00004014
  H1 msg state MSG_CHUNK_SIZE(26), H1 msg flags 0x00001716
  H1 chunk len 0 bytes, H1 body len 0 bytes :
  
  00000  HTTP/1.1 200 OK\r\n
  00017  Content-Type: application/json\r\n
  00049  Vary: Origin\r\n
  00063  Date: Fri, 22 Jan 2021 18:41:20 GMT\r\n
  00100  Transfer-Encoding: chunked\r\n
  00128  \r\n
  00130  d8930676\r\n
  00140  {"jsonrpc":"2.0","id":1,"result":{"gas":4300872,"failed":false,"return
  00210+ Value":"","structLogs":[{"pc":0,"op":"CALLDATASIZE","gas":5810070,"gas
  00280+ Cost":2,"depth":1,"stack":[],"memory":[],"storage":{}},{"pc":1,"op":"R
  00350+ ETURNDATASIZE","gas":5810068,"gasCost":2,"depth":1,"stack":["000000000
  00420+ 0000000000000000000000000000000000000000000000000000b24"],"memory":[],
  00490+ "storage":{}},{"pc":2,"op":"RETURNDATASIZE","gas":5810066,"gasCost":2,
  00560+ "depth":1,"stack":["00000000000000000000000000000000000000000000000000
  00630+ 00000000000b24","00000000000000000000000000000000000000000000000000000
  00700+ 00000000000"],"memory":[],"storage":{}},{"pc":3,"op":"CALLDATACOPY","g
  00770+ as":5810064,"gasCost":558,"depth":1,"stack":["000000000000000000000000
  00840+ 0000000000000000000000000000000000000b24","000000000000000000000000000
  00910+ 0000000000000000000000000000000000000","000000000000000000000000000000
  00980+ 0000000000000000000000000000000000"],"memory":["0000000000000000000000
  01050+ 000000000000000000000000000000000000000000","0000000000000000000000000
  01120+ 000000000000000000000000000000000000000","0000000000000000000000000000
  01190+ 000000000000000000000000000000000000","0000000000000000000000000000000
  01260+ 000000000000000000000000000000000","0000000000000000000000000000000000
  01330+ 000000000000000000000000000000","0000000000000000000000000000000000000
  01400+ 000000000000000000000000000","000000000000000000
2

https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#8.5

PD The proxy blocked an incorrectly formatted chunked encoded message in
a request or a response, after the server has emitted its headers. In
most cases, this will indicate an invalid message from the server to
the client. HAProxy supports chunk sizes of up to 2GB - 1 (2147483647
bytes). Any larger size will be considered as an error.

3

This is interesting as the logs indicate that the block was due to “PH” so I didn’t look at “PD”

I take it this is not configurable?

4

No, I don’t think this is configurable, I don’t think anybody ever anticipated the need for such huge chunks. If you need this, you may want to file a feature request on github. This limitation dates back quite some time, and large parts of haproxy have been rewritten, so it’s possible we don’t really need this limitation anymore.

That said, if this shows up as PH instead of PD then it’s a bug either in the code or in the documentation, an should be filed as such:

5

I’ll file both. Thank you, sir.