Hi,
is Brotli compression support somewhere on the roadmap? Seems to gain quite some traction.
Regards,
Robert
I’m not aware of anyone working on brotli support for haproxy, no.
We currently support gzip/deflate through libz or libslz (memory-less compression), but the most important issue for haproxy as a reverse proxy is performance. We don’t want to spend too much CPU cycles and memory on compression (which is the whole point of libslz).
That doesn’t mean brotli is not interesting for haproxy - it is. But the 10 - 15% file size improvement on small static files is not something that makes brotli very interesting at this point. HTTP/2 is certainly more important right now.
Anyone interested in contributing should coordinate on the mailing list first (and check the file CONTRIBUTING).
If you are worried about cpu speed, brotli is one of the fastest compression algorithms out there. Brotli quality 0 is about three times faster than gzip at lowest quality setting, and compresses more, too.
The 10-15 % does not match what people have measured. Something was off with your measurement. For static files of HTTP, you will get 25-30 % size reduction with slowest brotli option compared to the slowest gzip option (zopfli). Also, Zopfli is 2-3x slower than Brotli q11. https://engineering.linkedin.com/blog/2017/05/boosting-site-speed-using-brotli-compression
Brotli typically gives 25 % size savings on large static files, 30 % if it is Asiatic or other multi-byte UTF-8 heavy html doc. On lowest quality compression, you can be both faster and compress quite a lot more – or choose a compromise where you are just a lot faster for the same density. Even you can choose to be worse in density than gzip and just a lot faster. Brotli has quite a lot more dynamics in this regard.
Typically one core can brotli compress 450+ MB/s (compared to 150 MB/s with zlib), fully saturating one gbps NIC with compressed bytes. If you are running a well-designed proxy/webserver it tends to be I/O heavy, and you can actually increase the throughput by compressing more.
Since we are not interested in the slowest option for on-the-fly compression anyway (whether it’s gzip or brotli), the 25-30% size reduction is really not that relevant to this use case, and the article you are referring talks about 10 - 20 % size reduction while maintaining compression and decompression times:
Our initial experiment shows that for typical LinkedIn dynamic content, Brotli provides 10%-20% size reduction without sacrificing compression and decompression time compared to gzip.
Brotli looks very effective for static compression (on the backend webserver), but for on-the-fly compression the leap doesn’t appear that big to me. Most of the CDN’s do not support it (pretty sure they would, if it would be such a gamechanger), it’s only usable on HTTPS (because of the middleboxes) and it appears to perform worse than gzip for smaller files (< 32KB).
Brotli support in haproxy would be great, IF we have all the necessary knobs to configure it appropriately (like requiring a minimal content-length for brotli to be effective) and proper gzip coexistence.
However, I disagree that it is a gamechanger, especially for on-the-fly compression (as opposed to static compression on the webserver).
Be that as it may, someone has to actually write the code for it, and I’m not aware anyone is planning to do so. If someone wants to do this, I suggest reading CONTRIBUTING and coordinating the implementation on the mailing list.
Something is off again. Brotli should be increasingly more beneficial for the smallest files. For example this chart shows behavior that I can see in my experiments: https://blogs.dropbox.com/tech/2017/04/deploying-brotli-for-static-content/ shows roughly the same performance for below 32 kB and above. Both the smallest and the largest files compress more densely than with gzip.
If you have a corpus of files where brotli compresses worse than gzip, I’d love to have my hands on it. Might be easy to fix the encoder if you give me such files. Even if you have one file where this happens, I’d be interested getting the link.
No compression algorithm is a “gamechanger”, it is just simple bytes-in-bytes-out. There are just subtle differences in the cost that the user has as the amount of data transferred and in user experienced latency. At world level something like brotli can map to savings of ten billion annually – if everyone used it instead of gzip.
I did not test on my own. I was rather reading the cloudflare blog post about brotli, which suggested they’d use gzip for files below 32KB.
That was 2 years ago though, so this may have changed over time.
Like I said, brotli support in haproxy would be great. But someone has to write the code for it.
You can safely ignore that blog post. They developed a new technique to compare compression algorithms. A method that no one else has used, and for good reasons. I couldn’t reproduce anything like it with proper methods. As one example of bad methodology, their main result interpretation equation ((size A) - (size B))/((time B)-(time A)) is both unstable and commutitative. It is a mathematical impossibility to use a commutitative algorithm to make differences betweeen A and B, since the result is the same for op(A,B) and op(B,A).
Any updates on this matter? 4 years passed since this discussion and for specific use cases, that 20% reduction in size means a lot of cost saving for DTO
Feature request was filed:
A RFC patchset was published, however it did not make it into haproxy:
https://www.mail-archive.com/haproxy@formilux.org/msg32964.html