New Codec Offers Faster Decompression Than LZ4

Codec Breakthrough Achieves State-of-the-Art Decompression Throughput

Researchers have made significant strides in the development of a cutting-edge codec, boasting state-of-the-art decompression throughput in its ratio class. A recent benchmarking exercise, conducted on the Intel x86-64 architecture and utilizing the tarred silesia corpus, has yielded impressive results. The codec's decompression capabilities have been optimized for out-of-order cores, resulting in substantial gains in throughput.

According to the benchmarking data, the codec achieved a decompression throughput of 5219 MB/s on the misa77 -0 dataset, outperforming LZ4 by a significant margin. The codec also demonstrated competitive compression ratios, comparable to those of LZ4 at high effort levels. Notably, the results showed that the codec's slow compression rate is offset by its exceptional decompression capabilities. In comparison, LZ4 and its high-compression variant, LZ4hc, achieved lower decompression throughputs of 2505 MB/s and 2531 MB/s, respectively.

The breakthrough codec's design has been optimized for efficiency, with a focus on reducing branches and streamlining decompression. This approach has enabled the codec to take full advantage of out-of-order core architectures, resulting in its impressive performance. The full details of the codec and its development can be found in the comments section of a recent post on Y Combinator's news site.