As before, this is Windows x64 on a Core i7-3770, and we're looking at compression ratio vs. decode speed, not considering encode speed, and running all compressors in their highest compression level.
On the "seven" testset, compressing each file independently, then summing decode time and size for each file :
|
|
|
The loglog chart shows log compression ratio on the Y axis and log decode speed on the X axis (the numeric labels show the pre-log values). The upper right is the Pareto frontier.
The raw numbers are :
total : lzma 16.04 -9 : 3.186 to 1 : 52.84 MB/s
total : lzham 1.0 -d26 -4 : 2.932 to 1 : 149.57 MB/s
total : brotli24 2017-12-12 -11 : 2.958 to 1 : 203.91 MB/s
total : zlib 1.2.11 -9 : 2.336 to 1 : 271.61 MB/s
total : zstd 1.3.3 -22 : 2.750 to 1 : 474.47 MB/s
total : lz4hc 1.8.0 -12 : 2.006 to 1 : 2786.78 MB/s
total : Leviathan8 : 3.251 to 1 : 675.92 MB/s
total : Kraken8 : 3.097 to 1 : 983.74 MB/s
total : Mermaid8 : 2.846 to 1 : 1713.53 MB/s
total : Selkie8 : 2.193 to 1 : 3682.88 MB/s
Isolating just Kraken, Leviathan, and ZStd on the same test (ZStd is the closest non-Oodle codec), we can look at file-by-file performance :
|
|
|
The loglog shows each file with a different symbol, colored by the compressor.
The speed advantage of Oodle is pretty uniform, but the compression advantage varies by file type. Some files simply have more air that Oodle can squeeze out beyond what other LZ's find. For example if you only looked at enwik7 (xml/text), then all of the modern LZ's considered here (ZStd,Oodle,Brotli,LZHAM) would get almost exactly the same compression; there's just not a lot of room for them to differentiate themselves on text.
Runs on a couple other standard files :
On the Silesia/Mozilla file :
|
|
|
mozilla : lzma 16.04 -9 : 3.832 to 1 : 63.79 MB/s
mozilla : lzham 1.0 -d26 -4 : 3.570 to 1 : 168.96 MB/s
mozilla : brotli24 2017-12-12 -11 : 3.601 to 1 : 246.68 MB/s
mozilla : zlib 1.2.11 -9 : 2.690 to 1 : 275.40 MB/s
mozilla : zstd 1.3.3 -22 : 3.365 to 1 : 503.44 MB/s
mozilla : lz4hc 1.8.0 -12 : 2.327 to 1 : 2509.82 MB/s
mozilla : Leviathan8 : 3.831 to 1 : 691.37 MB/s
mozilla : Kraken8 : 3.740 to 1 : 985.35 MB/s
mozilla : Mermaid8 : 3.335 to 1 : 1834.49 MB/s
mozilla : Selkie8 : 2.793 to 1 : 3145.63 MB/s
On win81 :
|
|
|
win81 : lzma 16.04 -9 : 2.922 to 1 : 51.87 MB/s
win81 : lzham 1.0 -d26 -4 : 2.774 to 1 : 156.44 MB/s
win81 : brotli24 2017-12-12 -11 : 2.815 to 1 : 214.02 MB/s
win81 : zlib 1.2.11 -9 : 2.207 to 1 : 253.68 MB/s
win81 : zstd 1.3.3 -22 : 2.702 to 1 : 472.39 MB/s
win81 : lz4hc 1.8.0 -12 : 1.923 to 1 : 2408.91 MB/s
win81 : Leviathan8 : 2.959 to 1 : 757.36 MB/s
win81 : Kraken8 : 2.860 to 1 : 949.07 MB/s
win81 : Mermaid8 : 2.618 to 1 : 1847.77 MB/s
win81 : Selkie8 : 2.142 to 1 : 3467.36 MB/s
And again on the "seven" testset, but this time with the files cut into 32 kB chunks :
|
|
|
total : lzma 16.04 -9 : 2.656 to 1 : 43.25 MB/s
total : lzham 1.0 -d26 -4 : 2.435 to 1 : 76.36 MB/s
total : brotli24 2017-12-12 -11 : 2.581 to 1 : 178.25 MB/s
total : zlib 1.2.11 -9 : 2.259 to 1 : 255.18 MB/s
total : zstd 1.3.3 -22 : 2.363 to 1 : 442.42 MB/s
total : lz4hc 1.8.0 -12 : 1.849 to 1 : 2717.30 MB/s
total : Leviathan8 : 2.731 to 1 : 650.23 MB/s
total : Kraken8 : 2.615 to 1 : 975.69 MB/s
total : Mermaid8 : 2.455 to 1 : 1625.69 MB/s
total : Selkie8 : 1.910 to 1 : 4097.27 MB/s
By cutting into 32 kB chunks, we remove the window size disadvantage suffered by zlib and LZ4. Now all the codecs have the same match window, and the compression difference only comes from what additional features they provide. The small chunk also stresses startup overhead time and adaptation speed.
The Oodle codecs generally do even better (vs the competition) on small chunks than they do on large files. For example LZMA and LZHAM both have large models that really need a lot of data to get up to speed. All the non-Oodle codecs slow down more on small chunks than the Oodle codecs do.
Read more about Leviathan and Oodle 2.6.0 in these other posts on my blog :
Leviathan Rising
Everything new and tasty in Oodle 2.6.0
Leviathan performance on PS4, Xbox One, and Switch
Leviathan detailed performance report
Oodle Hydra and space-speed flexibility
or visit RAD to read for more information about the Oodle SDK
No comments:
Post a Comment