Almost two years ago, we released Oodle Kraken. Kraken roared onto the scene with high compression ratios and crazy fast decompression (over 1 GB/s on a Core i7 3770). The performance of Kraken immediately made lots of codecs obsolete. We could also see that something was wrong in the higher compression domain.
Kraken gets high compression (much more that zlib; usually more than things like RAR, ZStd and Brotli), but it gets a little less than 7z/LZMA. But to get that small step up in compression ratio, you had to accept a 20X decode speed penalty.
For example, on the "seven" testset (a private test set with a variety of data types) :
LZMA-9 : 3.18:1 , 3.0 enc MB/s , 53.8 dec MB/s
Kraken-7 : 3.09:1 , 1.3 enc MB/s , 948.5 dec MB/s
zlib-9 : 2.33:1 , 7.7 enc MB/s , 309.0 dec MB/s
LZMA gets 3% more compression than Kraken, but decodes almost 20X slower.
That's not the right price to pay for that compression gain. There had to be a better way to
take Kraken's great space-speed performance and extend it into higher compression without giving up
so much speed.
It's easy to see that there was a big gap in a plot of decode speed vs compression ratio :
(this is a log-log plot on the seven test set, on a Core i7 3770. We're not looking at encode speed here at all, we're running the compressors in their max compression mode and we care about the tradeoff of size vs decode speed.)
We've spent the last year searching in that mystery zone and we have found the great beast that lives there and fills the gap : Leviathan.
Leviathan gets to high compression ratios with the correct amount of speed decrease from Kraken (about 33%).
This means that even with better than LZMA ratios on the seven test set, it is still over 2X faster to
decode than zlib :
Leviathan-7: 3.23:1 , 0.9 enc MB/s , 642.4 dec MB/s
LZMA-9 : 3.18:1 , 3.0 enc MB/s , 53.8 dec MB/s
Kraken-7 : 3.09:1 , 1.3 enc MB/s , 948.5 dec MB/s
zlib-9 : 2.33:1 , 7.7 enc MB/s , 309.0 dec MB/s
Leviathan doesn't always beat LZMA's compression ratio (they have slightly different strengths) but they are comparable. Leviathan is 7-20X faster to decompress than LZMA (usually around 10X).
Leviathan is ideal for distribution use cases, in which you will compress once and decompress many times. Leviathan allows you to serve highly compressed data to clients without slow decompression. We do try to keep Leviathan encode's time reasonable, but it is not a realtime encoder and not the right answer where fast encoding is needed.
Leviathan is a game changer. It makes high ratio decompression possible where it wasn't before. It can be used on video game consoles and mobile devices where other decoders took far too long. It can be used for in-game loading where CPU use needs to be minimized. It can be used to keep data compressed on disk with no install, because its decompression is fast enough to do on every load.
Leviathan now makes up a part of the Oodle Kraken-Mermaid-Selkie family. These codecs now provide excellent solutions over a wider range of compression needs.
Read more about Leviathan and Oodle 2.6.0 in these other posts on my blog :
Leviathan Rising
Everything new and tasty in Oodle 2.6.0
Leviathan performance on PS4, Xbox One, and Switch
Leviathan detailed performance report
Oodle Hydra and space-speed flexibility
or visit RAD to read for more information about the Oodle SDK
No comments:
Post a Comment