cbloom rants: 02/2018

2/27/2018

Oodle 2.6.0 : Leviathan Rising

We're pleased to announce the release of Oodle 2.6.0 featuring the new Leviathan codec. Leviathan achieves high compression ratios (comparable to 7z/LZMA) with unparalleled fast decompression.

Oodle is an SDK for high performance lossless data compression. For more about Oodle, or licensing inquiries, visit the RAD Game Tools web site. This is my personal blog where I post supplemental material about Oodle.

Almost two years ago, we released Oodle Kraken. Kraken roared onto the scene with high compression ratios and crazy fast decompression (over 1 GB/s on a Core i7 3770). The performance of Kraken immediately made lots of codecs obsolete. We could also see that something was wrong in the higher compression domain.

Kraken gets high compression (much more that zlib; usually more than things like RAR, ZStd and Brotli), but it gets a little less than 7z/LZMA. But to get that small step up in compression ratio, you had to accept a 20X decode speed penalty.

For example, on the "seven" testset (a private test set with a variety of data types) :

LZMA-9     :  3.18:1 ,    3.0 enc MB/s ,   53.8 dec MB/s
Kraken-7   :  3.09:1 ,    1.3 enc MB/s ,  948.5 dec MB/s
zlib-9     :  2.33:1 ,    7.7 enc MB/s ,  309.0 dec MB/s

LZMA gets 3% more compression than Kraken, but decodes almost 20X slower. That's not the right price to pay for that compression gain. There had to be a better way to take Kraken's great space-speed performance and extend it into higher compression without giving up so much speed.

It's easy to see that there was a big gap in a plot of decode speed vs compression ratio :

(this is a log-log plot on the seven test set, on a Core i7 3770. We're not looking at encode speed here at all, we're running the compressors in their max compression mode and we care about the tradeoff of size vs decode speed.)

We've spent the last year searching in that mystery zone and we have found the great beast that lives there and fills the gap : Leviathan.

Leviathan gets to high compression ratios with the correct amount of speed decrease from Kraken (about 33%). This means that even with better than LZMA ratios on the seven test set, it is still over 2X faster to decode than zlib :

Leviathan-7:  3.23:1 ,    0.9 enc MB/s ,  642.4 dec MB/s

LZMA-9     :  3.18:1 ,    3.0 enc MB/s ,   53.8 dec MB/s
Kraken-7   :  3.09:1 ,    1.3 enc MB/s ,  948.5 dec MB/s
zlib-9     :  2.33:1 ,    7.7 enc MB/s ,  309.0 dec MB/s

Leviathan doesn't always beat LZMA's compression ratio (they have slightly different strengths) but they are comparable. Leviathan is 7-20X faster to decompress than LZMA (usually around 10X).

Leviathan is ideal for distribution use cases, in which you will compress once and decompress many times. Leviathan allows you to serve highly compressed data to clients without slow decompression. We do try to keep Leviathan encode's time reasonable, but it is not a realtime encoder and not the right answer where fast encoding is needed.

Leviathan is a game changer. It makes high ratio decompression possible where it wasn't before. It can be used on video game consoles and mobile devices where other decoders took far too long. It can be used for in-game loading where CPU use needs to be minimized. It can be used to keep data compressed on disk with no install, because its decompression is fast enough to do on every load.

Leviathan now makes up a part of the Oodle Kraken-Mermaid-Selkie family. These codecs now provide excellent solutions over a wider range of compression needs.

Read more about Leviathan and Oodle 2.6.0 in these other posts on my blog :

Leviathan Rising
Everything new and tasty in Oodle 2.6.0
Leviathan performance on PS4, Xbox One, and Switch
Leviathan detailed performance report
Oodle Hydra and space-speed flexibility

or visit RAD to read for more information about the Oodle SDK

Oodle 2.6.0 : Everything new and tasty

A quick run down of all the exciting new stuff in Oodle 2.6.0 :

1. Leviathan!
2. The Kraken, Mermaid & Selkie fast-level encoders are now much faster.
3. Kraken & Mermaid's optimal level encoders now get more compression.
4. Kraken & Mermaid have new bit stream options which allow them to reach even higher compression.
5. Kraken and Mermaid are now more tuneable to different compression ratios and decode speeds.

1. Leviathan!

Read the Leviathan detailed performance report.

2. The Kraken, Mermaid & Selkie fast-level encoders are now much faster.

The non-optimal-parsing encoder levels that are intended for realtime use in Oodle are levels 1-4 aka SuperFast, VeryFast, Fast & Normal.

Their decode speed was always best in class, but previously their encode speed was slightly off the Pareto frontier (the best possible trade off of encode speed vs compression ratio). No longer.

"win81" test file (Core i7-3770 Windows x64)

Oodle 255 :

Kraken1   :  2.33:1 ,   83.3 enc MB/s ,  911.8 dec MB/s
Kraken2   :  2.39:1 ,   68.5 enc MB/s ,  938.0 dec MB/s
Kraken3   :  2.51:1 ,   24.4 enc MB/s , 1005.5 dec MB/s
Kraken4   :  2.57:1 ,   17.3 enc MB/s ,  997.3 dec MB/s

Oodle 260 :

Kraken1   :  2.33:1 ,  135.1 enc MB/s ,  928.1 dec MB/s
Kraken2   :  2.41:1 ,   94.0 enc MB/s ,  937.4 dec MB/s
Kraken3   :  2.52:1 ,   38.3 enc MB/s , 1020.6 dec MB/s
Kraken4   :  2.60:1 ,   23.0 enc MB/s , 1022.9 dec MB/s

Oodle 255 :

Mermaid1  :  2.10:1 ,  106.4 enc MB/s , 2079.4 dec MB/s
Mermaid2  :  2.15:1 ,   78.9 enc MB/s , 2161.1 dec MB/s
Mermaid3  :  2.24:1 ,   26.4 enc MB/s , 2294.0 dec MB/s
Mermaid4  :  2.29:1 ,   26.6 enc MB/s , 2341.1 dec MB/s

Oodle 260 :

Mermaid1  :  2.14:1 ,  161.1 enc MB/s , 2012.1 dec MB/s
Mermaid2  :  2.22:1 ,  104.1 enc MB/s , 2007.7 dec MB/s
Mermaid3  :  2.29:1 ,   39.0 enc MB/s , 2181.7 dec MB/s
Mermaid4  :  2.32:1 ,   32.1 enc MB/s , 2294.0 dec MB/s

Oodle 255 :

Selkie1   :  1.76:1 ,  146.8 enc MB/s , 3645.0 dec MB/s
Selkie2   :  1.85:1 ,  100.5 enc MB/s , 3565.0 dec MB/s
Selkie3   :  1.98:1 ,   28.2 enc MB/s , 3533.2 dec MB/s
Selkie4   :  2.04:1 ,   28.9 enc MB/s , 3675.9 dec MB/s

Oodle 260 :

Selkie1   :  1.78:1 ,  181.7 enc MB/s , 3716.0 dec MB/s
Selkie2   :  1.87:1 ,  114.7 enc MB/s , 3595.7 dec MB/s
Selkie3   :  1.98:1 ,   42.1 enc MB/s , 3653.1 dec MB/s
Selkie4   :  2.02:1 ,   34.9 enc MB/s , 3818.3 dec MB/s

The speed of the fastest encoder (level 1 = "SuperFast") is up by about 60% in Kraken & Mermaid.

Kraken's encode speed vs ratio is now competitive with ZStd, which has long been the best codec for encode speed tradeoff. For example, matching Kraken1 to the closest comparable ZStd levels on the same machine :

at similar encode speed, you can compare the compression ratios :

Kraken1       :  2.33:1 ,  135.1 enc MB/s ,  928.1 dec MB/s
zstd 1.3.3 -4 :  2.22:1 ,  136   enc MB/s ,  595   dec MB/s

or you can look at equal file sizes and compare the encode speed :

Kraken1       :  2.33:1 ,  135.1 enc MB/s ,  928.1 dec MB/s
zstd 1.3.3 -6 :  2:33:1 ,   62   enc MB/s ,  595   dec MB/s

Of course ZStd does have faster encode levels (1-3); Oodle does not provide anything in that domain.

3. Kraken & Mermaid's optimal level encoders now get more compression. (even with 2.5 compatible bit streams)

We improved the ability of the optimal parse encoders to make good decisions and find smaller encodings. This is at level 8 (Optimal4) our maximum compression level with slow encoding.

PD3D : (public domain 3D game data test set)

Kraken8 255    :  3.67:1 ,    2.8 enc MB/s , 1091.5 dec MB/s
Kraken8 260 -v5:  3.72:1 ,    1.2 enc MB/s , 1079.9 dec MB/s

GTS : (private game data test set)

Kraken8 255    :  2.60:1 ,    2.5 enc MB/s , 1335.8 dec MB/s
Kraken8 260 -v5:  2.63:1 ,    1.2 enc MB/s , 1343.8 dec MB/s

Silesia : (standard Silesia compression corpus)

Kraken8 255    :  4.12:1 ,    1.4 enc MB/s ,  982.0 dec MB/s
Kraken8 260 -v5:  4.18:1 ,    0.6 enc MB/s , 1018.7 dec MB/s

(speeds on Core i7-3770 Windows x64)
(-v5 means encode in v5 (2.5.x) backwards compatibility mode)

Compression ratio improvements around 1% might not sound like much, but when you're already on the Pareto frontier, finding another 1% without sacrificing any decode speed or changing the bit stream is quite significant.

4. Kraken & Mermaid have new bit stream options which allow them to reach even higher compression.

PD3D :

Kraken8 255    :  3.67:1 ,    2.8 enc MB/s , 1091.5 dec MB/s
Kraken8 260 -v5:  3.72:1 ,    1.2 enc MB/s , 1079.9 dec MB/s
Kraken8 260    :  4.00:1 ,    1.0 enc MB/s , 1034.7 dec MB/s

GTS :

Kraken8 255    :  2.60:1 ,    2.5 enc MB/s , 1335.8 dec MB/s
Kraken8 260 -v5:  2.63:1 ,    1.2 enc MB/s , 1343.8 dec MB/s
Kraken8 260    :  2.67:1 ,    1.0 enc MB/s , 1282.3 dec MB/s

Silesia :

Kraken8 255    :  4.12:1 ,    1.4 enc MB/s ,  982.0 dec MB/s
Kraken8 260 -v5:  4.18:1 ,    0.6 enc MB/s , 1018.7 dec MB/s
Kraken8 260    :  4.24:1 ,    0.6 enc MB/s ,  985.4 dec MB/s

Kraken in Oodle 2.6.0 now gets Silesia to 50,006,565 bytes at the default space-speed tradeoff target. Kraken in max-compression space-speed setting gets Silesia to 49,571,429 bytes (and is still far faster to decode than anything close).

If we look back at where Kraken started in April of 2016 , it was getting 4.05 to 1 on Silesia , now 4.24 to 1.

Kraken now usually gets more compression than anything remotely close to its decode speed.

Looking back at the old Performance of Oodle Kraken , Kraken only got 2.70:1 on win81. On some files, Kraken has always out-compressed the competition, but win81 was one where it lagged. It does better now :

"win81" test file (Core i7-3770 Windows x64)
in order of decreasing decode speed :

Kraken8       :  2.86:1 ,    0.6 enc MB/s ,  950.9 dec MB/s
old Kraken 215:  2.70:1 ,    1.0 enc mb/s ,  877.0 dec mb/s
Leviathan8    :  2.96:1 ,    0.4 enc MB/s ,  758.1 dec MB/s

zstd 1.3.3 -22           3.35 enc MB/s   473 dec MB/s    38804423  37.01 win81 = 2.702:1
zlib 1.2.11 -9           8.59 enc MB/s   254 dec MB/s    47516720  45.32 win81 = 2.206:1
brotli24 2017-12-12 -11  0.39 enc MB/s   214 dec MB/s    37246857  35.52 win81 = 2.815:1
lzham 1.0 -d26 -4        1.50 enc MB/s   158 dec MB/s    37794766  36.04 win81 = 2.775:1
lzma 16.04 -9            2.75 enc MB/s    51 dec MB/s    35890919  34.23 win81 = 2.921:1

At the time of Kraken's release, it was a huge decode speed win vs comparable compressors, but it sometimes lagged a bit in compression ratio. No longer.

NOTE : Oodle 2.6.0 by default makes bit streams that are decodable by version >= 2.6.0 only. If you need bit streams that can be read by earlier versions, you must set the backward compatible version number that you need. See the Oodle FAQ on backward compatibility.

5. Kraken and Mermaid are now more tuneable to different compression ratios and decode speeds.

The new v6 bit stream has more options, which allows them to smoothly trade off compression ratio for decode speed. The user can set this goal with a space-speed tradeoff parameter.

All the Oodle codecs have a compression level setting (similar to the familiar zip 1-9 level) that trades encode time for decode speed. Unlike many other codecs, Oodle's compressors do not lose *decode* speed at higher encode effort levels. We are not finding more compact encodings by making the decoder slower. Instead you can dial decode speed vs ratio with a separate parameter that changes how the encoder scores decisions.

More about this in : Oodle Hydra and space-speed flexibility.

Read more about Leviathan and Oodle 2.6.0 in these other posts on my blog :

Leviathan Rising
Everything new and tasty in Oodle 2.6.0
Leviathan performance on PS4, Xbox One, and Switch
Leviathan detailed performance report
Oodle Hydra and space-speed flexibility

or visit RAD to read for more information about the Oodle SDK

Oodle 2.6.0 : Leviathan performance on PS4, Xbox One and Switch

Oodle's speed on the Sony PS4 (and Microsoft Xbox One) and Nintendo Switch is superb. With the slower processors in these consoles (compared to a modern PC), the speed advantage of Oodle makes a big difference in total load time or CPU use.

These are run on the private test file "lzt99". I'm mainly looking at the speed numbers here, not the compression ratio (compression wise, we do so well on lzt99 that it's a bit silly, and also not entirely fair to the competition).

On the Nintendo Switch (clang ARM-A57 AArch64 1.02 GHz) :


Oodle 2.6.0 -z8 :

Leviathan   : 2.780 to 1 : 205.50 MB/s
Kraken      : 2.655 to 1 : 263.54 MB/s
Mermaid     : 2.437 to 1 : 499.72 MB/s
Selkie      : 1.904 to 1 : 957.60 MB/s

zlib from nn_deflate

zlib        : 1.883 to 1 :  74.75 MB/s

And on the Sony PS4 (clang x64 AMD Jaguar 1.6 GHz) :


Oodle 2.6.0 -z8 :

Leviathan   : 2.780 to 1 : 271.53 MB/s
Kraken      : 2.655 to 1 : 342.49 MB/s
Mermaid     : 2.437 to 1 : 669.34 MB/s
Selkie      : 1.904 to 1 :1229.26 MB/s

non-Oodle reference (2016) :

brotli-11   : 2.512 to 1 :  77.84 MB/s
miniz       : 1.883 to 1 :  85.65 MB/s
brotli-9    : 2.358 to 1 :  95.36 MB/s
zlib-ng     : 1.877 to 1 : 109.30 MB/s
zstd        : 2.374 to 1 : 133.50 MB/s
lz4hc-safe  : 1.669 to 1 : 673.62 MB/s
LZSSE8      : 1.626 to 1 : 767.11 MB/s

The Microsoft XBox One has similar performance to the PS4. Mermaid & Selkie can decode faster than the hardware DMA compression engine in the PS4 and Xbox One, and usually compress more if they aren't limited to small chunks like the hardware DMA engine needs.

Note that the PS4 non-Oodle reference data is from my earlier runs back in 2016 : Oodle Mermaid and Selkie on PS4 and PS4 Battle : MiniZ vs Zlib-NG vs ZStd vs Brotli vs Oodle . They should be considered only rough reference points; I imagine some of those codecs are slightly different now, but does even a 10 or 20 or 50% improvement really make much difference? (also note that there's no true zlib reference in that PS4 set; miniz is close but a little different, and zlib-ng is faster than standard zlib).

Leviathan is in a different compression class than any of the other options, and is still 2-3X faster than zlib.

Something I spotted while gathering the old numbers that I think is worth talking about:

If you look at the old Kraken PS4 numbers from Oodle 2.3.0 : Kraken Improvement you would see :


PS4 lzt99

old :

Oodle 2.3.0 -z6 : 2.477 to 1 : 389.28 MB/s
Oodle 2.3.0 -z7 : 2.537 to 1 : 363.70 MB/s

vs new :

Oodle 2.6.0 -z8 : 2.655 to 1 : 342.49 MB/s

(-z8 encode level didn't exist back then)

Oh no! Oodle's gotten slower to decode!

Well no, it hasn't. But this is a good example of how looking at just space or speed on their own can be misleading.

Oodle's encoders are always optimizing for a space-speed goal. There are a range of solutions to that problem which have nearly the same space-speed score, but have different sizes or speeds.

So part of what's happened here is that Oodle 2.6.0 is just hitting a slightly different spot in the space-speed solution space than Oodle 2.3.0 is. It's finding a bit stream that is smaller, and trades off some decode speed for that. With its space-speed cost model, it measures that tradeoff as being a good value. (the user can set the relative value of time & bytes that Oodle uses in its scoring via the spaceSpeedTradeoffBytes parameter).

But something else has also happened - Oodle 2.6.0 has just gotten much better. It hasn't just stepped along the Pareto curve to a different but equally good solution - it has stepped perpendicularly to the old Pareto curve and is finding better solutions.

At RAD we measure that using the "correct" Weissman score which provides a way of combining a space-speed point into a single number that can be used to tell whether you have made a real Pareto improvement or just a tangential step.

The easiest way to see that you have definitely made an improvement is to run Oodle 2.6.0 with a different spaceSpeedTradeoffBytes price so that it provides a simpler relationship :


PS4 lzt99

new, with spaceSpeedTradeoffBytes = 1024

Oodle 2.6.0 -z8 : 2.495 to 1 : 445.56 MB/s

vs old :

Oodle 2.3.0 -z6 : 2.477 to 1 : 389.28 MB/s

Now we have higher compression and higher speed, so there's no question of whether we lost anything.

In general the Oodle 2.6.0 Kraken & Mermaid encoders are making decisions that slightly bias for higher compression (and slower decode; though often the decode speed is very close) than before 2.6.0. If you find you've lost a little decode speed and want it back, increase spaceSpeedTradeoffBytes (try 400).

Read more about Leviathan and Oodle 2.6.0 in these other posts on my blog :

Leviathan Rising
Everything new and tasty in Oodle 2.6.0
Leviathan performance on PS4, Xbox One, and Switch
Leviathan detailed performance report
Oodle Hydra and space-speed flexibility

or visit RAD to read for more information about the Oodle SDK

Oodle 2.6.0 : Leviathan detailed performance report

Let's have a look at a few concrete tests of Leviathan's performance.

All of the tests in this post are run on Windows, x64, on a Core i7-3770. For performance on some other game platforms, see : Leviathan performance on PS4, Xbox One, and Switch .

All the compressors in this test are run on in their slowest encoding level. We're looking for high compression and fast decompression, we're not looking for fast compression here. See @@ for a look at encode times.

Tests are on groups of files. The results show the total compressed size & total decode time over the group of files, but each file is encoded independently. The maximum window size is used for all compressors.

The compressors tested here are :

Kraken8          : Oodle Kraken at level 8 (Optimal4)
Leviathan8       : Oodle Leviathan at level 8 (Optimal4)

lzma_def9        : LZMA -mx9 with default settings (lc3lp0pb2fb32) +d29
lzmalp3pb3fb1289 : LZMA -mx9 with settings that are better on binary and game data (lc0lp3pb3fb128) +d29
zlib9            : zlib 1.2.8 at level 9

LZMA and zlib here were built with MSVC; I also checked there speed in a GCC build and confirmed it is nearly identical.

Two settings for LZMA are tested to try to give it the best chance of competing with Leviathan. LZMA's default settings tend to be good on text but not great on binary (often even Kraken can beat "lzma_def" on binary structured data). I've also increased LZMA's fast bytes to 128 in the non-default options, the default value of 32 is a bit of a detriment to compression ratio, and most of the competition (ZStd, Brotli, LZHAM) use a value more like 128. I want to give LZMA a great chance to compete; we don't need to play any games with selective testing to make Leviathan look good.

Let the tests begin!

PD3D :

Public Domain 3d game data test set :

Kraken8         :  4.02:1 ,    1.0 enc MB/s , 1013.3 dec MB/s
Leviathan8      :  4.21:1 ,    0.5 enc MB/s ,  732.8 dec MB/s
zlib9           :  2.89:1 ,    5.2 enc MB/s ,  374.5 dec MB/s
lzmalp3pb3fb1289:  4.08:1 ,    3.4 enc MB/s ,   65.4 dec MB/s
lzma_def9       :  3.97:1 ,    4.3 enc MB/s ,   65.4 dec MB/s

Game BCn :

A mix of BCn textures from a game (mostly BC1, BC4, and BC7) :

Kraken8         :  3.67:1 ,    0.7 enc MB/s ,  881.0 dec MB/s
Leviathan8      :  3.84:1 ,    0.4 enc MB/s ,  661.9 dec MB/s
zlib9           :  2.72:1 ,    6.2 enc MB/s ,  321.5 dec MB/s
lzmalp3pb3fb1289:  3.67:1 ,    2.0 enc MB/s ,   66.9 dec MB/s
lzma_def9       :  3.68:1 ,    2.3 enc MB/s ,   67.0 dec MB/s

ASTC testset :

A private corpus of game ASTC texture data :

Kraken8         :  1.15:1 ,    1.7 enc MB/s , 1289.6 dec MB/s
Leviathan8      :  1.17:1 ,    1.0 enc MB/s ,  675.8 dec MB/s
zlib9           :  1.10:1 ,   18.9 enc MB/s ,  233.8 dec MB/s
lzma_def9       :  1.17:1 ,    6.2 enc MB/s ,   21.4 dec MB/s
lzmalp3pb3fb1289:  1.19:1 ,    6.1 enc MB/s ,   21.4 dec MB/s

gametestset :

A private corpus of game test data :

Kraken8         :  2.68:1 ,    1.0 enc MB/s , 1256.2 dec MB/s
Leviathan8      :  2.83:1 ,    0.6 enc MB/s ,  915.0 dec MB/s
zlib9           :  1.99:1 ,    8.3 enc MB/s ,  332.4 dec MB/s
lzmalp3pb3fb1289:  2.76:1 ,    3.4 enc MB/s ,   46.5 dec MB/s
lzma_def9       :  2.72:1 ,    4.2 enc MB/s ,   46.3 dec MB/s

Silesia :

Silesia standard public compression test corpus :

Kraken8         :  4.25:1 ,    0.6 enc MB/s ,  996.0 dec MB/s
Leviathan8      :  4.35:1 ,    0.4 enc MB/s ,  804.5 dec MB/s
zlib9           :  3.13:1 ,    8.4 enc MB/s ,  351.3 dec MB/s
lzmalp3pb3fb1289:  4.37:1 ,    1.9 enc MB/s ,   80.2 dec MB/s
lzma_def9       :  4.27:1 ,    2.6 enc MB/s ,   79.0 dec MB/s

brotli24 -11    :  4.21:1 ,                    310.9 dec MB/s
zstd 1.3.3 -22  :  4.01:1 ,                    598.2 dec MB/s
(zstd and brotli run in lzbench)  (brotli is 2017-12-12)

Read more about Leviathan and Oodle 2.6.0 in these other posts on my blog :

Leviathan Rising
Everything new and tasty in Oodle 2.6.0
Leviathan performance on PS4, Xbox One, and Switch
Leviathan detailed performance report
Oodle Hydra and space-speed flexibility

or visit RAD to read for more information about the Oodle SDK

2/12/2018

Oodle 2.6.0 : Impending Deprecation of pre-Kraken codecs

Starting in Oodle 2.6.0 the pre-Kraken codecs will be slowly phased out.

In 2.6.0 they will still be available for both encoding & decoding. However, they will be marked "deprecated" to discourage their use. I'm doing this two ways.

One : if you encode with them, they will log a warning through the new Oodle "UsageWarning" system. Decoding with them will not log a warning. This warning can be disabled by calling Oodle_SetUsageWarnings(false). (Of course in shipping you might also have set the Oodle log function to NULL via OodlePlugins_SetPrintf, which will also disable the warning log). (Also note that in MSVC builds of Oodle the default log function only goes to OutputDebugString so you will not see anything unless you either have a debugger attached, change the log function, or use OodleX to install the OodleX logger).

Two : the enum for the old compressor names will be hidden in oodle2.h by default. You must define OODLE_ALLOW_DEPRECATED_COMPRESSORS before including oodle2.h to enable their definition.

As noted, decoding with the old codecs will not log a usage warning, and can be done without setting OODLE_ALLOW_DEPRECATED_COMPRESSORS. That is, Oodle 2.6.0 requires no modifications to decode old codecs and will not log a warning. You only need to consider these steps if you want to *encode* with the old codecs.

In some future version the encoders for the old codecs will be removed completely. All decoders will continue to be shipped for the time being.

I'm intentionally make it difficult to encode with the old codecs so that you transition to one of the new codecs.


Long term support codecs :

Kraken, Mermaid, Selkie (and Hydra)

codecs being deprecated :

LZNA, BitKnit, LZA, LZH, LZHLW, LZNIB, LZB16, LZBLW

The new codecs simply obsolete the old codecs and they should not be used any more. The old codecs are also a mix of some fuzz safe, some not. The new codecs are all fuzz safe and we want to remove support for non-fuzz-safe decoding so that it's not possible to do by accident.

In pursuit of that last point, another change in 2.6.0 is removing the default argument value for OodleLZ_FuzzSafe in the OodleLZ_Decompress call.

Previously that argument had a default value of OodleLZ_FuzzSafe_No , so that it would allow all the codecs to decompress and was backwards compatible when it was introduced.

If you are not explicitly passing something there, you will get a compiler error and need to pass something.

If possible, you should pass OodleLZ_FuzzSafe_Yes. This will ensure the decode is fuzz safe (ie. won't crash if given invalid data to decode).

The only reason that you would not pass FuzzSafe_Yes is if you need to decode some of the older non-fuzz-safe codecs. We recommend moving away from those codecs to the new post-Kraken codecs (which are all fuzz safe).


Fuzz safe codecs :

Kraken, Mermaid, Selkie (and Hydra)
BitKnit, LZB16, LZH, LZHLW

Non-fuzz-safe codecs :

LZNA, LZA, LZNIB, LZBLW

If you need to decode one of the non-fuzz-safe codecs, you must pass FuzzSafe_No. If you pass FuzzSafe_Yes, and the decoder encounters data made by that codec, it will return failure.

Fuzz safety in Oodle means that unexpected data will not crash the decoder. It will not necessarilly be detected; the decode might still return success. For full safety, your systems that consume the data post decompression must all be fuzz safe.

Another semi-deprecation coming in Oodle is removing high-performance optimization for 32-bit builds (where 64-bit is available).

What I mean is, we will continue to support & work in 32-bit, but it will no longer be an optimization priority. We may allow it to get slower than it could be. (for example in some cases we just run the 64-bit code that requires two registers per qword; not the ideal way to write the 32-bit version, but it saves us from making yet more code paths to optimize and test).

We're not aware of any games that are still shipping in 32-bit. We have decided to focus our time budget on 64-bit performance. We recommend evaluating Oodle and always running your tools in 64-bit.

If you're a user who believes that 32-bit performance is important, please let us know by emailing your contact at RAD.