09-22-12 - Oodle Beta and Roadmap

Oodle went Beta a few weeks ago (yay). If you're a game developer interested in Oodle you can mail oodle at rad.

I wanted to write about what's in Oodle new and what's coming, as a sort of roadmap for myself and others. This is not a detailed feature list; contact RAD for documents with more details about Oodle features.

Oodle Beta : (now)

What's in Oodle at the moment :

  • OodleLZH and LZHLW : the LZ-huff compressors I have written about repeatedly. Designed for offline compression and very fast low memory decompression in game.

  • PC, PS3 (+SPU) and Xenon support

  • OodleIOQ : Fast low-level async IO

  • OodleIOQStream : Streaming async IO (keeping buffers full for media playback) ; with Bink and Miles drop-in IO layers.

  • OodleFile : Synchronous/buffered IO built on the async IO layer. This is a pluggable thing and there are memory OodleFiles and transparent LZ decompression OodleFiles and so on. Currently the ability to make your own OodleFile type is not exposed, but maybe it will be in 1.1 or so.

  • OodleWork : Work-stealing thread pool system. Includes stackless coroutines. A bit more conscientiously designed than most.

  • OodleHandle : Generic handle/dependency system so you can set up async chains (like load X then decompress it, then process it, etc.), WFMO mixing IO and work and SPU jobs and so on.

  • OodlePackage : "Packages" (archives) with solid archiving and very async decompressors.

  • OodleVFS : (virtual file system) for drive name mapping and per-drive file system overrides; also supports transparent mirroring from one drive to another.

  • VFS patches, for overlaying a set of files over an existing set.

  • Multithreaded memory allocator ; some other handy threading stuff like a tiny mutex; lots of other threading stuff that's in Oodle but not exposed to public API.

  • Oodle Thread Viewer (super-lite Telemetry of sorts) (and real Telemetry integration for people who have that)

Oodle RC / 1.0 : (2012-2013)

  • More platforms (Mac, Linux, iOS)

  • Mostly just loose ends from stuff in the Beta. eg. Packages need to have update scripts to rebuild them when the source content changes. Packages need to be able to have their header in a separate file from the body (for header aggregation). etc.

  • Lots of little stuff needs to get wrapped up; for example I need the ability to cache open file handles so that multiple opens for read on the same handle just use the cache, with some kind of LRU type of thing and a limit on the number of open handles kept.

Oodle 1.1 : (around GDC 2013)

  • All major/current platforms (Mac, Linux, WiiU, ARM/mobile, next-gen consoles)

  • A better cross-platform DVD to HD transparent mirroring system.

  • OodleLZB and LZA (LZ-bytewise and LZ-arithmetic) , for when the LZH's don't quite hit the space/speed tradeoff you need.

  • DVD layout tool and DVD emulator ; back when I was a real game dev I was dismayed at using the various platforms DVD emulators and layout tools, so I think doing a cross-platform one would be awesome. My goal for the emulator is that you should be able to just use your normal content on your disk, but with a layout file to tell me where on the virtual disc each file is I can emulate seeking and such. It should be possible to just flip a switch and test your game as if it was on DVD.

  • PC File Serving Host. Basically a server you can run on your dev PC that will serve files to Oodle on a console so that you get transparent mirroring. (some consoles provide support for this already, some don't; it should be possible to do it for all consoles in a unified way).

  • High level paging / resource manager interface. Oodle 1.0 is very imperative ; C-style. You can certainly make your own paging system on top of the Oodle 1.0 interfaces, but there's been a reasonable amount of queries about an out of the box ready to go resource manager. This would probably be a C++ style interface (perhaps in C though using function pointers).

  • Paging transition package optimizer : you give me a list of resources for each region and a region transition map, I generate transition bundles with constrained optimization.

  • OodleHandle for GPU Compute jobs ; be able to spawn & wait on them in the unified IO/Work system.

Oodle 1.2/2.0 :

  • Image stuff is probably the biggest thing for the major rev of Oodle. DXTC, R/D-lossy DXTC, some kind of lossy image codec, perhaps just a fast JPEG decoder. I have lots of this code already, but it needs a big cleanup and "publicating".

  • Patch generator.

  • Stackful coroutines ? If so, then also a set of primitives that yield the coroutine instead of doing an OS wait; eg. "coMutex" and such that yield if blocked, and of course "coFRead" for files that yields if it can't return immediately.


nothings said...

In your comparison of LZNib against the other compressors, you didn't include OodleLZs. But I take it LZNib's speed vs. size isn't interesting enough to be worth pursuing for Oodle?

(I though the fact that it beat zlib was pretty amazing, even if I am sad that you stole my thunder.)

cbloom said...

"In your comparison of LZNib against the other compressors, you didn't include OodleLZs."

Yeah, cuz they would just kill it, they're at a different place on the space/speed tradeoff; I was trying to only compare to other bytewise compressors, but tossed in Zip as well as a well-known baseline.

"But I take it LZNib's speed vs. size isn't interesting enough to be worth pursuing for Oodle?"

Yeah it definitely is interesting; it's on the list as "LZB" ; I'm not sure yet exactly what OodleLZB will be, I have a few different ideas to explore aside from LZNib before I set that in stone.

I see LZB as being ideal on platforms with very little spare CPU that can't even afford to run an LZH.

I wish I had a team of graduate students to go off and implement all my ideas; I have way more than I have time to get to. It was only the rumbling of your imminent thunder that got me motivated to go try LZNib, which is an idea I've been sitting on for a long time. Nothing like the whiff of competition to get me motivated.

nothings said...

Oh, somehow I misread "LZ-bytewise" as something else (I don't remember what, but I know I was looking for that).

Cool beans.

old rants