10-30-08 Oodle Rev Tracking

I'd like Oodle to work really nicely with running the "compiled" version all the time, and server compiles and local changes, but I'm starting to think it's impossible.

The way I would like it to work :

A server somewhere builds nice optimized bundles of the levels from checked in content
The server puts those on a share somewhere

Clients (artists, etc.) sync to the server bundles

When a client runs the game, by default it loads from the server bundles

That has many advantages - they get fast loads, and the game runs like it will on ship, so the artists can
see paging problems, memory use, all that good stuff.  I'm a big believer that having your developers run as
similar to the real shipping product as possible is a huge win.

When clients have newer local copies of content, the game loads those individual files as patches
on top of the server bundles.

Thus clients can make changes and see them instantly in the game without rebaking the whole level,
but they still get most of the advantage of using the compiler server bundling.

That's all great in theory, but it falls apart when you start looking at this question of which files go in the patch. That is, which files should I load from the compiled bundles and which should I load singly as local overrides.

The first idea is to use modtimes. That falls down quickly even in very simple cases. Just using the modtimes of the bundles works fine on your local machine if you never get stuff from a server, but once you share files across machines the modtimes are worthless (particularly if you use something like perforce and have server modtime turned off, which you probably do; the perforce modtime option is strongly discouraged by perforce and by me).

One option is just to punt and not automatically load the client versions. Make the client take an extra step to get the local overrides to load. One way to do that is to just go ahead and use modtime, which will fail sometimes, but tell them to "touch" a file to get it in the game. You could provide helpers to make that easier, like for example looking at the p4 open for edit list and making sure those all come in as overrides.


Ivan-Assen said...

There seems to be an irreconcilable problem between fast loading and flexibility of local artist overrides, patches, mods, localization and other legitimate reasons to replace a file nicely bundled and optimized and ordered and compressed by a long thoughtfull build process on a super-duper server in a basement. We end up erring on the side of flexibility (to quote a Russian colleague, Тьи - писи дивелопер, и ниибацо!, or Why give a fuck, you're a PC developer), so we do all kinds of horrible stuff like looking for each missing file in several mounted packfiles plus the underlying filesystem. And not even caching the results, because the file right now may not be there, but in a few seconds an imaginary artists might click Export in an imaginary Max somewhere, and we need to provide him with instant hot reload. Which is very important, because, you know, load times suck :-)

And we don't want to make a "developer mode" where you're flexible and hotloading and whatnot, and a "user mode" where you only load off of tightly packed, preordered packfiles, because we're scared to ship something which we didn't constantly use during development. Maybe if you have a massive QA department which will bang on your game for thousands of hours before release... but this is not the case for us.

cbloom said...

Yeah, I agree that a separate "dev mode" and "user mode" is a bad thing. My goal with Oodle is to make it easy for you to get the best of both worlds. I know it's hard, but hopefully I'm doing the hard work for you and then you will get all the nice fast load convenience when you can, but also the flexibility of hot loads and all that.

The core of the Oodle paging layer is the idea of a "package" which contains some # of files, and a map that tells you what package to get your resource from. For paging, there can sometimes be multiple packages that contain the same resource. That automatically provides a way to do overrides. You can just load up the big optimized package, and then load up single file overrides on top of it and tell the map to point at the overrides, so the game gets the new artist content.

Brian said...

Why make it so complex? Why not just have a local directory of an artist's working files and load those in preference to the common build?

Ivan-Assen said...

Allowing external files in addition to (and overriding) what's packed means you hit the file system with DoesFileExist queries on every file access; at one point, on a previous project, we had on the order of 80k such requests on startup - all of them returning "nope" on a normal installation. This adds up.

cbloom said...

brian -

that's the idea, the problem is I don't want the artist to have to know about that.

If they're using source control I can get the list of checked out files and use that to build the list of overrides.

But that's still not correct, because you can have sequences like this :

-- artist checks out file A
-- server syncs to perforce
-- server starts building packs
-- artists checks in file A
-- server checks in packs
-- artist syncs to perforce

any naive system will mean that the artist will not see their latest changes on file A. I would like to have a system where the artist still gets to see their latest work and doesn't have to worry about it.

cbloom said...

ivan - yeah I know DoesFileExist and modtime checking at startup can be really slow. First of all, in the shipping game that's all disabled and you just load packs.

During development I believe that can be accelerated by keeping a cache. I plan to have a "Watcher" app running all the time which tracks directory changes on the PC. This can be used to do "hot loading" of content directly into the game, and also automation of transfers to the Xenon or PS3. The Watcher can also keep a cache of all the dir listings on the disk in memory, so that the game can get the override list very quickly just by asking the Watcher.

Brian Demsky said...

Yeah, I understood the point. I just wonder whether sophisticated resolution algorithms are really worthwhile. Unless you embed version numbers in the files or have accurate mod times, you will have things break in subtle ways with complex explanations of why they broke.

Donbot said...

What we do is that we have another step to release a resource file to the game (simply checking stuff into perforce doesn't do that). So if you work on something and check it in for you personal work safety we still know that you have a newer version of the file on your hard drive (because we have a little app running on all workstations tracking all check-ins). If you are happy with the result you "release" the file for the game and the server will process and package the file into the global gamebuild which is then synced to everyone when they sync to the server.
That normally works pretty well, but has the limitation that your packgaging must be really fast (replacing/patching new files instead of repackaging everything).


cbloom said...

Yeah, if you have an extra step where people select things manually it's much simpler.

Certainly I want Oodle to be able to that. I don't want to force any policy on the client, if you want to run things like that you definitely can.

One thing I've been contemplating is I could actually provide a list of all resources in the game, and show what version is in the compiled pack and if I think you have a newer version on your disk. Then there could be a check box next to all those and you could manually check them on and off to get overrides.

And yeah, the repackage is very fast, I do it like an incremental link. The slow part of the compile is anything that the game tools do, stuff like mesh optimization or lightmapping or whatever you do.

old rants