"Hey, I'm loading stuff in my game and only getting like 20 MB/sec , what gives?"
So we started trying to dig into what he was doing exactly - are you opening/reading the files with all the right flags, are you on 4k alignment, are you on a thread so you aren't just stalling out for the seeks, etc. ? In the end I think we figured out that his problem was he was reading too many small files, to which he replied :
"Oh yeah, duh, I've always packed files into bundles and loaded big chunks in previous games, but I just hadn't gotten around to it yet in this and it was bugging me that my level loads were taking so long."
! Ah ha ! To me this is where Oodle comes in as a product. It's not super hard stuff, but it's something that people always put off until the very end, which is kind of a shame because it means their level loads take forever during dev. So for three years while you're making your game you suffer through annoying slow level loads. Instead you buy Oodle on day 1 and your level loads are automagically fast.
I believe in this as a product, in the sense that I think we can make it, and it would be extremely valuable, but I'm not convinced that the game industry is mature enough to buy it. The game industry has always suffered from the mistaken thinking of "I could write that myself, so I won't pay for it". That's silly. What you should look at is what's the full cost of writing it yourself, including debugging, and perhaps most importantly including the opportunity cost of spending that time on this instead of something else, or the opportunity cost of not having feature X until you've gotten around to writing it.
For example, say you're on a one man dev team and you lay out your coding tasks for your project in order {A,B,C,D,E} . All during dev you are suffering a penalty from not having feature E done. If it would make dev a lot easier to have feature E done up front, you should pay a *lot* of money to get it immediately. Some of the classic mistakes in this vein are things like profile HUDs which people put off until the end, but would provide huge benefit if you had from the beginning; another one people aren't aware of is level save/load and memory card support - you think of that as a minor detail to do at the end, but as soon as you do it level designers are walking around with scenarios saved on memory cards to show to each other and they get a huge productivity boost, so it would have been a nice win to do early (though then you have maintenance pain).
To me the big win of Oodle is :
Client just writes code to load files one by one with plain old fopen/fread/whatever.
Behind the scenes Oodle magically robustly incrementally syncs PC files to consoles, packages files into bundles, makes prefetch lists and prefetches bundles, compresses & decompresses bundles on threads. You have ship quality fast loading from the beginning.
That's a small, simple, great product I think. The problem is to make it something compelling enough to sell we start to add lots of features : high performance paging in/out for seamless worlds, hot reloading of changed content, smooth IO integration with streaming data files like audio/video, DVD layout optimization, texture compressors, etc. which really just wind up clouding the tiny little valuable product at the core.
"The game industry has always suffered from the mistaken thinking of "I could write that myself, so I won't pay for it". "
ReplyDeleteThis is so true. At a leading mobile gaming publisher (that I won't cite) I used to work at for 6 years, they're doing exactly that:
They're in big need to unify their audio work, because they make the same games on lots of different machines. Fmod + Designer would be very handy => they sound designers would do the work once and it would all be ported almost magically to the other consoles/iPhone/whatever, plus a lot of the work is taken out of the hand of programmers (no more begging the programmer for varying the pitch of a sound randomly for example).
They really don't want to spend $3000 on this because it's too expensive. So what are they doing instead? The obvivous "cheaper" alternative: trying to make a custom audio engine + tools from scratch. Which is going to be more expensive than Fmod from the first month of development, and will do 0.01% of what it does.
Now that's their mindset for just about everything, 3d engines, tools, servers. EVERYTHING. In the end, everything they make/use costs 10 times more than existing, proven solutions, works 1/100th as well, and offers 1/100th of the possibilities.
There's no point trying to think rationaly in some game companies. They're just retarded.
Yeah, well some managers are just hopeless.
ReplyDeleteWhat I'm talking about is even reasonable managers. They often do some computation like :
we can write that in 20 man hours. A man hour costs us $300 , so if it costs $6000 or more, we'll just write it ourselves.
No! You feel all analytic because you did some math, but you aren't including all the factors correctly.
One issue I believe is that people overvalue the benefit of having written the code themselves. They imagine this means they will know it better, it will be better quality, etc. Yes, in some cases that's true, but in others the opposite is true.
Obviously code that's at the core of your project you want to write yourself, but super peripheral edge stuff has very little value.
Yeah, the Not-Invented-Here disease is just crazy (my employer also suffers from this), since the right reasons for doing this are pretty rare. There is a potentially huge opportunity cost of losing an in-house engineer to a non-core aspect of your product. For the peripheral stuff, of course you should try to benefit from the economies of scale.
ReplyDeleteThat calculus also lacks a measure of risk. Even if the 20 hour estimate were accurate, it is almost certainly not precise. It is probably like 20 +/- 50 hours.
There's also this "race to the bottom" problem, where maybe the manager asks a few engineers how long it would take to do something. Invariably, the shortest estimate heard is what goes into the schedule.