12-22-04If I look at my hourly salary (before tax) as if I had a 40 hour work week, it looks really good. I go wow, it seems like I could just work a few hours and be able to do things like buy a nice dinner. Then I look at it in terms of the actual # of hours I work (probably about 55/week on average), after tax, and suddenly it looks like shit. All of a sudden that ridiculous sandwich from across the street looks way too expensive. Plumbers, city workers, etc. all make more per hour than me, and I work way harder, not just in terms of hours, but in terms of stress and intensity of work. Cursed ambition and desire to do something productive with my life!
On the other hand, if you actually think about working for minimum wage, it's mind boggling. Think about it - $6/hour, for 8 hours, gives you $50 a day. I spend nearly that much on food on many days! $50 a day, for your rent, health care, utilities, clothes, transportation, food, everything!? My god, what's the point of even going to work if you make so little. That's $1 for 10 minutes of work. You surely make more money begging. If you ever have any hope of making decent money, then it's foolish to work for minimum wage. eg. if you're a kid in college - don't work some stupid job, just get loans. You'll easily pay them off later, and it's better to just have that time to enjoy. If you don't understand this, think of the value of an hour of leisure. Maybe it's $20. Thus, if you earn $40/hour, you're effectively getting a +$20 value for that hour, since you're losing an hour of your life in exchange for the work you're doing. If you work for $10/hour, you are losing $10 each hour! That is, the value of your life that you are giving up for that hour is worth more than what they are paying you. If you really think your life is only worth $6/hour, that's very sad.
12-22-04In the last rant I wrote about optimization and life decisions. There are two related things I've been thinking about a lot recently.
Responsibility and impulsiveness. This is sort of an interesting and difficult life trade-off. On the one hand, if you are wise in your decisions, you can avoid regrets - note that this does not necessarilly mean you are just a cautious stick in the mud, you still might make decisions like running away with the Circus or doing drugs or whatever, it's just that you actually thought about it, you saw the consequences, and decided that was the good thing to do. To be bold & thoughtful is difficult, but it is in some ways great. One problem with this is that making wise decisions takes time, and by taking that time, you will miss many opportunities, especially in social situations where split-second reactions are important. On the other hand, if you are impulsive and don't consider your actions carefully, you are liberated in a way, free from considering consequences, and you can do a lot more fun wild things; the disadvantage is you will wind up doing a lot of things that cause you trouble and pain, and of course you risk a small chance of very bad things like injury or jail. Finding a good balance here is almost impossible, because once you start considering what the right balance is, you've fallen into the careful/thoughtful camp. Of course, what many people do is behave very carefully most of the time, and then get themselves drunk to free themselves from that thought and allow themselves to be impulsive. This is of course very silly - if impulsiveness is good and fun, then it's good all the time, and you should do it when sober too.
The other thing I've been thinking about a lot is the way very small differences in the way you weight various outcomes in decision making lead to very large differences in behavior. Let's imagine you're making decisions as I described previously, by considering the choices and generating an EV based on your perceptual rating of the value of the various outcomes. If we were all perfectly smart, we would still make different choices, because the way we rate the outcomes is different. As a simple example, you have people who generally find confrontation to be very unpleasant. So, any outcome involving confrontation they would rate very negative, and that biases their whole decision making to avoid those situations. Every normal person enjoys good times and doesn't like bad times, but even small differences in how you rate them lead to very large behavior variation. For example, someone might really enjoy the good times (as opposed to neutral times), and not care too much about bad times - this person is more likely to make decisions that have a chance of going very good or very bad, such as running off to Vegas with a stranger; someone else might not rate the good times very high, but rate the bad times very low - this person will have totally different behavior, making decisions based primarily on avoiding the bad times, so they're more likely to just stay home all the time and watch TV.
12-16-04Every day in life, with almost every action, I consider mathematical optimization of the outcomes. Of course mathematics is the basis of everything - math is just abstract expression and formalism for dealing with things. The most obvious cases are if you're making decisions about investing - how do you maximize your return? But that's even getting complicated, because in life your overall goal is maximizing your happiness (with uneven weighting factors - eg. it's not a pure EV, but more on that later). Of course you can optimize with things like route planning on your drive to work; if you turn one way or another, what's the expected time? You have to consider the probability of hitting red lights, the probability of hitting traffic; eg. one path might take roughly 15 minutes always, another way might be 10 minutes 75% of the time, but it crosses a train track, so it takes 30 minutes 25% of the time, so the average time there is also 15 minutes. But now we also remember that our overall goal is optimizing happiness, and that's not a straight weight. Let's say the 15 minute path is the baseline, so that's the 0 happiness point. If you go the other way and get a 10 minute trip, that might be a +1 happiness score (that sets the scaling factor). If you go that way and hit traffic and get a 30 minute trip, that might give you a -5 happiness (the value here depends on your exact personality - how much do you enjoy being in your car? how angry does traffic make you? how much do you need those 15 minutes on that day?). One thing I always think about is the idea of pipelining, latency, parallelization. These are software optimization ideas, but anyone who's good at time management (like a chef) thinks about them. Let's say I want to make a meal - how long will it take? Well, if I have lots of people to help, then the limitting factor is the longest serial path - eg. any things that can be done simultaneously without dependencies, I split out to a lot of people (of course then there's also the overhead of the splitting and bringing back together). Even the longest serial path may have long steps that are low bandwidth (eg. not much work) but high latency (won't be done for a long time), like waiting for something to bake. During that time I can divert my processor (myself) to other tasks without slowing down the longest serial path. This stuff is all sort of very obvious, but hardly anyone thinks about these things and uses them in their daily life. When I wake up in the morning - I need to get to work as quickly as possible and still do some things. First I start the water boiling, then that has long latency, so I prep everything for the coffee, get my toast out (but don't push it down), by that time the water has boiled so I start dripping the coffee; while it's dripping I can usually run around and get my brief case together and ready to go, then push down the toast (again, latency) so it will finish about the same time as the coffee; start sipping the coffee and check the morning emails (I turned on the computer earlier so it would be all ready to go when I'm ready for it); deal with the mails, then hop over to the shower, turn it on so it starts warming (more latency), and while that's going assemble my clothes and put the brief case next to them; hop in the shower, throw on the clothes and run out the door. How can you be in a rush and yet sit around and wait for long latency operations to finish? There are two things that makes all this optimization even more interesting in real life. First of all, there's the non-even weighting. The exact rating of various possibilities depends on your personality. For example, let's say you have one job offer for $100k/year. You have another job offer for $300k/year, but it's all in stock options, so there's a 50% chance the company goes under and it's worth nothing. Now, in the second case, your EV is $150k/year, so with straight rating, that's the obvious choice. For most people, the first choice is better, because the rating for income is very uneven - eg. for most people it's very important to make at least $30k , then pretty important to make $50k, then each dollar after that is less important - eg. it's something like a log scale. However, if you don't mind being broke, then maybe the second choice is right for you. The other thing that really makes this optimization interesting in real life is that the process of doing the optimization is part of your life, and your overall goal is to maximize your happiness. For example, things like picking up pennies are generally bad for you financially if you make much money at all (the time spent is not worth the money made), but if you enjoy picking up pennies, then by all means, do it. If you really don't enjoy thinking about optimization, then it may be optimal for you to make poor decisions in an absolute sense, because you improve your happiness by not thinking about the decisions. For me personally, I usually enjoy doing correct optimization, but I like to be able to do things like go out on the town and just not think about it. Note that this is seperate from counting your contemplation time as part of the optimization problem - eg. you could just buy a video card at best buy for $300 , or you could probably find it online for $200 , but the time wasted searching for the best deal and going through the ordering, etc. is not worth the savings (or even if it's still a win, it's a much smaller win that it seems on paper). That's just part of what you should consider when optimizing, and actually if you consider that and decide not to do further contemplation, you have in fact optimized.
12-15-04Finally watched "Wag the Dog" last night. Kind of a crappy movie; DeNiro is really a rather bad actor, and Anne Heche is horrendous. As political satire, it's rather thin. The "dirty tricks" that they use to cover up the sex scandal are really very tame in comparison to the real things done by Nixon's crew, and more recently by Rove et.al. Sort of as a funny historical side bar - you remember when Clinton was in the shit over the whole Lewinski thing, and he ordered those missile strikes at a supposed "Al Qaeda" weapons lab in Africa? At the time no one knew anything about Al Qaeda, and many people accused Clinton of trying a "Wag the Dog" distraction technique. Turns out, Clinton was actually pursuing legitimate and important terrorism deterence, which our wise Bush derided and stopped doing when he took office. Anyhoo, one thing really bugged me in Wag the Dog - that stupid quote at the beginning. It's something like - "Why does a dog wag it's tail? Because it's smarter. If the tail were smarter, the tail would wag the dog". Wow, aside from being trite and pedantic, that's just wrong. The dog wags the tail because it's bigger, brains have nothing to do with it. The side that's bigger and stronger always controls the side that's smaller and weaker.
12-09-04I find myself more and more boring every day. I say things that are pretty reasonable, and reasonable things are bland. It's much more fun to say things that are extreme, ridiculous, not true, but controversial. I used to say offensive extreme things, that I basically believe to be true - like Republicans are corrupt, cheap people are fools, the Bible is fiction, etc. etc. Now, I find that lots of people that I really like would be offended by those comments, so I don't make them. I used to think that if you were dumb enough to be offended by something stupid like that, then fuck you; now I know that good people are rare, and they usually have many flaws, and driving them away does no good. So, I find myself unable to say anything interesting. I'm also crippled by having thought about almost every subject extensively; I don't mean to brag, it's just that I spend every waking moment in thought on all sorts of subjects, so almost anything that someone says to me, I've already thought of. It makes it really tough to have a conversation with someone. Of course, in those cases that I meet someone who's really done some deep thinking on issues that I haven't mastered, like at Game-Tech recently, I crave their wisdom, I want to drink deeply from the fountain of their thoughts, like a man who's been lost in the desert these many months.
Being a geek is really stupid. Did you hang out with other geeks in high school, play computer games and D&D ? Yeah, maybe we went to good colleges, maybe we have good jobs now, are we happy? We like to pretend we're so smart and superior to the "jocks", but they were out there living life, having fun - the whole point of life is to maximize your total enjoyment - we have failed miserably. Being a geek is really dumb.
12-07-04Another process note about code development - we try to keep everyone in the company basically on the main line in perforce at all times. Letting people go off on branches (or even just not sync for a while), or hold modified files open - all of these have caused us huge problems, because someone will report a problem, and you don't know if that problem is something weird on their box or a real problem on the main line. We also try to make everyone in content wipe their xbox and sync to perforce daily. Before we did these we would have tons of problems where someone would report a bug, we'd have to go investigate it - and hey, they just didn't sync, or they had a bunch of files open for edit, etc.
Coders have a bad habit of holding big change lists open as work in progress for a long time. I try to kill that - check it in, or revert it. Either it goes in the main line or it doesn't belong on your system. We also try to get people to check in often - once a feature is working or crash-free, check it in, even if it's not done, then keep checking in the little improvements. Also, we fire builds after each check-in. If there are problems, I want to find them right away, not the next day when everyone syncs. As always, the goal is to isolate the problem - if one guy checks in, fires a build, uh-oh, it failed, he can fix it real quick before too many coders sync to the broken build. Another little thing that helps here is whenever you are adding files or changing the project, go ahead and add empty files to p4 and put them in the project, and check that in so the project changes are done. Then check the files back out and work on them just as edits.
12-07-04I enjoy something about getting in shape the old school way - pushups, pullups, situps, running, medecine balls, running through tires. Avoid the gym, avoid pilates and yoga and machines, all that fru-fru crap, just go sweat and move your own body around.
12-06-04I basically never blame the development team for their failures or mistakes. If a team was not directed to do the right thing, how can you blame them for doing the wrong thing? If there were not policies and procedures in place for smooth execution, how can you expect them to work smoothly? 99% of these breakdowns are from management and leads - if there isn't good direction and procedure, it IS their fault. Yes, it would be nice if the team members would take care of their shit even if they're not directed to, but that's above and beyond the call of duty - yes, it makes my life much easier when guys take care of themselves, but I don't expect that. Sure, sometimes people are told to do the right thing and still don't do it, but even then that's often management's fault, because they're pulled in different directions by different people, they're distracted, etc.
12-06-04I don't believe that good people do bad things. I believe bad people do bad things. Also, I don't believe that difficult circumstances are an excuse for bad behavior. Difficult circumstances are the test that reveal your true character. Anybody can be nice when everything's going well, it's when the shit hits the fan that you need to step up. Now, I don't expect everyone to be perfect all the time - it's about how the handle the shit. Certainly I have plenty of weaknesses myself; when I'm stressed I get very snippy and rude, sometimes I just can't handle a situation and I run away from it. Those behaviors suck, but I also try to get a hold of myself and go back to it and apologize after the fact. If the shit hits the fan, and you turn on the people who are trying to help you - that I can never forgive; if, in the moment of trial, you abandon your friends and screw your neighbor and save yourself, you are a bad person and I want nothing to do with you.
12-06-04A couple of followups on my talk at game-tech :
First and mainly, I think I made a mistake in presenting the reason for robustness and decoupling. Robustness is the idea that the game never goes down, even with horribly broken use, and decoupling is the idea that when someone horribly breaks one system, the other systems keep running so people can still work. These are not just important for small teams or with junior people, they're always important. We acheive these things primarily by making the engine and code resilient to even fatal errors. There are other ways to meet these goals, namely Branches (source-control branches, so people work in isolation until their work is stable, then they merge to the main tree), and Build Escrow (in which the coder's build first goes through test approval before going to the content team). Branches and Build Escrow are both good, but they only work once the game is reasonably well established. We were building the code base from scratch, and rapidly iterating on design trying to hit milestones and demos, so it was frequently important to be able to code up a feature in the morning, deliver it to design and have it done by evening. For larger teams, the ideas of robustness and decoupling are even more important. This is part of the idea of overall team productivity. If a programmer takes the time to make his code extra safe, maybe it takes him 5% or 10% more time. If he does not, and he checks in code that breaks the build for the entire company, he's costing 100% of the work of maybe 50 other people. That's a catastrophic loss of productivity, even if it only happens once every hundred days it's a disaster. Most games these days are "design limitted", that is, the things that really prevent you from shipping a great game on time are in design, so it's silly to save coder time and potentially cost a lot of designer time.
What are the disadvantages of a clean self-protecting C++ coding style? Well, the compile times are slightly slonger, but our build is around 10 minutes and Halo 2's is around 7 minutes, so the difference is not very big. Also, that has more to do with arranging your headers well, we're currently sloppy about that, we could do much better to hide implementations. Having more clean separation of modules and opaque interfaces around whole systems would help that immensely. A related problem is that things like smart pointers and classes used in scopes forces you to put some things in the headers; you can't have a pure C-style hidden interface without doing a lot of work with pimpls and such. A lot of people think it takes too much time to code this way - that's just not the case; once you get used to it, the additional time needed is totally negligible, and is more than made up for with the time savings of having nice templates and classes that automate so many operations for you. The only two real disadvantages that I know of are - 1) when you hire new people, you have to teach them your base classes and your system; our system forms a sort of meta-language on top of C++, which is mostly enforced through compiler errors, but there are some things that you just have to know to do; 2) the protections seem to make people lazy, both in code and content; the protections are supposed to be in addition to proper testing and good algorithms, etc, not instead of them.
A little thing - the hierarchical allocation parser can obviously be used on things other than memory size; you can do it on allocation count, you can do it on CPU usage, vert count, etc. It's a nice way to track anything and figure out where it's coming from. This is especially nice with a "long frame tracker". To do the long frame tracking, you just run the tracking stats every frame, and you reset them to zero between frames. Then, as soon as you see a frame that you consider long, eg. longer than 1/20th of a second for example, you log the stats for that frame.
12-04-04I was just away at the Game Tech conference, giving a talk on OSW. It was really interesting to see the talks on Halo 2 and Half-Life 2, mainly to see that they really aren't do anything we aren't. I look up to those games immensely, and I always imagine in the back of my head that maybe they have some amazing technology or process or something that is helping them be so superior. In fact, their tech/engine process is very similar to ours and the goals are mostly pretty similar, though there are some differences in philosophy. Mainly we at OW just can't trust anyone on the team to do things right if left to their own devices - we have to enforce a lot of structure and rules through the code, whereas Valve and Bungie can be more flexible, because they have the process and responsibility in other departments that makes that possible. The biggest differences between OW and those two H2's are - A) their teams are huge, and B) they have good process outside of code. We at OW have a big company - almost 50 people - but we only have about 20 in game production (!!), whereas Valve and Bungie both have over 50 people in actual production (and a fraction of the admin staff!). Also, it's not like the H2's are really these ivory towers - their tools still have a lot of problems; in Halo 2's case, they were way behind schedule and the game suffered badly, mainly due to game design, in HL2's case, they just took a really long time. Both of them really managed to make great games I think because the core direction was good - eg. the base mechanics are identical to the previous games, so everyone on the team knows how to do that and agrees on it, and then the core philosophy of game design was good - for HL2, it's fully interactive, immersive, for Halo 2, it's systems-based gameplay, etc. Compare that to us where our direction didn't really crystallize until a few months before shipping.
I had to come back from the talk a bit early because we're trying to go gold here (maybe today!). We had four crash bugs when I got back (!!). Only one of them was found by the lovely EA test, and three of them were found by us internally, despite the fact that we have basically no internal test department at all. Lovely. The four crashes were - 1) XACT seems to have a bug with auto release cues and a bad linked list walk, 2) we had a resource registration bug in our code that caused a null deref in a very rare case (only on the DVD build), 3) one crash in the granny "UUU" department, 4) another crash in granny, again UUU/paging related. Amazingly, we fixed them all in like one hour. The UUU bugs in particular have been with us for a long time, and we never had a repro, and in fact I thought maybe they were fixed, but suddenly they came back. In fact, the previous fix I thought I'd done was a total red herring. When I did the fix, I also put in lots of catching to detect if the error happened again. It turns out that my check/catch code was being hit, and the damn content team was seeing those errors reported and skipping past them without telling us or sending us the logs. This has been going on for months, so we could have easily fixed this bug long ago. In general, our whole company and content departments are extremely unaware of how their behavior affects us. In code, we have to start first and crunch hard to get the engine and tools going; then we all work hard, and the content guys slip and slip and finish way late, and then we in code have to continue crunching to finish up. We try to make the code super robust so they can work, we try to put in error checking to help them and to help us find errors and fix them early so we don't wind up with crash bugs at the last minute. It's extremely bad to be making these fixes so late - we've had weeks of test, and we're making these fixes now, so all that testing has not tested these fixes. Also, our producers at OW and EA have become totally irresponsible. They're obviously sick of working on the game and with each other, they just want to kick the game to manufacturing, they don't really care if it's tested right; we make these fixes, they just want to play through it once, call it good and send it out. So, anyhoo, we found these UUU crashes which were tricky little buggers - we've known there was a problem for months and could never find it. I think it was just having those couple of days away at Game-Tech that made it possible. When you work 6 or 7 days a week for months, your brain just gets fried, you can't think straight. Game-Tech was intense, but it was still a break from coding, so I came back with a bit of a clear mind and was able to see the problem. Getting that time away is so valuable for intellectual labor, it's hard to quantify what a productivity boost you get.
- ► 2012 (134)
- ► 2011 (237)
- ► 2010 (311)
- ► 2009 (362)
- ► 2008 (697)
- ► 2007 (394)
- ► 2006 (654)
- ► 2005 (696)
- ▼ December (14)
- ► 2003 (74)