(* I should note up front that I am not talking about "severe" off-roading, like boulder-climbing or some such, which no production car is suited for, I'm talking about un-graded dirt/rock roads and such; I'm also assuming you're not doing something retarded like going off-roading when it's muddy, which is not so much foolish because you might get stuck, but ass-holish because it destroys roads and erodes hillsides)
1. Reliability. By far the most important thing is to have a vehicle that won't break down in the middle of nowhere, where cell phones won't work and AAA won't come. Above all else this is why the Toyota Truck is the greatest off-roader of all time (and the Honda Civic is the #2 off-roader (not really, but the point is that a reliable car with worse capabilities is much better than a very capable car that could crap out at any time)). Shit like old Land Rovers or International Scouts or what have you are actually terrible off roaders due to high probability of crapping out.
2. Comfort. This is one that I never see mentioned, but in fact the limitting factor on most bad roads is just your comfort. Most bad mountain roads are not impassable - in fact you could probably make it in a Honda Civic or some such, but when they are rutted, wash-board, rocky, pot-holed, it will be back-breaking hell and you will have to go 5 mph in a Civic. To realistically be able to go 50 miles on a bad back road, the most important thing is comfort, you need long travel suspension and a very soft ride. (BTW longer travel is almost always better with suspension). This is actually a bit hard to find these days as everything has gotten "sporty". I'm not sure what the great choices are for this, since the retarded car journalist corps doesn't correctly review offroaders for comfort.
3. Clearance. Far more important than 4wd or any such nonsense is clearance. For fording streams, getting over logs, big rocks, the things that will actually stop you - clearance is king. Cars with good 4wd but terrible clearance are not really great offroaders (yes, that means you, subaru).
Traction and power almost never come in to it for real-world offroading. When it does, it's better to have a Honda Civic with a good winch than a 4x4 without one.