09-27-08 - 1

It actually sort of sucks that there are so many super tuned tweaked compressors out there. It's really stifling to innovation. It means if you try something new and interesting, it's almost gauranteed to look bad by comparison because it's not as refined. This is a general problem that I often see in development. Somebody has a stupid O(N^2) sort that they put in assembly and optimized for ages. You try something better like an O(N log N) sort, but the speed numbers in your test application are worse. Well, that's not fair, your new algorithm hasn't gotten all the same tweaking and refinement. But now if I want to compare a bunch of possible solutions, I have to waste a ton of time tweaking them all out. The very existance of that one super optimized possibility really fucks up my entire algorithm search.

The thing that makes this super extra hard is that different algorithms lend themselves to more tweaking. So you really can't just implement them all in a rough high-level fashion and know that you're picking the right thing. In compression for example, you might look at Block Sorting and PPM and say hey these are both very simple elegant high level algorithms, and they perform very comparably. But they are not equal. PPM lends itself to far more tweaking and refinement and tricks and specialization, while with block sorting you're stuck in a dead end where the general solution is all you can do. Of course you see the same thing in speed and optimization.

No comments:

old rants