11/30/2006

11-30-06 - 2

Computer NFL picks this weekend :

New Orleans San Francisco 7 ; bet NO

Washington Atlanta 1.5 ; bet WA

Cleveland Kansas City -5.5 ; bet CL

Denver Seattle 3 ; bet Den

Philadelphia Carolina -3 ; bet Phi

The last two games are messed up by weird injuries/substitutions so I'm laying off them.

11-30-06 - 1

The problem with poker as a profitable gamble is that you can't just up your bet when you find an edge. You find a good table or you're beating a certain limit - you can't just throw your bankroll at it, because to gamble bigger in poker you have to move up in levels, which is a harder game. Roughly the difficulty is proportional to the level you play, which makes it hard to roll big.

In sports betting, you can generally go big when you find an edge. (that's not completely true because books will identify you as a sharp and start giving you bad lines or setting limits, but I'll ignore that for now). The problem with sports is that you get very few gambling events and they're very high variance. If you're a good capper you win 55% or so of your bets. That's good EV, but in the NFL you only get maybe 30 bets a year. The variance on that is huge. In contrast, in poker you will often be getting 55% shots, if you play a lot you might get 10,000 a year, which means your profit is much more regular. Even if you go to something like the NBA where you can get your money on more games, it's still only a few 55%'s a day, whereas in poker you'll get maybe 100 a day.

The third option are the financial markets. There's tons and tons of opportunities to trade so you can smooth out your variance. The return generally doesn't decrease with investment (for my money scales, it does if you're getting into huge monies). So, that's the best of both worlds. The only problem with finance is A) high transaction costs, and B) the markets are much more efficient so the edges are smaller.

11/28/2006

11-28-06 - 2

The computer picked GB to beat the spread on MNF and we just barely made it. I should've trusted my own analysis of Chi-NE and layed off the computer bet. It seems like the computer is solid, but it misses the intangibles of matchups, so it's worth using my own judgement to lay off sometimes. I'm not using my old system of really researching and looking for good matchups which I used back in 2000 or whatever, I don't want to put in all that time right now, so I'm seeing if a computer-aided picker can do well.

I'm going to write myself my guidelines. My computer system guesses the spread that it thinks is the "true spread". I want to decide to bet based on the position of the TS and the VL (Vegas Line).

1. If the TS and VL are on opposite sides of the zero, I like betting the Vegas dog to win. I especially like it if it can mean betting an underdog on the moneyline where I get +150 or some nice odds like that.

2. If the TS is close to zero and the VL is big, eg. if the TS is like 1 and VL is like 4, I again might bet the dog to win if my judgement of the intangibles is that they have a good shot. If not I drop down to the next rule :

3. If the TS/VL are on opposite sides of the 3 or 7 with at least 2 points of difference, bet the spread. eg. if the VL is a 9 and the TS is a 6, bet the spread. Teams will often win by exactly 3 or 7, so when you can bet across the 3 or 7 you're getting good value from the spread. On the other hand if the VL is a 6.5 and the TS is a 4, probably don't bet unless the intangibles really like the bet because you're not crossing a 3 or 7.

4. To make the intangibles a bit more concrete : A) Prefer betting home dogs. B) Prefer betting teams that "need a win" vs. teams that are either coasting to the playoffs are completely out of it. C) Beware of teams whose past performance is not a good indicator of their true quality, eg. if you judge they've gotten lucky in a lot of games. D) The computer isn't aware of news or injuries so mainly just avoid games where those are big factors.

Update : this week there seems to be a huge edge betting the Ravens to win vs. Cincinatti.

11-28-06 - 1

We did a sort of mini "Indie Game Jam" at Chris's over Thanksgiving. The theme was "Sound" - sound for input, output, gameplay, whatever. The engine was actually a pretty nice sound in/out/mixer and hopefully I'll be using it for some projects in the future; as usual everything will be released open source.

I had a lot of ideas for games, but didn't really like any of them. A few were sort of decent and obvious, like :

1. Make a game where your play creates a song. Basically different user actions or environmental objects create certain sounds or turn on different tracks. By playing the game correctly you make a cool song. The true "synesthesia" moment of brilliance would come if you could play either by just doing the visual gameplay things, OR purely by closing your eyes and trying to create a cool song, and if you create a cool song that also beats the shooter/platformer/whatever.

2. Make a game with sound input where doing the gameplay causes you to make sounds that are cool and/or feel like the play. The way I was thinking was similar to #1, for example you could make a "beat boxing platformer" where different beats make you jump or duck or whatever, then the level sends things at you and you wind up making a sound track by playing. Casey had a really cool idea for a variant on #2 but I won't out it here, hopefully he'll actually finish the game and it will be cool to play.

Anyway, maybe I shoulda done something like those, but they just felt like the same old game. Most video games feel like a series of rote/mindless tasks that are presented to the user in sequence. I'm sick of making those and playing those, and this was really just another of those in different garb.

So, I tried to play with some toys that would feel like different experiences, and mostly failed. On the plus side, now I know my arm is okay to type again, though I was irritating it quite a bit.

11/21/2006

11-21-06 - 1

In the past 10-15 years, the investment philosophy of following the pack and jumping on bubbles has been incredibly profitable, assuming you got out somewhere near the peak. We had the dot com bubble, real estate, oil, gold, etc. People who have profited from these are often people who know nothing of smart investing and are cocky bastards. It's easy to dismiss them and these investments, but maybe there really is a valid strategy here. In this age of get-rich-quick day traders and soccer mom investment clubs, the markets are very swingy and people tend to pile on gaining sectors which pushes them up. Some investments might be very profitable even though you know they are over-valued; as long as enough of the populace is driving them up, you can ride the wave up and sell out.

In this theory, you could even make big money from those stocks that you get in spam emails. Assuming you're one of the first buyers, even if it's a junk stock people will send it up and you can ride the bubble. Certainly you can make money by buying a penny stock and then talking it up all over the internet, which is essentially what major banks have been doing for years.

11/20/2006

11-20-06 - 1

Computer NFL picks :

Chicago +3 against NE
NO +3 against Atlanta

bet both to win at about +150. Personally I hate betting Chicago against NE but this is what the computer likes.

11/19/2006

11-19-06 - 1

I put a post on my wordpress blog for people to comment and hook up about GB : here

11/18/2006

11-18-06 - 5

If you like capitalism and self-interest and such you should love the populist movement in Venezuela. Often populism is cast as "liberal" and socialist, anti-free-market. In fact the opposite is true, populism just means the people are voting in their own best interest, which is what they should do in an Ayn Rand idealogy. For far too long people in American have voted very much against their own best interest, which wickedly distorts the action of the state. One of the candidates in Venezuela is offering direct money back from their oil income; yeah, that's a cheap move and that can revolve into a self-destructive cycle of pandering for votes as well, but at least the politicians are at least trying to win votes by appealing directly to the people, instead of these purely image-based campaigns of lies and no benefits.

11-18-06 - 4

In the long view, one of the really disastrous and disturbing aspects of the Bush presidency has been the systematic destruction of the executive agencies of our government. Almost across the board, leaders were appointed based on political favors and idealogy, people who were incompetent or unqualified, people who didn't work with their department or their beaurocracy, people who caused severe damage to the agencies. The agency leaders are obvious - people like Rumsfeld who is obviously intellegent but harmfully ignored his own people and created nasty infighting over power, obviously at FEMA and Homeland Security, we talked a big game about making the US safe but appointed a bumbling figurehead, at PBS & the EPA Bush appointed people who were intentionally trying to hurt the implementation of the laws that govern those agencies, at the FDA the appointed board acted repeatedly on ideological or industry-favoring grounds against their scientific recommendations, in the few departments that had qualified leaders (State & CIA) they were cut out of the loop because they didn't fall into step.

The fact is, democrat or republican, the people who are elected are generally totally unqualified to run the government. They usually have little experience (especially in this age when any record is a bad record). Even with experience government is a huge complex problem and you need long-term specialists, the beaurocrats who run the agencies and really make government work. Those people have been devalued and cast aside, ignored, and have resigned in droves. This has already been detrimental to the functioning of government, but if it becomes a pattern it could be even worse.

11-18-06 - 3

Computer's NFL picks :

Cleveland +4 vs. Pitt
San Francisco +4 vs. Seattle

This is using a system where I only bet when the computer-predicted spread is across the 7,3, or 0 from the betting line.

11-18-06 - 2

The way everyone talks about stock and pricing past performance is completely wrong. The NYT says housing prices from 1950 to 2000 have gone up 3.5% per year in San Francisco (inflation adjusted). That sounds like you're getting a 50 year average and that should be a pretty stable number. In fact it's not at all. People do the same thing with stocks, they look at the average rate of return over an X year period. The problem with this is that it's incredible sensitive to exactly when you pick your start & ending point, which means you're incorporating a huge random factor.

Imagine your value(time) function was a sine function. The correct average rate of return is 0. However, depending on where you put your start and end, it could be largely positive or negative. In particular, with SF housing, if you looked at 1950 to 1998 you would see maybe a 2% annual gain. If you looked at 1950 to 2005 you would see a 5% annual gain. The problem is that the time span we're looking at is not very large compared to the magnitude of the noise. Take the ending price minus the starting price is super sensitive to the short term noise.

A much better way is to best-fit a line through the prices and take the slope of the line. This is way less sensitive to the edge effects. There are probably even better statistical ways, but it seems nobody does this even though it's totally obvious and everyone should do this.

11-18-06 - 1

I recommend the "Mission Park" Rhone at T.J.'s. Spicy and balanced, light finish, about $5.

Watched "Miller's Crossing" last night. Good but rather pointless. Has that Coen brother unrealistic staged feel; their dialogue is really awful in that everyone is a charicature and noone ever says things like that, but their dialogue is really awesome in that the lines are clever and perfect for their surreal noir world. Anyway, that's not my point. The Gabriel Byrne character made me think about the way smart bad ass guys always are in movies - they think of a plan and then go with it. What they don't do is think of several plans, think of the advantages and disadvantages of each, compare those to inaction, think of how it affects other people, etc. get some advice from others, then make a decision. They just come up with something make a decision and then figure out how to make it work. I know it's movies and it's ridiculous but I think it's a major problem in the way I run my life. The truth is you can make any decent plan work, so just pick one and do it.

11/17/2006

11-17-06 - 1

I just watched a debate on "Newshour" about whether the current housing slump is going to lead to a recession or a similar major problem. The guy on the recession side made lots of valid points and sounded reasonable, the guy on the "it's fine" side sounded like a dot com optimist, full of nonsense and rationalization. That's a very bad sign. I'm now pretty sure the US economy is headed for the shitter (unless we stumble on another piece of big luck similar to the internet boom, like if somehow all Linux software self-destructs and China has to buy into Windows, or maybe some US company invents and patents a teleporter or some other ridiculous market-changing invention).

The housing cheer-leaders say that spending is strong, job creation is strong, the US economy is flexible, and also that this isn't a housing bubble, it's a valid correction based on high demand for limited space. First of all, the valid pricing idea is nonsense. We haven't suddenly run out of land. Housing prices haven't changed significantly in the last 50 years and suddenly there's a huge shortage? Every time in the past that housing has shot up it's crashed back down to almost zero change inflation-adjusted.

The reality is that consumer spending has been powered by housing (the saving rate has been negative indicating people are taking loans against their houses). 33% of job creation since 2001 has been in housing or related fields, and perhaps even more so due to the trickle-out effects. Even despite that real spending power has decreased and we haven't actually cut unemployment. More and more people are not even showing up on the unemployment numbers because they're not seeking employment. Furthermore, our government has been desperately doing everything it can to stimulate the economy - interest rates have been rock bottom low, we severely cut taxes, both of which should have been huge short-term boosts to the economy, and have only just kept us at a sustaining growth rate (our GNP has to grow several % a year just to match population growth). The government's been doing massive debt spending, which again should stimulate short term growth and generally leads to constrictions later on.

We're now in a position where we can't really do too much more for stimulation. If housing does start to lead us to recession, our vault of reserve measures is depleted, we can't keep lowering interest rates, spending on debt, cutting taxes.

Anyway, as I've written before I'm still not sure how to play this. I do have a lot of money overseas already but maybe I should move even more.

11/16/2006

11-16-06 - 2

Dan made Monkfish last night. It really is a lot like lobster. Monkfish is a cartilage fish and only the tail is edible, it looks like some weird evolutionary left-over. You have to be very careful to not overcook it. I recommend a simple preparation treating it a lot like a steak - sear it first, then remove from the pan and saute onions, mushrooms, and fennel in the pan, add a bit of wine and stock, reduce a bit, put the monkfish back and just simmer until done, which is *before* the flake point. It becomes rubbery and non-lobster-like if overcooked at all.

We also got some Muscat grapes at farmer's market. They're quite amazing, very sweet, they taste just like the wine! Everyone was going crazy over all the persimmons, but I just don't understant it, I find them quite awful. I had a persimmon tree all those years at Alrita and didn't even bother to eat them off the tree.

My arm is getting much better, but I seem to have caught another cold !?!? WTF. Dan seems to get them at work and bring them home :( So I'm in awful shape again. Sigh.

In other news, I got a wicked Trojan by clicking around porn sites. WTF. It makes me angry how unbelievably broken IE and Windows is. It shouldn't be so easy to run malicious apps. Anyway, System Restore seems to have gotten rid of it pretty easily. All of my many Anti-Virus apps didnt detect it. It was a DLL in windows\system that inserted itself in winlogon. If you edited the registry to take it out of winlogon it would put itself back in.

I put some money on some football games partly out of boredom. I couldn't resist betting the Colts at -1 even though that's the "public" (sucker) bet and all the sharps are on the Cowboys. I believe Romo is probably over-rated because we just haven't seen him tested yet and everyone thinks he's amazing, and Parcells has really made awful decisions of late. If the Cowboys just run the ball and keep it low scoring, they will win. If it's a shoot-out, the Colts win.

11-16-06 - 1

All this mobilizing, canvasing, marching, all this shit is the wrong way to advance a cause. It can create some short term response but doesn't create long term powerful change. All major political shifts happen in response to massive world events - economic depressions, military coups, wars, etc. - this has created the big socialist movements. What created the Republican shift of 1980-2000 ? Partly it goes back to LBJ and Vietnam, the cold war of course, the oil embargo, and some arrogance of the coastal liberals, the job loss and fall of the middle of the country. How then do you create real change? You create the events that cause people to really react. Just scandals and horrible dishonest crooked behavior don't do it any more. Of course people will just swing over to the other party. To get any improvement, something else is needed. They need to see that the system is totally corrupt and needs to be changed.

"Crooklyn" is a god awful movie. If you just keep playing clips of songs from the 70's will it make it fun to watch? Yeah, it sort of does, but the complete lack of non-superficial story and characters is too bad for all the cool style and "authenticity" (what?) to cover. Anyway, the first 2 custom made "Crooklyn" hip hop tracks are hot yo . Somebody find me the mp3's.

11/13/2006

11-13-06 - 2

I've been really disappointed by the movies I've rented recently. Damn Netflix won't give me my history but some have been Ararat, Spanking the Monkey, Dressed to Kill, Old Boy, Ocean's 11, Happenstance, ... all have been really boring and pointless, the only highlights being a few erections. Even the brutal violence and shocks of Old Boy just felt gratuitous and inhuman. I've been trying to think of the movies that I actually really like so that I can rent ones that I actually want.

11-13-06 - 1

The stupid American government system has this property that if you can ever get 51% of the vote, you can do incredibly extreme thing that piss off the other 49%. You can also very easily stay in power by drawing district lines, giving favors/contracts to big donors, using your office for PR, etc. In a reasonable system you would have representation proportional to the backing of the people, instead we have a very fucked up threshold system where 51%->100%. In the US system, things seem to work better if power is split by the two parties, eg. one holds the presidency and the other holds congress, but that fact is simply an indication that the system is badly broken and too boolean.

11/12/2006

11-12-06 - 1

The killing and beating of nonviolent protestors in Oaxaca has gone almost completely unreported in mainstream US press. Partly because of the smoke screen of elections but also because the powers in the US dont like to report on the corruption and violation of civil rights by the government of Mexico.

Similarly the Beit Hanun/Hanoon Accident/Massacre is a huge big deal in Israel and the Arab world but has hardly received any attention here.

11/10/2006

11-10-06 - 1

Daniel Negreanu uses the "astrologer's fallacy". He calls out people's hands semi-randomly based on good guesses just like anyone would. Each time has maybe a 1/4 or 1/10 chance of being correct. When he's wrong you just forget it, but when he's right it's very memorable and seems amazing. You wind up remembering that he seems to name people's hands very often. Of course TV helps him because they show hands where he gets it right.

11/09/2006

11-09-06 - 4

It's sort of odd that there's still no really good image compressors. I mean "good" like PAQ or RK are good on text/exe data - really fucking good. In comparison the image compressors are super primitive. I guess the big problem is that for an image format to be useful at all it has to be supported in IE, Photoshop, etc etc. and since Windoze is such a broken piece of garbage there's no modular data abstraction layer, every app implements its own set of codecs. In a proper OS apps would just open an image and a codec library would handle it.

For lossy coding, the literature is quite strong and J2K would be fine, but I guess it has patent problems cuz the design committee was full of cock-munchers where each company forced in bits that they had patented. For lossless image coding there really isn't even super strong literature. The "MRP" method seems okay, but it's incredibly obvious how to make a big gain in practice : use Volf switching or something like that to blend in a dictionary/text coder which will work well on binary/artificial images.

11-09-06 - 3

I'm gettin sick of comedies that are just about how miserable and awful human beings are. Unfortunately the best comedies of this era are in that category. Some key examples are The Office and I'm Alan Partridge, two of the best comedies in the decade. To some extent Curb is in this group, though it's a bit more situational/whimsical. Contrast to previous eras, like Monty Python which started a wave of pure Absurdism carried on now by Family Guy, or of course eras when things like puns, double entendres and clever turns of phrase ruled. This modern "humanity is rotten" comedy is sort of taking the idea at laughing at our own faults to an extreme that's quite unpleasant.

11-09-06 - 2

Half the reviewers on Yelp need to get punched in the face. In fact in general we'd be much better off if computers and phones and cars could taser the user. Look, if you're a vegetarian and you went to a place called "Joe's House of Meat and all we got is fucking Meat" it's your own damn fault, and don't fucking give them a low rating. Also if you went to a restaurant drunk and thought the greasy food hit the spot, you probably shouldn't write a review. Finally if you go to a tacqueria and order a Carne Asada taco and complain that all you got was some dry beef in a tortilla, that's what you fucking ordered you stupid gringo!! You have all had your internet priviledges suspended. You will now wander aimlessly without the web to guide you and accidentally wind up in a real dive bar, not an ironic one, and get your head bashed in by real blue collar guys who don't find mocking GWBush at all funny.

11-09-06 - 1

Part of the problem of course with pointless/counterproductive wars like Iraq is that it makes it impossible to exercise intervention when its necessary, as it might be in Iran, or NK. It is impossible, not just logistically, but politically, which will last 10-20 years. Not that I think intervention is a good idea, but having the option & credible threat is valuable. And intervention might have been necessary in a situation like Lebanon.

On a related note, I'm sickened by the stupid fucking American electorate. They vote D because the war is going badly. WTF you morans. If the war happened to go well and the economy happened to swing up they would've voted R. You go all in with 23o preflop and happen to win against AA, most americans look at that result and decide you made a good play and get to wager their money.

11/08/2006

11-08-06 - 3

TiVo dropped a software update on me which has done some awful thing where it now collapses episodes of the same show in a folder. This is yet another instance of their software sins. First of all, the modern phenomenon of self-updating software is god awful and I wish it would end, though I know it won't. Software doesn't get old or wear out, there is no particular advantage to getting newer versions, it should be up to the user. Second, they have broken one of my basic design rules : don't change something in your UI unless there is a significant benefit. Users get familiar with the UI. This familiarity has a huge amount of value. In order for a change to be good it must have greater value than the familiarity. Making small or neutral changes is extremely foolish, and yet major software packages do it all the time. Photoshop seems to completely reorganize its menus with each version and even renames common operations. Visual Studio does that and also changes te default keyboard shortcuts!? wtf. I feel tempted to write a manual on software design, but plenty exist already and the fact is that people making these design errors wouldn't read it anyway.

11-08-06 - 2

How to do image enlargement with wavelets :

Do a float wavelet transform with the lowest-entropy wavelet (D97,CDF22,Haar,whatever). The imagine is now an LL and a series of bands, HL1,HL2,...HH1,...

Fit the linear relationship of bands : HLn ~= C * HLn-1. Fit the best C for all n. Seperate constant for HH and LH.

Use this linear relationship to add new LH,HL,HH bands with the scaled data. Perhaps also apply an additional user-specified scale. Do the inverse transform. The resulting image is 2X bigger in each dimension.

This will do things like zoom hard diagonal lines perfectly due to the wavelet zoom scale invariance property.

ps. this doesn't actually work because Band (n) to (n-1) correlation is very strong for coefficient magnitude, but not for signs, which are hard to predict.

11-08-06 - 1

OMG I'm starting to get really depressed. I can't fucking do anything, partly because I got the flu on top of the shoulder. Even computer use is so frustrating. I'm so fucking bored and sick of TV. Fortunately it shouldn't be too much longer. It seems like in a week or so I'll at least be typing and walking around again. Making it all worse the flu makes alcohol unpleasant so I can't just drink my days away, all the TV gives me a headache, and Dan is getting annoyed with me. I think if I ever had a severe injury where I was immobilized I would try to kill myself. Of course the worst thing would be something like "The Sea Inside" where you cant even kill yourself anymore.

11/07/2006

11-07-06 - 1

One interesting thing to me about the 2+2 poker community is seeing what they do outside of poker. A lot of the people are involved in money making schemes. Lately of course this is often real estate, but the small time operators do a lot of funny things. They scalp tickets to big concerts, they do things like buy up PS3 pre-orders and resell them on ebay, they go to salvage + police auctions to buy things and resell them, some of the guys own & stock soda machines or ATMs. Some of them make quite a lot of money doing those things.

11/04/2006

11-05-06 - 1

I'm gonna try to write a cripple's summary of ML. In all cases I care about supervised learning. That is, given a big sample of parameters x and ouputs y you have the training set {y_i,X_i} where X is a vector. The goal is to find the function f() which minimizes the error |y_i - f(x_i)| over all samples. This is a regression problem but you can easily convert it into a "classification" problem or use a different error metric or whatever.

Searching for arbitrary f() is NP hard. In fact, you can see that finding an optimal f() is equivalent to the Kolmogorov compressor. If we use an "MDL" error we think of f() as a coder; to reproduce the output y_i we have to transmit the specification of f() and also the residuals (y_i - f(x_i)).

For all the approaches we will be approximating f() as a combination of simple functions. You can of course get arbitrarily close to any f() by using a linear sum of enough functions. One thing I've come to realize is that all the ML techniques are very very similar. Almost all of them have a form like this :

f(X) = Sum[j to M] w_j * h( A_j * (X - L_j) )

Where h() is some function, "w" are weights, "A" are parameters, and "L" is a location in the same space as X. There are M terms. Possibly there are other transforms or linear combos or whatever, those issues are sort of minor (but messy).

Anyway, all the schemes come down to how you pick these terms and optimize them. Almost always you pick your basis functions h() in advance. This could be your Kernel, your fit function, whatever. Common choices are the sigmoid or radial gaussian. Obviously you could just put a delta function at each sample, but that would give you a large model that would not generalize well. So you sort of want that smallest/smoothest model that fits the training set. This is a very hacky problem with ML and you really just have to tweak and test by cross-training.

Neural Nets are a silly way of talking about basic linear algebra. You choose M and h() in advance and then optimize the parameters by greedy gradient descent. Obviously you try to choose the smallest M possible to get a decent fit on the data, but you can only do this by trial & error with cross-training. Also the gradient descent can get stuck in local minima and you get no gaurantee of maximum margin or other ideas that the model will generalize well. The nice thing about neural nets is they're pretty simple and straightforward. Unfortunately NN's have been conflated with all this brain modeling and circuit diagram nonsense which makes it much harder to just see the linear algebra.

SVM's are actually extremely similar, but instead of greedy optimization, they set up a nice constrained optimmization which solves for maximum margin. The L's are not free parameters but are instead points chosen from the training set (called the support vectors). Instead of controlling M directly, you set a parameter which controls the model size vs. closeness of fit tradeoff, and the SVM solves for the best M. The h() can only be functions that work as "kernels" but that's not too bad, it's a large set.

There are some nasty problems with SVM's. The parameter that controls model size has to be tweaked by cross-training which is hacky and time consuming. For clean seperable data svms do an awesome job of picking a minimal model (eg. for linearly seperable data with a plain dot kernel they pick a single perfect plane), however for noisy or ill fit data, M can get arbitrary large (up to N the training set site), and the time to train and test scale like M*N which is very bad if M ~ N. Finally h() must be chosen in advance and many factors you might want to tweak are actually implicity hidden inside the choice of h() such as the relative scaling of parameters. Most simple Neural Nets are equivalent to SVM's, and the SVM will find better solutions if you can make it behave the way you want.

Another method is local regression. Here the L are chosen to be some regions, and then some sort of standard regression is done on that region, such as linear regression, or splines, or whatever. The regression is easy so the whole problem is the choice of regions. The standard ways are K-nearest-neighbors, K-means, and Decision Trees. These are all pretty solid yet all very hacky in their own way. The big problem is that just like kernels they implicitly rely on a distance measure between points. It's crucial for the similarity of samples to be proportional to distance. With lots of different complicated parameters this is ill defined; you can do thing like SVD or PCA on the paramets and then take distances in that space, but that actually may or may not be good depending on the data. In fact if you knew the ideal distance metric you would have the solution to the whole problem. DT's cut up space to define the regions. With Decision Trees most people just build axis-aligned trees and try all possible splits. This is slow and also ignores the fact that you could get much better splits by trying all possible off-axis splits. Ideally you would use an SVM to pick the optimal split at each level.

K nearest neighbors is pretty simple and solid, of all the methods it seems the least "tweaky" and requires the least a priori knowledge of the data, but in practice it is very sensitive to choice of K, choice of how you define "nearest", and you should probably use a distance-weighting and it will be sensitive to that. As usual these things can only be tested brute force by trial on cross-training.

The choice of distance metric or pre-transformation of the parameters is important, as is possible elimination of useless parameters (usually after SVD). Unfortunately this is basically a hacky trial & error thing and can have huge affects on the model's performance. Also with Neural Nets the exact pre-ops and post-ops you perform can make the problem either much easier or much harder to model which strongly affects performance. Basically if you know a lot about your data (like it basically responds linearly to parameters, or it's generated by independent sources on the parameters, etc.) you can fit it well, but if you don't you will do badly and there aren't good ways to "learn to learn".

Fully Bayesian approaches are much more sound, but are computationally infeasible for anything but trivial models because they basically sum over all possible models & parameters weighted by the likelihood. Basically rather than guessing the params or optimizing them, you give them probabilities and sum on all possibilities. Check out the NIPS tutorial here Basically you correctly consider the model+parameters to be unknown and use the probability that they are the true model.

In general the more modern techniques like SVM don't perform significantly better on complex data than just something like a good local regression. I should add that the truly modern techniques seem to be blend models which are provably good as the number of blends gets large, things like AdaBoost with ensembles of decision trees, etc. It appears to me that these are computationally infeasable except on trivial problems.

Way too much of the literature compares their method to some garbage old method, like plain old k-NN with no distance weighting or outlier discarding or local distance metric. The other big sin is to do comparisons in some really complex domain like image matching where a huge amount of performance comes from how you preprocess the data and take advantage of the specific nature of the problem.

11/03/2006

11-04-06 - 1

So I've read like four books and a hundred papers and now have a strong grasp of collaborative filtering and machine learning. Basically CF is garbage that was made by bad engineers who didn't understand ML. All the true strong modern approaches are more theoretically-sound ML approaches. I should add many people in CF know this and have been advocating standard ML techniques for a while. For example Pazzani has some good papers and one suggests using an SVD->ANN. I don't like the details of his approach, but thebasic idea of SVD->ANN is solid, and just one example of "data reducer"->ML.

Unfortunately even robust ML approaches like SVM are full of unsolved and unsolveable problems which you have to address through "tricks" and guesses and brute force optimization.

Anyway I've decided on my overall approach and believe I'm on the right track. Now I just need my arm to heal so I can type agian. I'm not even trying to code lefty, it's too frustrating and requires too much chording.

One nice online resource is http://www.learning-with-kernels.org/ ; it's a really good book and they put the best chapters online. Chris Burges and Radford Neal also have great tutorials online. I have yet to find really good Neural Net material, there's a ton of garbage on Nets and it's hard to sift through. The Wikipedia pages on all this stuff are good jumping off points, they have good intros then links out to more info.

11/02/2006

11-03-06 - 3

So, I'm getting pretty good at typing lefty. I'm still totally out of commision and going mad from boredom.

11-03-06 - 2

I looked for a secure text editor so I could easily keep my personal/password/financial files encrypted. There are a lot of them, but they all appear to have been written by morons - the keep the file encrypted on disk but the editor uses standard windows edit controls which are trivial to scrape. Even if they called DrawString that would be very easy to scrape. The only way for them to be hard to scrape is if they do their own text drawing, which of course would be easy enough. I bet all those password manager things use edit controls too.

11-03-06 - 1

I'm having a lot of fun learning about "machine learning" for Netflix. I'm working hard at it, and Dan asked a sage question : couldn't you get paid to do work like this? In my experience, no, you can't. You could maybe find a technical engineer job where you do this kind of learning for a few weeks, but then you have to implement it and deal with real world crap issues for the next year. Or you could be a professor where you do this kind of learning, but then you have to specialize and get into some arcane pointless detail; that wouldn't be so bad if you didnt have to spend half your time writing grant proposals or whatever. In theory there should be cool jobs at a place like MS Research, but I've heard tales that the reality of working there really sucks.

old rants