tag:blogger.com,1999:blog-5246987755651065286.post8682752063818413992..comments2024-02-22T16:15:42.388-08:00Comments on cbloom rants: 12-09-10 - Rank Lookup Errorcbloomhttp://www.blogger.com/profile/10714564834899413045noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-5246987755651065286.post-56984190465259163412010-12-10T22:32:44.786-08:002010-12-10T22:32:44.786-08:00"In general, wouldn't you ideally want to..."In general, wouldn't you ideally want to use a measure of the sequence's relative sortedness?"<br /><br />The point I've been trying to make is that that is not what I want to measure. Kendall and Spearman *are* measured of relative sortedness, and I'm arguing that since sorting is not what you care about, that it can give the wrong picture of which metric is best.<br /><br />It appears that "rank lookup mse" does in fact give ratings to the metrics which are much closer to the order of the rating I actually care about (which is the ability to fit the scores).cbloomhttps://www.blogger.com/profile/10714564834899413045noreply@blogger.comtag:blogger.com,1999:blog-5246987755651065286.post-56444095864533802992010-12-10T21:36:04.584-08:002010-12-10T21:36:04.584-08:00"Obviously this isn't quite ideal; it doe..."Obviously this isn't quite ideal; it does handle ties and near-ties pretty well though (better than Spearman/Kendall, because you get less error contribution when you get the rank wrong of two items with very similar value)."<br /><br />Squaring the error seems combinatorially perverse.<br /><br />In general, wouldn't you ideally want to use a measure of the sequence's relative sortedness? Classical choices include the inversion number, the crossing number, the number and length of runs, the minimum number of adjacent transpositions to sort the sequence, etc.<br /><br />These all reflect different aspects of relative sortedness you might want to capture. In your application, where it's just a basic sanity check, it probably doesn't matter.Per Vognsenhttps://www.blogger.com/profile/06042681400980641198noreply@blogger.com