1/27/2012

01-27-12 - Some Game Reviews

Mass Effect 2 - it's pretty good looking (mmm space camel toe), but this is not a game; it's like semi-interactive fiction or something. I just feel like nothing that I do (in the game, though also in life) actually matters; all the "challenges" are absurdly easy, and your hand is held. Then it goes into a long cinematic talky blah blah. This type of game is after my era and I really don't understand it. It's not good gameplay, and it's also just not a good movie; it's sort of like a bad game combined with a bad movie, and people seem to just love that. I'd rather just watch a good movie where I don't have to hold a controller in my hand and press A once in a rare while.

Skyrim - my god, I am surprised that this game is so widely praised. I get that there's definitely a niche market that wants to collect thirty snails so you can make dye so you can tie-dye shirts to sell in the town fair. But I'm shocked that that market is so large. Also, the general game mechanics are so fucking janky, it feels like Everquest or something built by amateurs on a free 3d engine. There's weird pointy collision geometry that you get stuck on, the graphics are full of artifacts, wtf.

Gran Turismo 5 - whoah this game is super disappointing. First of all the graphics are surprisingly terrible. How hard is it to render a race track? The basic stuff, the pavement, trees, etc. just don't look good and you wind up seeing big zoomed-in pixels all the time. The menu flow of the game is atrocious, and the load times are totally unforgivable. Typical game session goes like this - click though some damn movies, click through several fucking popup boxes about how I need to set the date & time and connect to the network, okay, finally the main menu (which takes a while to load); click into a sub-menu (loading...), again and again, like 5 or more menus before I get to race (each menu taking quite some time to load), and then *loading* ... it loads each fucking opponent car one by one and they take several seconds each, WTF. Every time you go into a menu it loads, and then just to back out to the previous menu it has to reload it! WTF WTF. But by far the most disappointing thing is the physics; I thought that with a wheel it would actually behave like a real car, and it just doesn't. Which brings me to -

Logitech Driving Force GT Wheel - GT5 was sucking so bad I thought I would try it with a wheel, so I bought this piece of crap. The force feedback is completely terrible; it's a nice idea, it should give you some of the feedback you get from a real steering wheel, which should give you information about if your tires are sliding or in the air or whatever. Nope, none of that. Instead you just get the motor seemingly randomly turning on and moving around the wheel really hard, like you have to fight it with fierce muscle flexing. It's garbage. So I found this trick that you can unplug the power from it (after it does its self-calibrating thing) and that disables the horrible motor jamming you up, and then it's sort of okay, obviously no feel or response in the wheel, but that's better than horribly wrong random force feedback.

Of course all that shittiness pales in comparison to the basic shittiness of the entire modern console gaming experience. I only play games like once a year, so when I do decide "hey let's try this game out" the experience is - "sorry, you need to update your system software" ; urgh , wtf, okay ... then I do it and start the game - "this game needs to download an update" ; urgh, wtf, okay ... then "this game must be installed to the harddrive to run; it needs 18.4 GB" ; urgh wtf, okay... finally it's installed and starts up - "this game requires a newer version of the system software"... WTF !? I already did that and you didn't give me the latest version? (drop out to the main menu), yep, there's a newer version, get that. By this point I fucking hate the game and have no interest in playing it.

1/24/2012

01-24-12 - Protectionism Part 3

One of the stranger things that's happened in the past 20 years or so is that the religion of free-marketing has become widely accepted dogma. I say "the religion" because these people believe in it beyond rationality. For examply the widely known ways in which the mathematical abstraction of ideal free markets does not apply to the real world (such as rational actors, perfect information, shared goods, etc.) are waved off as insignificant.

Part of this weird mass delusion is the belief that "protectionism failed" or "the welfare state failed" ; you will frequently see these assertions in supposedly factual news articles. Papers like the NYT which will parrot ridiculous government press releases without question will also say things "free markets have clearly proven themselves" or "nobody wants a return to a welfare state, that model has clearly failed".

Huh, really? I find that interpretation highly dubious; certainly not something that can be dropped as an aside as if it's incontrovertible. During the golden age of America (roughly 1950-70) we were highly protectionistic and we had our most generous welfare state. So, where was the failure? I'm sure the popular memory is somewhat fixated on the bad times of the 70's-80's that were blamed on corrupt unions and inefficient protected businesses and supposedly saved by Reagan/Thatcher free marketism. But there are numerous complicating factors that make that interpretation questionable at best.

The only indisputable success of the free market era was the dot com boom, but that would probably have happened anyway in a more controlled market system (and we may have avoided some major crashes due to excessive speculation and capital liquidity, such as the southeast asian crash and the south american crash).

In the mean time, highly protectionist / socialist states have been doing very well (eg. most of the scandanavian and german speaking states). I don't mean to imply that they are a good model for us to copy, but if you're just looking at world evidence for certain economic/political schemes, it seems there's more pro-socialist examples than there are good "free market" examples.


I believe that part of the problem is the mathematical appeal of free markets in theory. You can set up these toy problems with idealized conditions, and talk about what creates the "global optimum" for all.

There's nothing wrong with playing with toy models, except that people then think that it applies to the real world. A serious economist would be able to list the various ways that the idealized models don't match reality, but then you start drawing yield curves and run simulations and cook up formulas and it all looks so nice that by the time you get to the end you forget that it doesn't actually connect to reality.

1/18/2012

01-18-12 - Stop SOPA but don't stop there

I'm impressed with the breadth of support for this cause, and it is a good one. But at the same time I find it a bit disturbing that people can get mobilized for this, but can't get mobilized for things that are actually much worse, like NDAA, the Patriot Act, campaign finance reform, lobbyist reform, the crushing of consumer protection or glass-steagall or media ownership rules, etc. etc.

In the end, the reason that SOPA exists is because corporate interest groups are literally writing our laws. If you stop SOPA it's like cutting off the top of a dandelion - as long as the roots are in the ground it will just come back.

Lobbyists and persistent and clever, they will slip the laws they want in as a rider on a budget bill, or they'll just keep trying until the opposition movement peters out.

We have to attack the cause.

A very similarly evil act, the Research Works Act (RWA) is moving forward. In brief, the RWA makes it illegal for the government to require that researchers that receive government funding make their results available for free (eg. on the internet). The RWA was written by the AAP (American Association of Publishers), which is made up of the IEEE and ACM and all the usual suspects, who want to have exclusive rights to research paper copyrights.

It should be noted that even before the RWA has passed, we currently have pathetically weak open access to research requirements. Only research that receives direct grant funding from something like the NIH is required to supply open access. But almost all research is actually government funded, because the researchers are paid by colleges, and the colleges get government funding (and of course all research is based on past research, which is a public good, etc). But this type of funding does not require the documents to be open access. So, first of all :

Colleges : require all of your professors' publications to be open, NOW!

Apparently Harvard has sort of done this but not many others have followed suit.

(colleges also have no right to own patents or create public/private partnerships for business development of professors' work. That research is owned by all of us. But that's another topic...)

The IEEE and ACM's standard publication terms gives them exclusive ownership of the copyright, which really they have no business in getting, since their contribution to the work is near zero (eg. peer review is done by unpaid volunteers, they provide almost zero paid editing, etc.; they aren't like a book publisher that actually does something).

Authors : stop publishing your papers in works that take your copyright! Make all your works "open access" only!

Pick an Open Access Journal instead.

Some of the blog posts I've read about the RWA have been very naive in my opinion ; they seem to think that the scholarly organizations like the ACM or IEEE are better than the RWA and aren't specifically the people behind it. I think it should be absolutely clear by now that the RWA is exactly in line with the sleazy character of these usurious organizations.

Scholars : quit the ACM and IEEE right now.

It's very simple, these publishers are bad for science, and they serve no purpose at all in a world where the internet is the most important form of publication.

But more generally it's just another case of corporations writing the laws and pushing an agenda that's bad for America. We may stop SOPA and RWA, but it will just happen over and over again until we fix the problem. Stop corporate personhood. Make lobbying illegal. Make it illegal for corporations to write laws.

1/16/2012

01-16-12 - Some TV Reviews

Recommended :

"The Hour" - period TV is getting a bit old already; this one is sort of lumpy, it lurches around between almost being really good and getting itself stuck in plot deadends and thin characters. It also sort of builds up a lot of promise that doesn't really pay off, but overall I enjoyed it.

"State of Play" - this is old, but I just finally saw it. Yeah it's good, worth watching. I like Bill Nighy a lot. You can tell that the writer Paul Abbott uses that horrible lazy style of writing where he doesn't actually have a plan, he just sort of keeps stringing together one scene at a time to keep it exciting; the writing is not so much logical as it is crafted to create a pacing, to create a cliff-hanger, etc. It can be very effective (just ask JJ Abrams), but in the end if you stop and look back you see that the whole thing has no substance, and the work has no long term impact.

(aside / general rant : I feel like most TV writers these days have got the formula down pretty well for building up a complex story, generating tension, stroking the right emotional peaks at the right time; they keep adding little plot twists, get you more and more excited, and then it all just sort of peters out because they never really had a plan for where to go with it all, and the ending feels forced and rushed; it's kind of like going on a date and having really exciting flirting and foreplay, and then the sex is over in a few seconds and you're like WTF, maybe you could like actually plan out the overall arc a bit more)

"Boss" - pretty great; rich atmosphere, lots of scheming; the long term cross-episode plot is a bit weak, but Kelsey Grammer is fantastic.

"Workaholics" - something is a little off about this show (like none of the stars are actually very good comic actors (the way eg. Charlie Kelly is) and I sort of get the impression that they are like the douchebags that they're making fun of, which creates a weird vibe); nevertheless it's just about the only actually funny show on all of TV at the moment (I hate 30 Rock and Parks & Rec). Every once in a while it really gets it right and is absolutely hillarious.

"Human Planet" - like Planet Earth but with people. Obvious must see.

Adam Curtis Documentaries : google for "adam curtis collection" and get the torrent. These are some of the most beautiful films I've ever seen. Incredible archival footage, nice music, good taste and something exciting, something inspiring. These make me feel like there are other human beings out there who are going about their lives with some intelligence and some creativity, and that gives me great solace. The actual substance of his narration is a bit questionable at times, but I forgive that, and in any case it will make you think about the world. Probably start with "Machines of Loving Grace".

Meh-commended :

"The Take" - well made (all BBC stuff now seems to be incredibly well made; as opposed to 10 years ago when it was way behind American TV; it used to all look like it was shot on handicam, now it all looks as good as the best HBO stuff) ... but crap. The story just has no point to it other than "criminals are bad". The guy who plays Freddie is quite good and carries it for a while, but even his act gets repetitive and I just wound up bored by the end. "Whitechapel" is roughly the same, very generic crime junk, nothing exciting or new or fun about it ("Collision" is even worse; there are an awful lot of BBC police procedurals and most of them are nothing special). Watch "The Shadow Line" and "Red Riding" again instead.

"Downton Abbey" - mmm I dunno; it's beautiful and all, but pretty vapid. Most of the drama is created by the fact that the despicable characters are kept around when they should just be told to bloody well fuck off. It's a pretty common TV method, and I just never enjoy it. It's also written very much like a serial soap opera; the evil characters are almost Susan Luchis of absurd scheming, and you have your naifs and your star crossed lovers. And the overall message is very scummy conservative nonsense.

"Billy Connolly - Journey To The Edge Of The World" : one of the better travel shows I've seen in a while (Riku and Tunna, come back!). His good spirit is infectious, though it does get tiresome; you have to watch in small doses. Didn't like his Route 66 very much though.

Crap :

"Doctor Who" (the new series) - I was pretty sure this would be terrible so put off watching it for a long time; yep, it's terrible. So cheesy, and the pseudo-science is painful. I guess it appeals to the Babylon 5 / Buffy crowd, which is a surprisingly large segment of the scifi viewers (personally I've never seen the appeal of the campy/cheesy in scifi). I watched S01E01 and hated it, so I tried S02E01 to see if it got better in the next year; nope, if any thing Tennant is worse than Ecclestone, he makes it even cheesier; IMO Ecclestone had the right attributes for The Doctor, which is a bit befuddled but also a bit menacing, whereas Tennant has no creepiness.

"Southland" - junk, generic cop crap, all the characters are unpleasant, the stories are super generic, I just don't see anything appealing in this show and am baffled at its high ratings.

"Stephen Fry in America" - meh, he doesn't actually do much driving and seeing America; instead it's these formulaic set up events where he talks to someone who is doing something stereotypically American and thus "learns something" (you never learn anything when you have constructed a pantomime that is just supposed to portray the stereotype you already had (just ask GWB's CIA)). I quite like Stephen Fry but this was not very entertaining.

Michael Wood Documentaries : terrible. Too bad cuz there are lots of them. Similar thing for Brian Cox science shows. I was shocked to read that Cox is actually a professor; from watching his shows I did not get the impression that he understands physics at all.

"Sons of Anarchy" : sort of cartoony silly unpleasant people doing unpleasant things. I have no idea why this show is rated so highly by critics. I dunno, I guess a lot of people like reality TV, which is basically dumb unpleasant people being dumb and unpleasant with no point to it.

"Community" : the protagonist is so profoundly unlikeable that I found nothing funny in this. John Oliver is the best part, and that's not a good thing (because John Oliver is not very funny). Also highly rated and not sure why.

"Homeland" : extremely unpleasant to watch, and also mentally dangerous right wing propaganda (and yes I know the "twists" and no that doesn't change the overall message). If you enjoy this or "24" there is something seriously wrong with you.

New shows that look promising :

"Luck" , "On Freddie Roach" , "Key & Peele"

I've also tried a few shows like "Cougar Town" and "Happy Endings" because they get very high metacritic or IMDB ratings, even though my spidey senses knew they were going to be awful. Wow, it kind of blows my mind that people are still making this super-sitcomy crap. Like, the characters all stand side-by-side last-supper style while they talk to each other so you can get everyone in frame. One of them is always the "wacky" one who starts doing the robot dance randomly and everybody just thinks it's funny, not incredibly strange. And there's some big drama and confusion and everyone cries then makes a big gesture like professing their love over the speakers at Karoake night and then Urkel stumbles in and the gang all says "that's Urkel!". WTF TV, WTF.

1/09/2012

01-09-12 - LZ Optimal Parse with A Star Part 5

Wrapping up the series with lots of numbers.

Previous parts :

cbloom rants 12-17-11 - LZ Optimal Parse with A Star Part 4
cbloom rants 12-17-11 - LZ Optimal Parse with A Star Part 3
cbloom rants 12-17-11 - LZ Optimal Parse with A Star Part 2
cbloom rants 10-24-11 - LZ Optimal Parse with A Star Part 1

I'm delirious with fever right now so I might write something inane, but I'm so bored of lying in bed so I'm trying to wrap this up. Anyhoo..

So first of all we have to talk a bit about what we're comparing the A Star parse to.

"Normal" is a complex forward lazy parse using heuristics to guide parsing, as described in Part 1. "Fast" is like Normal but uses simpler heuristics and simpler match finder.

"Chain" is more interesting. Chain is a complex "lazy"-type parser which considers N decisions ahead (eg. Chain 4 considers 4 decisions ahead). It works thusly :

Chain Parse : first do a full parse of the file using some other parser; this provides with a baseline cost to end from each point. Now do a forward parse. At each position, consider all match and literal options. For each option, step ahead by that option and consider all the options at the next position. Add up the cost of each coding step. After N steps (for chain N) add on the cost to end from the first baseline parse. Go back to the original position and finalize the choice with the lowest cost. Basically it's a full graph walk for N steps, then use an estimate of the cost to the end from the final nodes of that sub-graph.

To make Chain parsing viable you have to reduce the number of match options to a maximum of 8 or so. Still Chain N has a complexity of 8^N , so it becomes slow very quickly as N grows.

Chain forward parse is significantly better than LZSS style backwards optimal parse for these LZ coders that have important adaptive state. The baseline parse I use for Chain actually is a backwards LZSS optimal parse, so you can see how it does by looking at the "Chain 0" results.


First overall results. Chain 6 is the most amount of steps I can run in reasonable time, and AStar 2048 means the quantum length for dividing up the file for AStar was 2048.

raw Fast Normal Chain 6 AStar 2048
lzt00 16914 5179 5016 4923 4920
lzt01 200000 198313 198321 198312 198312
lzt02 755121 181109 177792 173220 173315
lzt03 3471552 1746443 1713023 1698949 1690655
lzt04 48649 13088 12412 10407 10249
lzt05 927796 368346 367598 355804 354230
lzt06 563160 352827 351051 344721 343173
lzt07 500000 226533 215996 209133 208566
lzt08 355400 250503 249987 230541 230220
lzt09 786488 302927 287479 268544 265525
lzt10 154624 11508 10958 10307 10291
lzt11 58524 20553 19628 19139 19087
lzt12 164423 29001 26488 23966 23622
lzt13 1041576 935484 931415 924510 922745
lzt14 102400 47690 47298 46417 46350
lzt15 34664 10832 10688 10269 10260
lzt16 21504 10110 10055 9952 9927
lzt17 53161 19526 18514 17971 17970
lzt18 102400 64280 63251 59772 59635
lzt19 768771 322951 288872 269132 269162
lzt20 1179702 888881 872315 856369 855588
lzt21 679936 91677 88011 83529 83184
lzt22 400000 287715 284378 279674 279459
lzt23 1048576 807253 804048 798369 798334
lzt24 3471552 1418076 1411387 1399197 1388105
lzt25 1029744 113085 107882 97320 100175
lzt26 262144 212445 210836 207701 207552
lzt27 857241 237253 235137 222023 220837
lzt28 1591760 332660 308940 260547 252808
lzt29 3953035 1193914 1180823 1147160 1135603
lzt30 100000 100001 100001 100001 100001
10800163 10609600 10337879 10289860


Now number of Chain steps for the chain parser : (that's O0 - O6)

U N O0 O1 O2 O3 O4 O5 O6
lzt00 16914 5016 5024 4922 4922 4922 4922 4923 4923
lzt01 200000 198321 198321 198312 198312 198312 198312 198312 198312
lzt02 755121 177792 177877 175905 174835 174073 173759 173509 173220
lzt03 3471552 1713023 1712337 1704417 1703873 1702651 1701635 1700282 1698949
lzt04 48649 12412 11315 10516 10481 10457 10427 10416 10407
lzt05 927796 367598 368729 365743 364332 360630 356403 355968 355804
lzt06 563160 351051 350995 346856 345500 344778 344739 344702 344721
lzt07 500000 215996 215644 211336 209481 209259 209244 209138 209133
lzt08 355400 249987 249372 239375 237320 231554 231435 233324 230541
lzt09 786488 287479 284875 280683 275679 270721 269754 269107 268544
lzt10 154624 10958 10792 10367 10335 10330 10311 10301 10307
lzt11 58524 19628 19604 19247 19175 19225 19162 19159 19139
lzt12 164423 26488 25644 24217 24177 24094 24108 24011 23966
lzt13 1041576 931415 931415 929713 927841 926162 924515 924513 924510
lzt14 102400 47298 47300 46518 46483 46461 46437 46429 46417
lzt15 34664 10688 10656 10317 10301 10275 10278 10267 10269
lzt16 21504 10055 10053 9960 9966 9959 9952 9948 9952
lzt17 53161 18514 18549 17971 17970 17974 17971 17973 17971
lzt18 102400 63251 63248 59863 59850 59799 59790 59764 59772
lzt19 768771 288872 281959 277661 273316 269157 269141 269133 269132
lzt20 1179702 872315 872022 868088 865376 863236 859727 856408 856369
lzt21 679936 88011 88068 84848 83851 83733 83674 83599 83529
lzt22 400000 284378 284297 281902 279711 279685 279689 279696 279674
lzt23 1048576 804048 804064 802742 801324 799891 798367 798368 798369
lzt24 3471552 1411387 1410226 1404736 1403314 1402345 1401064 1400193 1399197
lzt25 1029744 107882 107414 99839 100154 99710 98552 98132 97320
lzt26 262144 210836 210855 207775 207763 207738 207725 207706 207701
lzt27 857241 235137 236568 233524 228073 223123 222884 222540 222023
lzt28 1591760 308940 295072 286018 276905 273520 269611 264726 260547
lzt29 3953035 1180823 1183407 1180733 1177854 1170944 1162310 1152482 1147160
lzt30 100000 100001 100001 100001 100001 100001 100001 100001 100001
10609600 10585703 10494105 10448475 10404719 10375899 10355030 10337879

Some notes : up to 6 (the most I can run) more chain steps is better - for the sum, but not for all files. In some cases, more steps is worse, which should never really happen, but it's an issue of approximate optimal parsers I'll discuss later. (*)

On most files, going past 4 chain steps helps very little, but on some files it seems to monotonically keep improving. For example lzt29 stands out. Those files are ones that get helped the most by AStar.


Now the effect on quantum size on AStar. In all cases I only output codes from the first 3/4 of each quantum.

raw 256 512 1024 2048 4096 8192 16384
lzt00 16914 4923 4923 4920 4920 4920 4921 4921
lzt01 200000 198312 198312 198312 198312 198312 198314 198314
lzt02 755121 175242 173355 173368 173315 173331 173454 173479
lzt03 3471552 1699795 1691530 1690878 1690655 1690594 1690603 1690617
lzt04 48649 10243 10245 10234 10249 10248 10241 10241
lzt05 927796 357166 354629 354235 354230 354233 354242 354257
lzt06 563160 346663 343202 343139 343173 343194 343263 343238
lzt07 500000 209934 208669 208584 208566 208556 208553 208562
lzt08 355400 228389 229447 229975 230220 230300 230374 230408
lzt09 786488 266571 265564 265487 265525 265559 265542 265527
lzt10 154624 10701 10468 10330 10291 10273 10273 10272
lzt11 58524 19139 19123 19096 19087 19085 19084 19084
lzt12 164423 23712 23654 23616 23622 23628 23630 23627
lzt13 1041576 923258 922853 922747 922745 922753 922751 922753
lzt14 102400 46397 46364 46351 46350 46350 46348 46350
lzt15 34664 10376 10272 10260 10260 10251 10258 10254
lzt16 21504 9944 9931 9926 9927 9927 9927 9927
lzt17 53161 17937 17970 17968 17970 17969 17969 17969
lzt18 102400 59703 59613 59632 59635 59637 59640 59640
lzt19 768771 269213 269151 269128 269162 269193 269218 269229
lzt20 1179702 855992 855580 855478 855588 855671 855685 855707
lzt21 679936 83882 83291 83215 83184 83172 83171 83169
lzt22 400000 279803 279368 279414 279459 279605 279630 279647
lzt23 1048576 798325 798319 798321 798334 798354 798357 798358
lzt24 3471552 1393742 1388636 1388031 1388105 1388317 1388628 1388671
lzt25 1029744 97910 101246 101302 100175 100484 100272 100149
lzt26 262144 207779 207563 207541 207552 207559 207577 207576
lzt27 857241 222229 220832 220770 220837 220773 220756 220757
lzt28 1591760 256404 253257 252933 252808 252737 252735 252699
lzt29 3953035 1136193 1135442 1135543 1135603 1135710 1135689 1135713
lzt30 100000 100001 100001 100001 100001 100001 100001 100001
10319878 10292810 10290735 10289860 10290696 10291106 10291116

The best sum is at 2048, but 1024 is a lot faster and almost the same.

Again, as the previous note at (*), we should really see just improvement with larger quantum sizes, but past 2048 we start seeing it go backwards in some cases.


Lastly a look at where the AStar parse is spending its time. This is for a 1024 quantum.

The x axis here is the log2 of the number of nodes visited to parse a quantum. So, log2=20 means a million nodes were needed to parse that quantum. So for speed purposes a cell one to the right is twice as bad. The values in the cells are the percentage of quanta in the file that needed that number of nodes.

(note : log2=20 means one million nodes were visited to output 768 bytes worth of codes, so it's quite a lot)

log2 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22
lzt00 0 0 0 18.18 59.09 18.18 4.55
lzt01 3.75 0.75 41.2 34.08 13.86 5.62 0.75
lzt02 1.81 1.36 25.37 34.09 13.59 13.02 8.15 1.93 0.23 0.23
lzt03 1.46 1.18 17.51 18.46 14.16 13.17 6.95 4.81 3.66 4.81 9.54 2.79 0.96 0.11 0.03
lzt04 1.67 0 0 1.67 0 21.67 5 18.33 3.33 10 16.67 16.67 5
lzt05 0.59 0.25 4.41 10.77 9.92 18.32 13.23 10.09 9.67 6.02 12.47 3.22 0.51 0.08 0.08
lzt06 0.8 0.93 6.81 23.77 14.69 16.96 21.09 11.48 2.67 0.8
lzt07 0.46 0.46 8.66 7.88 6.8 15.3 17 14.53 5.56 9.58 8.19 4.79 0.31 0.31
lzt08 0 0 0 0 0 0 1.68 1.68 1.47 27.67 53.88 11.95 1.68
lzt09 0.29 0.48 0.76 0.86 0.95 3.9 28.07 47.76 16.18 0.38
lzt10 0 0.56 10.17 12.99 9.04 9.04 10.17 41.24 4.52 0.56 1.13
lzt11 0 0 7.89 10.53 14.47 17.11 6.58 9.21 21.05 10.53 2.63
lzt12 0 0 0 0 0 4.27 28.91 59.24 7.58
lzt13 0 0 0.07 0.14 0.57 1.72 3.36 5.72 39.24 42.03 7.08 0.07
lzt14 0 0.83 0 2.5 8.33 34.17 42.5 5 2.5 1.67 0.83 0 0.83
lzt15 0 2.27 4.55 15.91 13.64 15.91 13.64 6.82 11.36 11.36 4.55
lzt16 0 0 3.57 0 14.29 42.86 32.14 3.57
lzt17 1.39 1.39 2.78 1.39 4.17 75 13.89
lzt18 0 0 0 0 0 0.72 0 2.17 2.9 11.59 56.52 23.19 2.9
lzt19 0 0 1.26 2.81 0.39 7.56 87.11 0.87
lzt20 0 0.13 2.08 2.02 4.29 67.07 24.29 0.06
lzt21 0.2 0.78 6.07 6.07 5.28 19.77 35.62 22.9 1.96 0.2 0.2
lzt22 0 0.56 2.98 5.59 26.82 62.94 1.12
lzt23 0 0 0 0 0 0.07 1.35 2.63 0.92 70.88 23.15 0.14 0.36 0.5
lzt24 0.44 0.61 4.14 37.41 7.62 12.68 12.72 8.52 6.11 5.19 3.11 0.94 0.31 0.04
lzt25 0.22 0.43 1.52 1.74 2.68 6.44 15.69 27.19 30.22 13.09 0.72
lzt26 0 0 0 1.15 3.15 2.58 77.65 14.61 0.57
lzt27 0.61 0.1 7.55 6.53 1.22 4.39 5 4.08 7.76 44.8 16.43 1.43
lzt28 0.25 0.1 3.71 0.94 0.74 6.77 15.56 10.08 10.97 14.82 18.68 11.41 4.05 1.24 0.1
lzt29 0.3 0.73 1.61 22.37 5.28 6.16 26.34 2.97 0.48 0.85 19.63 12.47 0.73
lzt30 3.7 0.74 47.41 34.07 12.59 0.74

Well there's no easy answer, the character of the files are all very different.

In many cases the A Star parse is reasonably fast (comparable to Chain 3 or something). But in some cases it's quite slow, eg. lzt04, lzt08, lzt28.


Okay, I think that's all the data. We have one point to discuss :

(*) = in all these type of endeavors, we see these anomolies where as we give the optimizer more space to make decisions, it gets better for a while, then starts getting worse. I saw the same thing, but more extreme, with video coding.

Basically what causes this is that you aren't optimizing for your real final goal. If you were optimizing for the total output size, then giving it more freedom should never hurt. But you aren't. With Chain N or with A Star in both cases you are optimizing just some local portion, and it turns out that if you let it make really aggressive decisions trying to optimize the local bit, that can hurt overall.

A similar issue happens with an Huffman optimal parse, becuase you are using the huffman code lengths from the previous parse to do the current parse. That's fine as long as your parse is reasonably similar, but if you let the optimal parser really go nuts, it can start to get pretty far off those statistics, which makes it wrong, so that more optimizing actually gives worse results.

With video coding the main issue I had was that the optimization was generally local (eg. just on one macro block at a time or some such), but it of course affects the future as a source for motion compensation (and in other ways), and it turns out if you do really aggressive optimization on the local decisions, that can wind up hurting overall.

A similar thing can happen in image and video coding if you let optimization proceed very aggressively, because you have to use some simple analytic criterion (such as RMSE - though even if you use a fancier metric the same problems arise). The issue is that the coder can wind up finding strange states that are a good trade-off for RMSE, but wind up looking just horrible visually.

Obviously the correct solution is to optimize with the true final goal in mind. But that's not always possible, either computationally, or because the final goal is subjective.

Generally the solution is to moderate the optimization in some way. You have some heuristic idea of what kind of solutions will provide good globally optimal solutions. (for example, in image/video coding, you might require that the bit rate allocation not create too big of a difference between adjacent blocks). So you sort of want to guide your optimization to start around where you suspect the answer to be, and then you tune it so that you don't allow it to be too aggressive in making whatever decision it thinks is locally optimal.

01-09-12 - Protectionism Part 2

Nobody likes the idea of protectionism, because it conjures idea of jingoism, as well as corrupt inefficient business locking in their market through political deals. But in fact basically every country in the world still engages in heavy protectionism, in the form of subsidizing local business in one way or another.

Subsidies can take many forms. Of course there are direct subsidies (Airbus, Boeing, etc.) and these are generally disliked. There are tax break incentives, which almost every big business in America gets. Obviously there are tons of small business incentives and employment incentives and so on.

But there are also more subtle and indirect form of subsidies.

China (and other asian countries) are big fans of the "business development zone" ; these are areas they construct for certain industries, provide tax breaks, and build up infrastructure for power, transport, etc.

Government paid health care is perhaps the biggest subsidy any country offers. It's a general subsidy for employment (notably, not for business). In the American model, employers pay for health care - only for employed people. With government paid health care, the employers are still paying for health care (through taxes) - but they are paying whether they employ the people or not.

To make that more clear - imagine a system where everyone was paid $100k by the government, and then corporations had to pay taxes to cover that - whether you were an employee or not. Then there's no such thing as "saving money by laying people off" ; the cost per employee is the same whether you hire them or not, so you may as well hire them. Obviously it's too much of a market distortion for the government to just pay all of everyone's expenses whether they are employed or not, but paying some amount regardless of employment amounts to a subsidy for employment. The more basic social welfare of a person is paid by the government from the general tax fund, the more incentive there is to hire people.

So social welfare (health care being the biggest one) is actually a subsidy for local employment. There are other things that reduce the cost of an employee, such as government child care, good public transit (allows you to pay less because employees don't need cars),

Another big one is education. But of course subsidized education is a form of development of a local business resource (people). Some countries have well developed industry-eduction partnerships to provide students with the skills needed.

etc.

I believe that one of the problems in America is that we are in fact engaging in heavy protectionism to this day, but we are not doing it in a very smart way. Defense contractors probably get the biggest subsidy (actually I take that back, finance gets the biggest subsidy by a huge huge margin; finance has been getting around a trillion dollars in direct subsidy while defense only gets a few hundred billion), but so does aviation, mineral/mining, lumber, agriculture. They are generally in the form of direct subsidies (eg. have some free money), which is a very bad way to do it. Direct subsidies tend to go straight into rich people's pockets, they don't promote employment. Subsidies can be crafted in more clever ways, and the best way is not to favor one particular industry over another, but rather to favor employment over outsourcing and let the market decide what type of employment is best.

Of course the US government does heavily subsidize certain behaviors, so the idea of subsidy to affect the market is not at all exotic. The subsidies for real estate investment are massive; even ignoring TARP and the FM's and such, the mortgage interest deduction and capital gains exclusion are huge behavior modifiers for very questionable benefit. The lower capital gains tax (vs. income tax) and dividend tax rate are huge subsidies for investors. And of course the tax code in general is a huge subsidy for corporations (vs. individuals) and particularly for multi-national corporations. Why are we massively subsidizing all those things, and not employment and local business?

1/07/2012

01-07-12 - Protectionism

There are some basic economics that I just don't understand, and a lot of the times the accepted "right answer" conflicts with common sense.

One example is it seems to me that buying something locally made is better for the local area than buying something made far away. (for "locality" you may substitute "state" or "nation" or whatever region you want to divide things by).

For me this conjures bad memories of the anti-"Jap" "USA USA" crowd of the '80's that had "buy American" bumper stickers and such ; something in my moral fibers says you should buy the best quality cheapest product. But I don't think that's true.

Consider for the moment the case that there are two products, identical in price and quality. One is locally made, one is foreign made. I contend it is better for the local area (and usually better for you personally) to buy the locally made product. When you do that, the money goes to someone who lives nearby, who spends that money again, and that person spends it, again, etc. This makes the local area prosperous.

(in a purely selfish sense, whether or not making the local area prosperous is good for you or not depends on the details of your situation; if you are a merchant or an altruist, it is good for you; but if your business is international and you would prefer local property values to be low, it might be bad for you; we will assume for the moment that you want the local area to benefit).

So there is some value to buying local and keeping money and industry circulating locally.

So, even if the local product is somewhat more expensive, it still might be better overall if you bought that instead of the foreign product. You have to weigh the benefit of both; the region gains some utility from access to cheaper foreign products, but that is traded off against not circulating that money around the local economy. eg. there's some break even point (in terms of overall utility) ; maybe if the local product costs 20% more, that's the actual break even point.

Of course consumers should not have to make that decision themselves, they should just be able to buy the cheapest product. The correct way to fix that is with government - one of the valuable things that government can do is to make apparent price equal to actual price (eg. to make price proportional to utility, or to move long term costs forward, etc.), or to use laws to bias pricing so that logical purchasing decisions lead to the greatest overall utility (eg. putting penalties on products that help you but hurt others).

The obvious way to make the prices match utility here is either with tarrifs on imports, or subsidies for local production. This is called "protectionism" to attack it, but it seems to me it's just a way of getting the benefit of circulating those dollars locally.

I'm a little disturbed by my conclusion because it's awfully close to the anti-globalization crackpots who claim that modern government financial policy benefits "wall street not main street" (and other slogans).


Granted, in reality, it's too late to go back to pre-1990's protectionism. The cat is out of the bag. And of course in reality protectionism degrades into political gifts for corrupt corporations. But we can ignore those issues for the theoretical discussion.

Also, if you are an extremely altruistic chap you might question the whole goal of maximizing the benefit to your locality (nation/state/fiefdom/whatever). You might say the goal of policies should be to maximize the good for the world. But for the moment let's ignore that and assume that the government of a nation should act to maximize benefit for that nation.

1/06/2012

01-06-12 - Surveying is a Powder Keg

So I got my property surveyed a while ago, because there were some boundaries I wasn't totally sure about, and wanted to see how much space I had for fences, etc.

I should have realized this but did not - surveying is a powder keg. When the surveyor comes out and puts out his stakes and flags, it's like a siren call for neighborhood crazies to come around and dispute the line.

Basically people are retarded and unreasonable, and before just calmly talking to you, they assume you will ask them to take down their fence which is across the line (or whatever). Depending on where you live, the exact force of crazy takes different forms; out in the country you might get shot at; over in the suburbs you might get served with a notice of adverse posession. All just because you hired a surveyor, before you even consider doing anything about it. The crazies seem to see the surveyor's flag as a declaration of war.

The mistake I made was I got the survey and then I wanted some time to think about how I was going to fence things, I didn't just jam up the fences right away. That gave the crazies time, and this is what they did :

(pink flag stake is official from the surveyor of course, and crazy neighbor has put up his own line two feet into my property from the official mark)

Well done, crazy neighbor.

A bit of forehead vein popping and yelling at them got them to remove the post, but I expect more complications of this issue before it is done.

(BTW yelling at people is never satisfying in real life the way it is in fiction. In fiction there's this myth that people are actually good and reasonable, and were just doing something bad for a moment, and when you yell at them they realize their mistake and reform; like if Gorden Ramsay walked in at the right moment and yelled at Hitler he would be like "oh gosh, you're right, I'm so ashamed, I'll try to do better". Furthermore in fiction, they respond to the yelling either by yelling back, giving a satisfying argument, or by accepting the scolding and apologizing. In real life that never happens, what really happens is they try to change the subject, or turn it around and somehow blame you, or make excuses, or bring up random other points that don't matter to the issue (*), and it just leaves you feeling derailed).

(* = this is maybe the most common and most effective response - the completely random diversion into some other story that just leaves me going "WTF?" and totally takes the wind out of my sails. The other super effective tactic that I've noticed from like car salesman and contractors and such is to just completely stone wall you, like they tack on some $500 extra fee that they specifically agreed they wouldn't do, and you're like "hey WTF is this fee that you said wouldn't be there" and they just act like they did nothing wrong and of course you're going to go along with them, like "yep that's the $500 extra rapeage surcharge, so you can pay now by giving me your bank account and social security number..." , wait, what? I'm in the middle of yelling at you, you can't just act like your way is the only way).

01-06-12 - Nice Wiring Bub

I took off a horrible track light (blyeck, so tacky, track lights and can lights are the worst), and underneath I found this gem :

at first I thought it was just a big wad of masking tape on the end of the wire (bad enough, not an actual insulator, and a fire hazard), but upon peeling the masking tape I found this :

Oh, of course. You spliced on an extra 1 inch of wire, no wire nut, wrapped in masking tape - and the thing that most boggles my mind is that the wire ends are not even twisted in any kind of sane way, they are just randomly balled up around each other. Not to mention the original wires are plenty long without the splice.

Pretty impressive piece of fail.

BTW one of the hazards of old knob & tube wires is that the insulation is only rated to 60 C, but newer light fixtures are allowed to heat up to 90 C (which new Romex can handle). So you need to be careful when installing new light fixtures, and at the very least don't over-bulb (*). One way to solve this (without a ton of rewiring) is to back up the knob and tube a few feet away, put a junction box there, then run new Romex for the last few feet.

(* = I just love to over-bulb ; I can never get enough light; I used to put 100 W's in everything whenever I moved into an apartment. Here I might be a bit more careful about that, because they do generate a lot more heat (BTW I despise the gross inhumane light of CFL's, but one advantage of them with old wiring is they draw much less power (which keeps the wires cool) and they themselves are cool (which doesn't heat up the light fixture and box)). I'm not a huge fan of dimmers (fucking 75 W is already dark enough, I don't need any less than that), but if I could install an *amplifier* that let me over-drive the bulbs that would be sweet (but not in my house, which is apparently wired by paper clips and masking tape)).

It's kind of scary what kinds of disasters can be hiding inside your walls that you don't know about upon purchase (or maybe ever, until they cause water damage or a fire or whatever). I really like doing home improvement work in the garage and the basement, because the walls are unfinished so I can see where the studs are, which is so handy, I can see all the wire runs and junction boxes. It's totally superior.

Covering up your walls is super over-rated. I think if I was designing a modern house it would be all Pompidou Center style with color-coded pipes running around where I could directly access the electricity, water, etc.

If you want a more old fashioned home look, you could still do all your major wire runs around the ceiling and then cover them with a removeable wood crown molding piece. That way if you want to get into the wires, you just pop off the crown molding and you have a wooden box for access.

1/04/2012

01-04-12 - Two laws you should hate

NDAA : makes legal the GWB/Obama policy of indefinite detainment (outside of war zone, Geneva convention, or any legal jurisdiction), and unilateral assasination orders -

� Obama�s Signing Statement on NDAA I have the power to detain Americans� but I won�t Alex Jones' Infowars There's a war on
The NDAA's historic assault on American liberty Jonathan Turley Comment is free guardian.co.uk
The Hit List The Public Applauds As President Obama Kills Two Citizens As A Presidential Prerogative � JONATHAN TURLEY
Senate Votes Overwhelmingly To Allow Indefinite Detention of Citizens � JONATHAN TURLEY

SOPA : basically makes free speech on the internet impossible, by making site hosts legally liable for any content posted on them. Basically allows private companies to censor the internet at will.

Stop Online Piracy Act - Wikipedia, the free encyclopedia
SOPA for Dummies - Google Docs
House takes Senate's bad Internet censorship bill, tries making it worse

SOPA might not pass in it's worst form, but some lobbyists are pushing very hard for something like this, so the internet is going to get censored unless we fight it very hard.

01-04-12 - Police Brutality

Much has been made of the outrageous treatment of Occupy protestors by police. But I believe it's been small potatos compared to the rampant, systemic brutality which pervades our nation's police departments. It's fallen out of the news because we're bored of it (we've seen black guys getting beaten by cops a million times) and because it doesn't affect the wealthy, but it has really not gotten better (or not enough, anyway).

Police in American are de-facto above the law. They violate human rights at will, with rarely a punishment greater than suspension or transfer.

Here in Seattle things have gotten shockingly bad, so bad in fact that even the DOJ has made an official report about how bad our police department is. (original here) . What is Seattle's official response? Not to do anything about it, it's to question the methodology of the report. Shameful.

To Protect & Skull-Fuck - Page 1 - News - Seattle - Seattle Weekly
SPDecay - Page 1 - News - Seattle - Seattle Weekly
Seattle sues attorney over public records request Local & Regional Seattle News, Weather, Sports, Breaking News KOMO News
Seattle Police Department Sued by KOMO News for Not Releasing Dash-Cam Videos - Seattle News - The Daily Weekly
Seattle Police A Department in Denial - Page 1 - News - Seattle - Seattle Weekly

One of the few ways that people are getting any justice these days is by getting the dash cam footage to prove that the cops' lies are in fact lies. (see for example Ian Birk lying about John T. Williams "lunging" at him (eerily similar to the very tragic case of Otto Zehm in Spokane in which officers also lied and claimed he "lunged" (with a soda pop bottle, which led them to kill him))). The result of course is that SPD is doing what it can to stop the dash cam system. They are now suing to stop releases of footage under public records disclosures, and have "accidentally" deleted many thousands of hours of footage.

Police Chief Diaz needs to be fired.

But in the larger picture, the big problem is the stone wall and loyalty attitude of police departments; that when there is a case where a police office may have killed a civilian without cause, their attitude is not to investigate and apologize, it's to cover it, draw ranks, support the officer, etc. This attitude makes not just the few bad cops responsible, but every cop who treats his compatriots as beyond reproach or above the law. Loyalty to evil is not admirable. (ask Joe Paterno).

Following a rash of unjustified killings in the 80's, many laws were passed that make it somewhat more difficult for police officers to use their guns. But the gap has been filled by stompings, clubbings, and taserings.

spokane police abuses summary

previous post at cbloomrants

Of course part of the problem is that there is a decent portion of the population that thinks "tough policing" is a good thing.

01-04-12 - Double Pane Glass is a scam

"Replacement Windows" are shit sold by the window industry to sucker homeowners. The tiny gap (typically less than 1 inch) in a standard double pane IGU (integrate glass unit) is no better (and sometimes worse) than a traditional window + storm.

Throwing out perfectly good lovely old windows for "environmental" reasons is of course retarded; if you want more air proofing and don't already have storm windows, just get some good storms and you're done.

Replacement windows almost always uglier than good old wood windows, which architecturally fit the house and have nice wavy old glass.

Furthermore, they cannot be maintained and repaired in the same way as an old window + storm. When an IGU fails (which they do in 10-20 years typically) it cannot be repaired, it has to be replaced. Old wood windows can be easily taken a part, cleaned, resealed, and can last 100 years. (Vinyl windows and caulk and foam seal strips and so on are similarly problematic - they seem great at first, but they all decay badly in sun and weather, so have to be replaced regularly and can't really be maintained).

Replacement windows are usually shoved inside the existing window framing, and add their own thin frame which makes the opening smaller and adds an extra ugly architectural detail.

It's a standard "sustainable" bullshit corporate ripoff. To sell you some new crap and get you to throw away your perfectly good old windows. (it's like the wonderful irony of "sustainable christmas trees").

This guy addresses the issue in much more detail.

old rants