Single speed bikes have no gears, no derailleur, no shifters, and the ones favored by hipsters have no hand brakes. That's all okay as long as you're on flat ground, but SF is full of crazy hills, and single speeds = the suck. Gears and derailleurs are AWESOME, they were invented for a reason and we are lucky to have them. Going down the steep hills with no front break is dangerous, and the way you break on single speeds with your feet makes it tricky, you have to sort of let your feet turn slowly, if you just lock your feet you will skid.
Single speeds are used on the track because 1) tracks are flat and you know what speed you will be going so you can just choose the gear that gives you your desired BPM at that speed, 2) track bikes are designed for minimum weight and air resistance and dropping the derailleur and cassette helps a bit there, 3) single speeds have a tiny bit less slip in the drivetrain so slightly more of your leg power is transformed into motion. These factors do not exist on the street.
Regardless, I'm still a bit jealous of how pretty they are when I see them go by. A single speed would be a good thing to have at Burning Man since that dust plays havoc on brakes and gears and such.