There are trees all around the river and the river runs north-south, which means you only get full sun exposure for maybe 3 hours in the middle of the day (east-west rivers get more continuous sun on the north bank). It takes about 80 minutes to get there from SF if you drive fast. It's like 5 minutes from Santa Cruz. The main swimming hole is called "Garden of Eden" and is a very short and easy walk from the road. The combination of easy access and proximity to Santa Cruz mean it's jam packed with people, both locals and college kids. It's not worth going to the main hole. Direct access to the holes is easy from the many parking spots along Highway 9 between Felton and SC. There's no need to pay the park day use fee or go into the main part of the park at all. Another major hole is "Big Rock Hole" that you have to hike about 2 miles to get to, starting at the Rincon Fire Road gate. I thought maybe that hike would make it more empty, but there were a ton of cars parked at the gate so we didn't check that out, maybe they all went to "Frisbee Beach" and not Big Rock Hole so maybe that would've been a good spot.
On the map there's only official parking at the Ox Trail gate and the Rincon Fire Road gate, but in reality there's parking all along, and most people park very close to Garden of Eden and cut down. Don't do that, it's a short walk from Ox Trail gate and there's no need to add to the erosion. It looks like "Ox Trail" is separate from the railroad, but actually it runs right along it. Also the railroad actually runs right along the river edge, so as you walk you can spy the river and check out the holes. There are various little paths people have cut down the bank - do not take them. They're really badly eroded and hard to climb, and it's also pointless. Just go down to Garden of Eden, which is well signed and easy to walk, and from there it's easy to walk along the river bank upstream to more holes. We went about 200 yards upstream from Eden and were alone almost all day.
I don't think the main holes are really worth a return visit due to overcrowding and excessive ease of access, but the river is very promising. I'm sure there are plenty of other holes that aren't so well known, and with a bit of scrambling around on rocks you should be able to get a secluded swim.
On the plus side, Henry Cowell is a pretty setting, lots of nice redwoods, a rich smell of redwood in the air, some decent trails and one hike to a view. It is a small park that is everywhere very near civilization, but you can occasionally forget and feel like you're far out and alone.
p.s. hole crap Santa Cruz is so bizarre and trashy. I actually sort of like the run down beachside carnival town aspect, but now it's got yuppie gentrification randomly scattered around it, with brand new condos and the big strip of chain store shopping which creates a really bizarre contrast. Scott's Valley is even worse. It seems to have sprung up over night and is now a tract-home + big-box-store suburb plunked down in the mountains.
All the trash left around these popular river swim spots really hurts my fragile faith in humanity. I start feeling okay about people again, I think, hey there are actually people who appreciate nature and get out and do fun things with their time, these are my kind of people. Then they go and fuck it up. It's not just that they fuck it up, it's that it's so purely destructive and retarded. It's not even the usual way that humanity fucking sucks - if people can give themselves some small pleasure at great cost to others, they generally do it. Okay, I've come to know and accept that, though I'm not happy about it. But this littering in nature, it's not really even much of a pleasure to the litterer, it's just a purely destructive asshole thing to do. Even among the decent subset of humanity that actually gets outside, a large percentage of them are fucking dicks. Sigh.