Well, at this point we should be unsurprised that it sucks balls :
log rmse :
ms-ssim-scielab :
One note : the rightmost data point from hipix is their "perfect" setting, which is very far from perfect. It's only a little over 2 bpp and the quality is shit. I feel bad for any sucker customers who are saving images as hipix "perfect" and thinking they are getting good quality.
I started to think , man maybe my ms-ssim-scielab is just way off the mark? How can everyone be so bad? Any time your test is telling you things that are hard to believe, you need to reevaluate your test. So I went and looked at the images with my own eyes.
Yep, hipix is awful. JPEG just blows it away.
A sample from the hipix image closest to the 0 on the x axis , and a JPEG of the same size : (HiPix is 230244 bytes, JPEG is 230794 bytes)
JPEG :
HiPix :
Note the complete destruction of the wood grain detail in the hipix, as well as introduction of blockiness and weird smudge shapes. Note the destruction of detail in the plate rim, and the ruining of the straight line edge of the black bowl.
BTW when you are evaluating perceptual quality, you should *NOT* zoom in! JPEG is optimized for human visual system artifact perceptibility at the given scale of the image. JPEG intentionally allows nasty artifacts that look bad when you zoom in, but not when you look at the image in its normal size.
Conclusion : Hipix needs to immediately release a "new and improved HiPix v2.0 that's way better than the last!" by just replacing it with JPEG.
Since they don't offer a command line app I won't be testing this on any more images.
ADDENDUM : Well I ran two points on Moses :
The two points are "High" and "Perfect" and perfect is way not perfect.
There's one thing I can't stand that's cheats. I declare in advance that I am an H.264 fan. I never liked you writing on x264, but this post made me very angry. Why so? Because it is clearly a fabrication. I saw this blog, where soemthing serious was done to test that hipix:
ReplyDeletehttp://cyleow.blogspot.com/2010/10/jpeg-hell.html
I took the trouble, went to hipixpro.com and understood that their technology is not really the codec but the container. So one could take what they have and use it with x264 as the core. What you can download from their "commercial" free - yes, free download, is a sample app for using the technology. OK. I downloaded the stuff to my PC and tried it. Nothing like what you claim. Actually it is great. If you told me how, I could send you a sample of 40MP image half the size of what Photoshop CS3 jpeg does at two thirds the size and two thirds the artifacts. So clear that you needn't be an expert to see the difference.
What made me most angry was your comment on the perfect quality they offer. I tried some of my RAW 16 and 21 (D5 MarkII) and got amazing quality at 2-5MB depending on the photo. I took the trouble to run some SSIM and PSNR tests using the MSU tool, and found out that these were identical to my Photoshop JPEG at about 80% of the size. So, to sum it all - it's all about your serving some H.264 oposition. Your mouth is big and your language is dirty, and I believe you are fabricating your graphs.
WTFBBQ
ReplyDeleteSome sort of astroturfing stunt, maybe? :)
ReplyDeleteThe one thing I will say about this series is that you very casually use jpeg+paq without always being clear about it (you'll often talk about that as "jpeg"), plus you're doing the thing of using different jpeg compression for the RMSE and the SSIM. Both of those are perfectly reasonable things to do, of course, in terms of critiquing claims that other formats blow JPEG away.
ReplyDeleteBut I think if people jump in in the middle, or are skimming or not reading very carefully, they may miss that, and think then when you're saying "jpeg" you mean plain old "jpeg".
Of course, I'm not sure that's what's going on here, because why anyone would think Photoshop CS3 jpeg is a good jpeg compressor I don't know. (Actually maybe it is now, but it was notoriously overly large back in the day.)
But if anybody *did* try to reproduce your results, who knows.
(Also, testing on only a single image is kind of odd. I'd think you'd at least test a couple modes out on a few different files, and say 'hey, all the files exhibit the same relationship between these codes, so I'm just going to use this one representative file for further testing'.)
"The one thing I will say about this series is that you very casually use jpeg+paq without always being clear about it (you'll often talk about that as "jpeg"), plus you're doing the thing of using different jpeg compression for the RMSE and the SSIM."
ReplyDeleteYep, absolutely. I try to drop reminders of this every so often but it's good to have them more often.
"jpeg" here is not an actual jpeg you can go run. It's intended as a reference point for "this is a level of quality that's completely trivial to get to" , so you better be competitive with that if you are serious. It's also meant as a counter-proposal to any new image format; instead of your whatever idea, just put a better entropy coder on jpeg.
But yeah, I am showing JPEG PAQ,Arith, and Huff on their now. Anybody can use JPEG Arith right now.
"Also, testing on only a single image is kind of odd. "
Yeah, I should be more clear about that.
This phase is the "qualifying round". Compressors that show they are at least vaguely working right will make it into the next round where I will run them on many images.
"Of course, I'm not sure that's what's going on here, because why anyone would think Photoshop CS3 jpeg is a good jpeg compressor I don't know."
ReplyDeletePhotoshop used to have two very different JPEG compressors (I haven't used any PS version later than 7.0 seriously so I have no idea if this is still true). There's the one you get via "Save", which is comparable to the IJG code compression-wise but puts tons of tags and metadata in so the JPEGs end up fairly big. And there's the JPEG compressor in "Save for the Web", which is different, doesn't write anything that doesn't need to be in there and seems to do at least some amount of RDO. Did some testing a while back and that was by far the best JPEG compressor I found in terms of (subjective) visual quality for file size. I didn't try any JPEG recompressors though (This was an actual web design thing, not a compression benchmark - I tried a few different programs and did a Google search or two, but I didn't go out of my way to find the best compressor, I just spent an extra half hour to keep page load times down).
So Photoshop JPEGs can be very good, but it really depends on how they're saved.
Tangentially related: I've found that progressive JPEGs are often a tad smaller - not much smaller, 0.5-1.0% maybe, but it's something to keep in mind when comparing.
Yeah I forgot about that, I need to try that in the redo.
ReplyDelete