10-14-10 - Image Comparison Part 2

Well, I wanted to post JPEG vs x264 numbers, but there's a problem.

The first problem I had was that the headers out of x264 are very large. JPEG has about 240 bytes of header ; MP4 has about 1200 bytes of header for one frame of video only , the .x264 internal format has about 600 bytes of header. So I'm just doing a subtract to correct for that, but that's rather approximate and ugly since you can store side information in headers, etc. But whatever, that's not the big problem.

The big problem is that the "lossless" x264 is actually really bad (it's only lossless after color conversion). Here is the quality for lossless x264 :

rmse : 4.9177 , psnr : 34.3296
ssim : 0.9840 , perc : 88.5788%
scielab rmse : 3.140
scielab ssim : 0.9844 , angle : 88.7457%

I confirmed that is caused by the chroma conversion & subsample by just making the y4m and then converting the y4m back to RGB, and I get the exact same numbers. You don't usually see this because people show PSNR in the Y color plane. That's absolutely terrible. For comparison, here's JPEG at 100% quality :

rmse : 2.1548 , psnr : 41.4967
ssim : 0.9928 , perc : 92.3420%
scielab rmse : 0.396
scielab ssim : 0.9997 , angle : 98.4714%

now JPEG q 100 isn't even lossless, but this is way more like what you'd expect. A 2.0 base rmse is about what you get from the "lossy" old school chroma conversion that JPEG does.

Unfortunately this large constant error ruins any attempt to measure x264's performance. You can see that the RMSE line for x264 is just offset way up :

x264 vs JPEG : log rmse vs log bpp

The slope is way better than JPEG so we think that if the color space wasn't throwing away so much information it would be much better.

So if anybody has a suggestion on how to run x264 without the destructive y4m color transform, I'd appreciate it. I believe the problem is that it's putting YUV back into bytes. I suspect that the color consersions and up/down sample are just not being done very well (I'm using ffmpeg) , maybe there's some -highquality setting that I don't know about that would make them better?

In any case, we can still see a few things. JPEG behaves very badly below 0.5 bpp which x264 degrades nicely into very low bit rates.

x264 vs JPEG : scielab SSIM :

And another point on graph scaling. SSIM is a dot product, and as such is nonlinear and distortion. In particular, in the functional range, SSIM values are usually between 0.980 and 0.990 , which is a tiny and confusing space.

One better way to map it is to turn the dot product into an angle (using acos). I then change the angle into a percent (100% = 0 degrees apart, 0% = 90 degrees apart). The "SSIM angle" plot looks like this :

scielab SSIM angle :

In particular, it's slightly easier to see the separation between normal JPEG and the "flatnosub" (RMSE tuned) JPEG in the SSIM angle plot than in the regular SSIM plot. In regular SSIM everybody goes into this asymptote to 1 together and it makes a mess. This scielab MS-SSIM is semi-perceptual so it rewards JPEG over jpeg-flat-nosub.

You should basically ignore the x264 results here because of the aforementioned problem with large constant error.

ADDENDUM : you can repro the x264 problem like this :

ffmpeg -i my_soup.avi -vcodec libx264 -fpre presets\libx264-lossless_slow.ffpreset test.mkv
ffmpeg -i test.mkv -vcodec png out.png

imdiff my_soup.bmp out.png

and the result is :

rmse : 5.7906 , psnr : 32.9104
ssim : 0.9783 , perc : 86.7096%
scielab rmse : 2.755
scielab ssim : 0.9928 , angle : 92.3736%

or without even getting x264 involved :

ffmpeg -i my_soup.avi -pix_fmt yuv420p my_soup.y4m
ffmpeg -i my_soup.y4m -vcodec png uny4m.png

same results, which shows that the problem is the yuv420 conversion

I confirmed that my_soup.avi is in fact a perfect lossless RGB copy of my_soup.bmp ; Note that this result is even worse than the one I reported above. The one above was made by first converting the AVI to Y4M using Mencoder , so apparently that path is slightly higher quality than whatever ffmpeg is doing here.

ADDENDUM : I think ffmpeg/x264 use "broadcast standard" YUV in the 16-235 range instead of 0-255 , so that might be a large piece of the problem.

latest attempt, still not good :

ffmpeg -f image2 -i my_soup.bmp -sws_flags +accurate_rnd+full_chroma_int -vcodec libx264 -fpre c:\progs\video_tools\ffmpeg-latest\presets\libx264-lossless_slow.ffpreset test.mkv
ffmpeg -i test.mkv -sws_flags +accurate_rnd+full_chroma_int -vcodec png uny4m_2.png
imdiff my_soup.bmp uny4m_2.png

10-12-10 - Image Comparison Part 1

First of all, let's talk about how we graph these things. For this comparo, I'm measuring RGB L2 error (aka MSE) using IJG JPEG. I'm comparing :

    jpeg : default settings (with -optimize)
    jpegflat : jpeg + flat quantization matrix
    jpegflatnosub : jpegflat + no subsample for chroma

what people usually plot is PSNR vs. bpp, which looks like this :

jpegs psnr vs bpp :

these psnr vs bpp graphs are total shit. I can't see anything. The area that you actually care about is around 0.5 - 1.5 bpp, and it's all crammed together. In particular it's impossible to tell if jpeg vs jpegflat is better, and I have no intuition for what the numbers mean. Please stop making these graphs right now.

(NOTE : bpp means bits per *pixel* not bits per byte; eg. uncompressed is 24 bpp)

IMO rmse is better than PSNR because it's more intuitive. 1 level of rmse is one pixel value, it's intuitive. Unfortunately, the plot is only slightly better :

jpegs rmse vs bpp :

What we want obviously is to expand the area around 1 bpp. The obvious thing should occur to - our bpp scale is wrong. In particular, what we really care about are doublings of bpp - eg. 0.25 bpp, 0.5 bpp, 1.0 bpp, 2.0 bpp - the step from 0.25 to 0.50 is about the same importance as the step from 4.0 bpp to 8.0 bpp. Obviously we should use a log scale. A similar argument applies to rmse, so we have :

jpegs log rmse vs log bpp :

which is much clearer. It also is amazingly linear through the "money zone" of -1 to 1 (0.5 to 2.0 bpp) which is where JPEG performs well.

BTW note of course PSNR is a log scale as well, it's just log rmse flipped upside down and then offset all weird by some constant. I don't like it as much, but PSNR vs. log bpp is okay :

PSNR vs log bpp :

Conclusion :

Plots vs log bpp are the way to go.

If you are showing L2 errors for JPEG you need to be using a flat quantization matrix with no subsampling of chroma. (note that I didn't optimize the relative scaling of the planes, which might change the results or improve them further).

Next post I'll move on to some perceptual measurements.

(BTW the JPEG numbers posted here are all with PAQ ; see later posts for full details)

10-11-10 - FFmpeg success

I've never gotten ffmpeg to work because I could never find a win32 build with everything integrated in it properly (for the win32 build to work easily you need to statically build in mingw, libavcodec, etc.). Anyway, I finally found one :

Automated FFmpeg Builds at arrozcru

And in particular it's got a mess of x264 presets included so you don't have to try to figure out that command line. In particular I can now do :

echo USAGE : [in] [speed] [out]
echo speed = medium,slow,fast,slower,etc.
%FFMPEG%\bin\ffmpeg -i %1 -threads 0 -acodec copy -vcodec libx264 -fpre %FFMPEG%\presets\libx264-%2.ffpreset %3

(my latest fucking video Babel problem is that god damn Youtube doesn't offer mp4 options for many videos anymore, they are all FLV, and my HTPC can't play FLV, so I have to convert youtubes friggle fracking frizzum). (BTW you can also change containers sometimes by using -sameq and not specifying anything else)

Some more reference :

ffmpeg.org FFmpeg FAQ
ffmpeg.org FFmpeg Documentation
VideoAudio Encoding Cheat Sheet for FFmpeg
Links - FFmpeg on Windows
FFmpeg x264 encoding guide robert.swain

BTW any time you are trying a format change pathway it's a good idea to check sync with something like this video

ADDENDUM : Well, as I predicted the Unicode Fuckitude on Windows command lines is coming true. FFmpeg won't open some videos with weird characters in file names, and Youtube in fact has a bunch of videos with weird chars in their names. The simplest solution is to run my Deunicode on your download directory periodically. Deunicode actually still allows 8-bit characters, I might make a -ascii option to restrict it even further to just 7 bit ascii. (8 bit characters work consistently on Windows but still have the annoying attribute that they may display differently on the CLI vs in Explorer).

BTW the problem is not really FFMpeg , it's the fucked up Windows Console CP thing. ( see previous ). The only way to make command line apps that work with unicode file names in Windows is to do the elaborate "GetUnicodeFileNameFromMatch" thing in cblib that I do. So far as I know I'm the only person in the whole world who actually does this (I guess not that many people actually use the command line in windows any more).

10-11-10 - DeUnicode v1.0

DeUnicode v1.0 is up at : Binaries on cbloom.com .

usage :

DeUnicode v1.0 by cbloom
usage:
 DeUnicode [-opts] < dir >
-r : recurse
-t : special the
-a : ascii mode
-d : display only (don't do)
-q : quiet
-v : verbose

I highly recommend using -d for a while at first to make it sure it's working right for you.

For some apps that don't handle even OEM / Console / "A" code page issues right, there's now a "-a" option to make the names into 7 bit ascii , which everyone should work with.

Now if only I could run this on the entire internet ... my god I can't believe they let URL's be non-ascii...

---

BTW I changed my license (bpl.txt) from BSD to Zlib since it's more permissive.

10-11-10 - Windows 1252 to ASCII best fit

I'd like to construct a Windows 1252 to ASCII (7 bit) best fit visual character mapping (eg. accented a -> a , vertical bar linedraw -> | , etc.). I can't find it. ... okay I did it ..

const int c_windows1252_to_ascii[256] = 
{
  0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63,
 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79,
 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95,
 96, 97, 98, 99,100,101,102,103,104,105,106,107,108,109,110,111,
112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,
 35,102, 34, 46, 35, 35, 94, 35, 83, 60, 79, 35, 90, 35, 90, 35,
 35, 39, 39, 34, 34, 46, 45, 45,126, 84,115, 62,111, 35,122, 89,
 32, 33, 99, 35, 36, 89,124, 35, 35, 67, 97, 60, 35, 45, 82, 35,
 35, 35, 50, 51, 35, 35, 35, 46, 44, 49,111, 62, 35, 35, 35, 35,
 65, 65, 65, 65, 65, 65, 65, 67, 69, 69, 69, 69, 73, 73, 73, 73,
 68, 78, 79, 79, 79, 79, 79, 35, 79, 85, 85, 85, 85, 89, 35, 35,
 97, 97, 97, 97, 97, 97, 97, 99,101,101,101,101,105,105,105,105,
 35,110,111,111,111,111,111, 35,111,117,117,117,117,121, 35,121
};

( see this table as visible chars at cbloom.com )

this was generated by the Win32 functions and it's not perfect. It gets the accented chars right but it just gives up on the funny chars and it puts the "default" char in, which I set to "#" (35) which is probably not the ideal choice.

So anyway, it would be better to have a hand-tweaked table if you can find it.

Some links I found that were not particularly helpful :

Index of PublicMAPPINGSVENDORSMICSFTWindowsBestFit
Character sets
Cast of Characters- ASCII, ANSI, UTF-8 and all that
ASCII Table 7-bit
ASCII Character Map
ANSI character set and equivalent Unicode and HTML characters
A tutorial on character code issues

---

Also, in further news of the printf with wide chars considered harmful front, I've discovered it can cause execution to break, not merely fail to convert the string well.

I get some wchar string from some perfectly reasonable source (such as MultiBytetoWideChar or from a file name) and try to print it with printf %S (capital S for wide chars). The problem is at this point (output.c in the MSVC CRT) :

    e = _WCTOMB_S(&retval, L_buffer, _countof(L_buffer), *p++);
    if (e != 0 || retval == 0) {
        charsout = -1;
        break;
    }

because it's decided that the wchar is no good for some reason. wctomb_s will fail "if the conversion is not possible in the current locale". It winds up failing the whole printf (which causes it to return -1 and set errno). WTF don't fail my entire printf because you can't map one of the wchars. So fucked.

(I also have no clue why this particular wchar was failing to convert; it was like a squiggly f looking thing, it showed up just fine in the MSVC watch window, but for some reason the CRT locale shit didn't like it).

see :

_set_invalid_parameter_handler (CRT)
_setmbcp (CRT)
setlocale, _wsetlocale (CRT)

Anyway, my recommended best practice remains "don't use wide chars in printf" , unless you use autoprintf and let it convert them to console code page for you. (note that wstrings are converted automatically, but raw wchars you have to call ToString() to make them convert)

If you use autoprintf and put this somewhere, it will handle the std string variants nicely :

START_CB
inline const char * autoprintf_StringToChar (const std::string & rhs)
{
    return rhs.c_str();
}

inline const String ToString( const std::wstring & rhs)
{
    return autoPrintfWChar(rhs.c_str());
}
END_CB

10-09-10 - Game Controls

I'm playing a bit of Xbox 360 for the first time ever on the loaner box from work (side note : jebus it makes a hell of a lot of racket, the DVD is crazy loud, the fans are crazy loud, it's pretty intolerable; I also get disc unreadable errors semi-frequently which I guess is because the work box is screwed, but between this and the Red Ring problems, I can only conclude that this console is built like a piece of shit, just epically bad manufacturing quality).

Anyway, one thing that I find very depressing is that the games have almost uniformly bad basic controls. This is the most fundamental aspect of any game and you all should be ashamed that you don't do it right.

Perhaps the most frustrating and unforgivable of them is that so many games just drop inputs. This happens because the game or the player is in some kind of state where you are not allowed to do that action. The result is that they just lose the button press. eg. say you have a jump and an attack. You can't do them at the same time. I jump, and then right about the time that I'm landing I hit attack. That needs to work whether or not I hit the attack button right before I land or right after I land.

There are various ways to do this, but an easy one is to store a little history of all inputs that have been seen in the last few millis (200 ms or so is usually enough), and whether or not they have been acted upon or not. Each frame, you act not just on the new inputs that are seen that frame, but also on inputs seen recently that have not been acted upon.

For doing things like "holding down A makes you sprint" , you need to be checking for button "is down" *not* the edge event. Unbelievably, some major games do this wrong, and the result is that they can miss the "is down" event, so I'm running around holding A and I'm not sprinting. (even if you do all the processing right as above, the player might press A down during the inventory screen or something like that when you are ignoring it).

For simultaneous button press stuff, you also need to allow some slop of course. Say you want to do something when both A and B are pressed. When you see an "A down" event you need to immediately start doing the action for A to avoid latency, but then if you see a "B down" within 50 millis or whatever, then you need to cancel the A action and start an "AB simultaneous press" action.

One pretty bad trend is that lots of people are using something like Havok for physics and they run their player motion through it. They are often lazy about this and just get it working the same way one of the Havok demos did it, and the result is incredibly janky player motion, where you stutter, get caught on shit in the world, etc. There's a mistaken belief that more finely detailed collision geometry is better. Not so. As much as I dont' really love the old Quake super-slidey style of motion that feels like the world is all greased up, it's better than this stuttering and getting caught on shit. The player's collision proxy should be rotationally symmetric around the vertical axis - you shouldn't be able to get stuck on junk by rotating. A vertical lozenge is probably the best player collision proxy.

A context-dependent generic "action" button is a perfectly reasonable thing to do and lots of games do it now, *however* it should not be one of your major buttons. eg. when you are not at an "action" location, the action button needs to do something pretty irrelevant to gameplay, eg. it should not be your "jump" or "attack" button or something important. That is, just because you're at an action location, you shouldn't lose the ability to do something major that you

Automatic lockons for combat and such are okay, but you need to provide a way to get in and out of that mode, or a button to hold down to override it or something. Specifically, you should never take over control of the player's movement or camera without their having some ability to override it.

It's well known that for 1st person games you need to provide an "invert look" option, but IMO it's just as important to provide inverts for 3rd person cameras too, not just up-down but also left-right. Everyone has a different idea of what the right way for a 3rd person camera to move is, so just expose it, it's fucking trivial, do it.

It's very depressing to me that devs don't spend time on this basic shit.

It basically comes down to an incorrect philosophy of design. The incorrect philosophy is :

we have given the player a way in which it is possible for them to make the character do what they want

the correct philosophy is :

the character should at all times do what the player obviously wants it to do

and any time you fail in that, you have a major deficiency that should be fixed. Any time you see someone press a button and the character doesn't do what they wanted, don't coach the player on how to do it right - fix it so that the game just does the right thing.

---

A couple other minor points :

You have to be careful about the keys that you use in your menu system and what they do in the game world. This is a general important UI issue (on mouse-based interfaces, you should be aware of where buttons on popups lie on top of buttons underneath them - the issue is if a user clicks the popup multiple times and it disappears they may accidentally click beneath it, so don't put a "delete all my work" button under the popup).

In games in particular, if you have A = okay and B = back in your menu system, then you should make sure that if the user is hammering on "B" and comes out of the menu system and applies to the game world, that it won't fuck them.

Context-sensitive and tap/hold buttons and modifiers (eg. hold R1 changes the action of "A") are all okay for overloading, but *modal* controls are never okay. That is, if I press R1 it changes my mode and then my buttons do different things. Modal controls are absolutely awful because it removes the players ability to act from muscle memory. It means I can't just hammer on "A" in a panic and get the expected result, I might wind up doing the wrong thing and going "fuck I'm in the wrong mode". Even worse are modal controls where the mode is not displayed clearly on screen in a reliable place because the game is trying do an "immersive" HUD-less interface.

I imagine most people would agree that modal controls are terrible, but in fact many games do them. For example any time you press a button to go into a selection wheel for spell casting or dialog or something like that it's a modal control (a better way to do this is hold the button down to bring up the wheel, but really selection wheels are pretty fucking awful in general).

That is, for each action in the game there should be a unique physical player motion. eg. "fireball" should be hold L1 , dpad up, release. Actions should never move around on the wheel, they should be in a reliable place, and there should never be any conditionals in the action sequence to do a certain action.

Another type of modality is when certain actions are not allowed due to some state of the world. eg. maybe you're near friendlies so your attacks are disabled, or you're in a cut scene so you can't save. Again most games get this wrong and don't correctly acknowledge that they have created hidden modes that affect input. The modes need to be clearly displayed - often there's no reliable way to tell you are in a certain mode or not, for example a cut scene needs to be clearly separated from normal gameplay (eg. by bringing in vignetting or letterboxing or something) if it changes my ability to use controls. If I can't do my attacks or sprint or whatever, I should at least get some ackowledgement from the game that it received my input, and it's just not doing it.

10-08-10 - Optimal Baseline JPEG

One of the things we are missing is a really good modern JPEG encoder/decoder. I mentioned most of this in the WebP post, but I thought it was important enough to repeat. This would be a great project if someone wants to do it; I'd like to, I think it's actually important, not just as a fair comparison between modern coders like x264 and good old JPEG, but also because it would actually be useful to people who care about JPEG images. (eg. a common use case is you have some old jpeg and you want to decode it as well as possible.

Using normal JPEG code streams, but trying to make the encoder & decoder as good as possible, you should do something like :

Encoder :

• RDO based structure; eg. encoder is given lambda and finds optimal R/D point. Unfortunately this has to be iterative because of huffman codes, decisions in one pass affect the huffman codes for the next pass.

• A good perceptual metric to target. Maybe SSIM or x264's funny SATD activity thing, or something else.

• Trellis quantization; the JPEG-huff code block structure lends itself to trellis state optimization pretty directly.

• Better chroma subsample (aware of the up-filter).

• Quant matrix optimization for perceptual metric.

Decoder :

• Deblocking filter, or maybe the "Unblock" histogram non-filter approach or some combination.

• Luma-aided chroma upsample

• Expectation-in-bucket instead of mean-in-bucket dequantization.

• Noise reinjection , perhaps predicting where some of the zeros in the DCT should in fact be small non-zeros.

• Shape-aware deringing ; similar to camera denoisers, there's a lot of work on this in the literature.

10-07-10 - Portable CRT God Dammit

What a god damn disaster. How is it that none of us have made our old portable wrapper for the CRT? I want a standard interface to the functions that do exist, and a standard way to ask for "does this exist", eg. something like :

#if fseek64_exists

fseek64(fp,off64,set);

#else

int32 off32 = check_value_cast_throw<int32>(off64);
fseek(fp,off32,set);

#endif

Mostly it should just be a wrapper that passes through to the underlying CRT, but for some things that are platform independent (eg. string stuff, sprintfs, etc.) it could just be its own implementation.

I see that Sean has started a portable types header called sophist that gives you sized types (int16 etc) and some #defines to check to get info about your platform. That's a good start.

For speed work you'd like some more things like "size of register" and something like "intr" (an int the size of a register) (one big issue here is whether the 64 bit type fits in a register or not). Also things like "can type be used unaligned".

Obviously C99 would help a lot, but even it wouldn't be everything. You want the stuff above that tells you a bit more about your platform and exposes low-level ops a bit. You also want stuff that's at the "POSIX" library level as opposed to just the CRT, eg. dir ops & renames and truncate and chmod and all that kind of stuff.

Every time I do portability work I think "god damn I wish I just made my own portability library" but instead I don't do it and just hack enough to make the current project work. If I had just done it the clean way from the beginning I would have saved a lot of work and been happier and made something that was useful to other people. And.. I'm just doing it the hacky way yet again.

(actually Boost addresses a lot of this but is just sick over-complex and inconsistent in quality; for example Boost.Thread looks pretty good and has Win32 condition variables for example). I also just randomly found this ptypes lib which is pretty good for Win32 vs POSIX implementations of threading stuff.

10-02-10 - WebP

Well, since we've done this in comments and emails and now some other people have gone over it, I'll go ahead and add my official take too.

Basically from what I have seen of WebP it's a joke. It may or may not be better than JPEG. We don't really know yet. The people who have done the test methodology obviously don't have image compression background.

(ADD : okay, maybe "it's a joke" is a bit harsh. The material that I linked to ostensibly showing its superiority was in fact a joke. A bad joke. But the format itself is sort of okay. I like WebP-lossless much better than WebP-lossy).

If you would like to learn how to present a lossy image compressor's performance, it should be something like these :

Lossy Image Compression Results - Image Compression Benchmark
S3TC and FXT1 texture compression
H.264AVC intra coding and JPEG 2000 comparison

eg. you need to work on a good corpus of source images, you need to study various bit rates, you need to use perceptual quality metrics, etc. Unfortunately there is not a standardized way to do this, so you have to present a bunch of things (I suggest MS-SSIM-SCIELAB but that is nonstandard).

Furthermore, the question "is it better than JPEG" is the wrong question. Of course you can make an image format that's better than JPEG. JPEG is 20-30 years old. The question is : is it better than other lossy image formats we could make. It's like if I published a new sort algorithm and showed how much better it was than bubblesort. Mkay. How does it do vs things that are actually state of the art? DLI ? ADCTC ? Why should we like your image compressor that beats JPEG better than any of the other ones? You need to show some data points for software complexity, speed, and memory use.

As for the VP8 format itself, I suspect it is slightly better than JPEG, but this is a little more subtle than people think. So far as I can tell the people in the Google study were using a JPEG with perceptual quantization matrices and then measuring PSNR. That's a big "image compression 101" mistake. The thing about JPEG is that it is actually very well tuned to the human visual system (*1); that tuning of course actually hurts PSNR. So it's very easy to beat JPEG in terms of PSNR/RMSE but in fact make output that looks worse. (this is the case with JPEG-XR / HD-PHOTO for example, and sometimes with JPEG2000 ). At the moment the VP8 codec is not visually tuned, but some day it could be, and when it eventually is, I'm sure it could beat JPEG.

That's the advantage of VP8 over JPEG - there's a decent amount of flexibility in the code stream, which means you can make an optimizing encoder that targets perceptual metrics. This is also what makes x264 so good; I don't think Dark Shikari actually realizes this, but the really great thing about the predictors in the H264 I frames is not that they help quality inherently, it's that they give you flexibility in the encoder. That is, for example, if you are targetting RMSE and you don't do trellis quantization, then predictors are not a very big win at all. They only become a big win when you let your encoder do RDO and start making decisions about throwing away coefficients and variable quantization, because then the predictors give you different residual shapes, which give you different types of error after transform and quantization. That is, it lets the encoder choose what the error looks like, and if your encoder knows what kinds of errors look better, that is very strong. (it's also good just when targetting RMSE if you do RDO, because it lets the encoder choose residual shapes which are easier to code in an R/D sense with your particular transform/backend coder).

My first question when somebody says they can beat JPEG is "did you try the trivial improvements to JPEG first?". First of all, even with the normal JPEG code stream you can do a better encoder. You can do quantization matrix optimization (DCTune), you can do "trellis quantization" (thresholding output coefficients to improve R/D), you can sample chroma in various ways. With the standard code stream, in the decoder you can do things like deblocking filters and luma-aided chroma upsample. You should of course also use a good quality JPEG Encoder such as "JPEG Wizard" and a lossless JPEG compressor ( also here ). (PAQ8PX, Stuffit 14, and PackJPG all work by decoding the JPEG then re-encoding it with a new entropy encoder, so they are equivalent to replacing the JPEG entropy coder with a modern one).

(BTW this is sort of off topic, but note that the above "good JPEG" is still lagging behind what a modern JPEG would be like. Modern JPEG would have a new context/arithmetic entropy coder, an RDO bit allocation, perceptual quality metric, per-block variable quantization, optional 16x16 blocks (and maybe 16x8,8x16), maybe a per-image color matrix, an in-loop deblocker, perhaps a deringing filter. You might want a tiny bit more encoder choice, so maybe a few prediction modes or something else (maybe an alternative transform to choose, like a 45 degree rotated directional DCT or something, you could do per-region quantization matrices, etc).)

BTW I'd like to see people stop showing Luma-only SSIM results for images that were compressed in color. If you are going to show only luma SSIM results, then you need to compress the images as grayscale. The various image formats do not treat color the same way and do not allocate bits the same way, so you are basically favoring the algorithms that give less bits to chroma when you show Y results for color image compressions.

In terms of the web, it makes a lot more sense to me to use a lossless recompressor that doesn't decode the JPEG and re-encode it. That causes pointless damage to the pixels. Better to leave the DCT coefficients alone, maybe threshold a few to zero, recompress with a new entropy coder, and then when the client receives it, turn it back into regular JPEG. That way people get to still work with JPEGs that they know and love.

This just smells all over of an ill-conceived pointless idea which frankly is getting a lot more attention than it deserves just because it has the Google name on it. One thing we don't need is more pointless image formats which are neither feature rich nor big improvements in quality which make users say "meh". JPEG2000 and HD-Photo have already fucked that up and created yet more of a Babel of file types.

(footnote *1 : actually something that needs to be done is JPEG needs to be re-tuned for modern viewing conditions; when it was tweaked we were on CRT's at much lower res, now we're on LCD's with much smaller pixels, they need to do all that threshold of detection testing again and make a new quantization matrix. Also, the 8x8 block size is too small for modern image sizes, so we really should have 16x16 visual quantization coefficients).

10-01-10 - Some Data Compression Corpora We Need Badly

If somebody wants a university project, these would be nice :

1. A lossless data compression corpus that is *broad* and also *representative*. That is, there are many types of data (probably 100-1000 files), some small, some large. Importantly the type of correlation structure in the data should be very diverse (eg. not just a ton of different English text files or executables). Too many of the corpora are simply too small, and even the ones that are reasonably large are too self-redundant, they wind up not containing a sample of a certain type of data that does occur in the wild.

Finally the thing that's really missing is there should be a weighting number assigned to each file such that they are given importance based on their chance of occurance in the wild. To get these numbers you could do a few different things - download every archive on thepiratebay and sample what's inside them (this gives you a sampling of the type of files people actually put in archives), or maybe put a snooper on the internet backbone and sample the total set of all data that flies on the internet. The point is that this sampling should be based on the actual frequency of various data types, not just an ad hoc composite.

2. An image set with human quality metrics. Somebody needs to take a big set of test images (32-100), munge them in various ways by running them through various compressors (as well as other ways of damaging them that aren't well known compressors), and then get actual human visual ratings on the damaged versions. Then provide all the damaged versions (or code to produce them) with the human ratings.

If we had a test set like that, we could tweak our algorithmic approximations of human quality rating (eg. SSIM etc) until they reproduce what the actual humans say. This is not a test set for image compressors, it's a test set for image quality metric training, which is what we really need to take image compressors to the next level.