In the end movec-residual coding is inherently limitted and inefficient. Let's review the big advantage of it and the big problem.
The advantage is that the encoder can reasonably easy consider {movec,residual} coding choices jointly. This is a huge advantage over
just picking what's my best movec, okay now code the residual. Because movec affects the residual, you cannot make a good R/D decision if
you do it separately. By using block movecs, it reduces the number of options that need to be considered to a small enough set that
encoders can practically consider a few important choices and make a smart R/D decision. This is what is behind all current good video
encoders.
The disadvantage of movec-residual coding is that they are redundant and connected in a complex and difficult to handle way. We send them
independently, but really they have cross-information about each other, and that is impossible to use in the standard framework.
There are obviously edges and shapes in the image which occur in both the movecs and the residuals. eg. a moving object will have a boundary,
and really this edge should be used for both the movec and residual. In the current schemes we send a movec for the block, and then the
residuals per pixel, so we now have finer grain information in the residual that should have been used to give us finer movecs per pixel, but
it's too late now.
Let's back up to fundamentals. Assume for the moment that we are still working on an 8x8 block. We want to send that block in the current frame.
We have previous frames and previous blocks within the current frame to help us. There are 256^3^64 possible values for this block.
If we are doing lossy coding, then not all possible values for the block can be sent. I won't get into details of lossiness, so just say there
are a large number of possible values for the pixels of the block; we want to code an index to one of those values.
Each index should be sent with a different bit length based on its probability. Already we see a flaw with {movec-residual} coding - there are
tons of {movec,residual} pairs that specify the same index. Of course in a flat area lots of movecs might point to the same pixels, but even
if that is eliminated, you could go movec +1, residual +3, or movec +3, residual +1, and both ways get to +4. Redundant encoding = bit waste.
Now, this bit waste might not be critically bad with current simple {movec,residual} schemes - but it is a major encumbrance if we start looking
at more sophisticated mocomp options. Say you want to be able to send movecs for shapes, eg. send edges and then send a movec on each side. There
are lots of possibilities here - you might just send a movec per pixel (this seems absurdly expensive, but the motion fields are very smooth so
should code well from neighbors), or you might send a polygon mesh to specify shapes. This should give you much better motion fields, and then the
information in the motion fields can be used to predict the residuals as well. But the problem is there's too much redundancy. You have greatly
expanded the number of ways to code the same output pixels.
We could consider more modest steps as well, such as sticking with block mocomp + residual, but expanding what we can do for "mocomp". For example,
you could use two motion vectors + arbitrary linear combination of the source blocks. Or you could do trapesoidal texture-mapping style mocomp. Or
mocomp with a vector and scale + rotation. None of these is very valuable, there are numerous problems : 1. too many ways to encode for the encoder
to do thorough R/D analysis of all of them, 2. too much redundancy, 3. still not using the joint information across residual & motion.
In the end the problem is that you are using a 6-d value {velocity,pixel} to specify a 3-d color. What you really want is a 3-d coordinate which
is not in pixel space, but rather is a sort of "screw" in motion/pixel space. That is, you want the adjacent coordinates in motion/pixel space to
be the ones that are closest together in the 6-d space. So for example RGB {100,0,0} and {0,200,50} might be neighbors in motion/pixel space if
they can be reached by small motion adjustments.
Okay this is turning into rambling, but another way of seeing it is like this : for each block, construct a custom basis transform. Don't send
a separate movec or anything - the axes of the basis transform select pixels by stepping in movec and also residual.
ADDENDUM : let me try to be more clear by doing a simple example. Say you are trying to code a block of pixels which only has 10 possible
values. You want to code with a standard motion then residual method. Say there are only 2 choices for motion. It is foolish to code
all 10 possible values for both motion vectors! That is, currently all video coders do something like :
Code motion = [0 or 1]
Code residual = [0,1,2,3,4,5,6,7,8,9]
Or in tree form :
0 - [0,1,2,3,4,5,6,7,8,9]
*<
1 - [0,1,2,3,4,5,6,7,8,9]
Clearly this is foolish. For each movec, you only need to code the residual which encodes that resulting pixel block the smallest under that
movec. So you only need each output value to occur in one spot on the tree, eg.
0 - [0,1,2,3,4]
*<
1 - [5,6,7,8,9]
or something. That is, it's foolish to have to ways to encode the residual to reach a certain target when there were already cheaper ways to reach
that target in the movec coding portion.
To minimize this defficiency, most current coders like H264 will code blocks by either putting almost all the bits in the movec
and very few in the residual, or the other way (almost none in the movec and most in the residual). The loss occurs most when you have many bits
in the motion and many in the residual, something like :
0 - [0,1,2]
1 - [3,4,5,6]
2 - [7,8]
3 - [9]
The other huge fundamental defficiency is that the probability modeling of movecs and residuals is done in a very primitive way based only on
"they are usually small" assumptions. In particular, probability modeling of movecs needs to be done not just based on the vector, but on the
content of what is pointed at. I mentioned long ago there is a lot of redundancy there when you have lots of movecs pointing at the same thing.
Also, the residual coding should be aware of what was pointed to by the movec. For example if the movec pointed at a hard edge, then the
residual will likely also have a similar hard edge because it's likely we missed by a little bit, so you could use a custom transform that handles
that better. etc.
ADDENDUM 2 : there's something else very subtle going on that I haven't seen discussed much. The normal way of sending {movec,residual} is
actually over-complete. Mostly that's bad, too much over-completeness means you are just wasting bits, but actually some amount of over-completeness
here is a good thing. In particular for each frame we are sending a little bit of extra side information that is useful for *later* frames.
That is, we are sending enough information to decode the current frame to some quality level, plus some extra that is not really worth it for
just the current frame, but is worth it because it helps later frames.
The problem is that the amount of extra information we are sending is not well understood. That is, in the current {movec,residual} schemes we
are just sending extra information without being in control and making a specific decision. We should be choosing how much extra information to
send by evaluating whether it is actually helpful on future frames. Obviously the last frames of the video (or a sequence before a cut) you
shouldn't send any extra information.
In the examples above I'm showing how to reduce the overcomplete information down to a minimal set, but sometimes you might not want to do
that. As a very course example say the true motion at a given pixel is +3, movec=3 to get to final pixel=7 , but you can code the same result
smaller by using movec=1 - deciding whether to send the true motion or not should be done based on whether it actually helps in the future,
but more importantly the code stream could collapse {3,7} and {1,7} so there is no redundant way to code if the difference is not helpful.
This becomes more important of course if you have a more complex motion scheme, like per-pixel motion or trapezoidal motion or whatever.