Compression Question: From 1080 to 720. How does it work?

[Miggy Sanchez]

Hi there

When an image is compressed from 1080 to 720, how is that being achieved?

Are pixels in the 1080 source simply being thrown away to get down to 720?

Are pixels being merged?

I have no idea really.

Any info/advice appreciated as always.

Mig

[Jim Andrada]

No they don't just throw away pixels, instead they re-sample the image and calculate new pixels based on the surrounding hi res pixels. There are quite a few ways to do it - and that's pretty much where my knowledge ends.

[Seth Bloombaum]

The exact resampling method depends on the tool you're using. For example, in Sony Vegas Pro, Draft, Good, and Best choices are available when rendering. Good works fine, until you want to change the size (rescale) the video when rendering. Best should always be chosen when rescaling, it uses an advanced "bicubic" method.

Bicubic refers to some maths that I'm really not familiar with.

Yadif is the open source method generally referred to as the best rescaler. Strangely, common NLEs don't use it. However, you can get it in the freeware Handbrake, which is a very good h.264/MP4 encoder, that uses the best available methods of rescaling, deinterlacing, and MP4 encoding.

[Miggy Sanchez]

Thanks Jim and Seth for taking the time to reply. Appreciate it.

In a sense then re-sampling the image and calculating new pixels based on the surrounding hi res pixels is a kind of "merging of the pixels"?

Does downscaling result in a better quality picture? (Do I hear a "it depends"?)

Cheers.

[Bryan Worsley]

Originally Posted by Seth Bloombaum

Yadif is the open source method generally referred to as the best rescaler.

Yadif is a deinterlacer - name derivation "Yet Another De-Interlace Filter"

[Jim Andrada]

I guess you could call it "merging" but it's pretty complicated and there are a lot of different algorithms. Have to admit you're making me curious about exactly how it's done but sampling is a tough science - a big part of reading hard drives and tape drives - no they don't just write neat little 1's and 0's on the medium. It's REALLY complicated.

And yes, down rezzing can improve SOME aspects of the image under SOME conditions. But probably not ALL aspects under ALL conditions.

Damn - now I AM curious. I'll research it a little and see if I can come up with a better explanation.

Thanks for asking

[Jim Andrada]

Found this - it's a simplified story but ideas are right.

Basics of Image Resampling

[Jon Fairhurst]

For more information, search the web on "Finite Impulse Response" filters and "Fourier Transforms" as well as "resampling algorithms". I expect that there are some good tutorials and lectures out there.

And yes, downsampling can have benefits. Assuming that there is high frequency information in the signal (as compared to a gray field), the obvious downside is that you lose resolution and detail. The upside is that high-frequency noise is reduced too. And if you preserve all the bits, you can get more bit depth.

For instance, if you have a camera signal with UHD (2160p) resolution and downsample it to FHD (1080p), you are reducing the spatial information by a factor of four. That information doesn't just vanish. It shows up as higher accuracy (more bits). Consider a simple algorithm where you average four pixels to create one new, larger pixel. And consider that the values of the original (integer-only) pixels are 13, 13, 14, and 13. The result is 13.25, which is no longer an integer. You need two more bits to represent this, meaning that you have two more bits of information than you had before.

Assuming that the processing is done with good quality, there are no downsides to shooting at a higher resolution than you deliver, except that it might take more initial storage, more transfer and processing time and that you (obviously) lose resolution. On the other hand, if the processing is done badly, you could get aliasing, ringing, banding, and all sorts of artifacts. So use good quality tools. :)

[Miggy Sanchez]

Ask and ye shall receive!!!

Jim, Jon, and Bryan, thanks for the links/references...

[David Heath]

Originally Posted by Jon Fairhurst

And yes, downsampling can have benefits. Assuming that there is high frequency information in the signal (as compared to a gray field), the obvious downside is that you lose resolution and detail. The upside is that high-frequency noise is reduced too.

Not sure I agree with the conclusions. If you downsample you are almost inevitably throwing information away, so in quality terms the overall quality can only be less. The point about noise reduction is a little more complex, since you're comparing lower overall noise with larger pixels - which being larger will show up a given level of noise more......

A rough analogy may be audio, where you start off with a noisy full frequency signal. Cut off a lot of the HF and it will certainly reduce the hiss - but at the expense of the higher frequencies. Does it make the overall result "better"? That's where it starts to become subjective.....

The real benefit to downsampling is it means a smaller image, which is likely to encode to a smaller file size/data rate. That may be more important than absolute quality, and that is likely to be the reason for doing any downsample.

Originally Posted by Jon Fairhurst

And if you preserve all the bits, you can get more bit depth.

For instance, if you have a camera signal with UHD (2160p) resolution and downsample it to FHD (1080p), you are reducing the spatial information by a factor of four. That information doesn't just vanish. It shows up as higher accuracy (more bits). Consider a simple algorithm where you average four pixels to create one new, larger pixel. And consider that the values of the original (integer-only) pixels are 13, 13, 14, and 13. The result is 13.25,.........

This has come up before, and the best I would say is that it's questionable logic and may depend much on how the original 8 bit signal is derived.

To try to illustrate the point, then we have to think how the four pixels in your example are derived, and whether it's a good real world model. If we assume the input is a uniform scene whose value in the "10 bit" world is 13.25, it has to be asked how it gets sampled in the first place.

Ideally, each sample should be rounded to the nearest ("8-bit") integer, so each of the samples would then be rounded to 13, so the average will also be 13! The process of downsampling has gained no bitdepth advantage - 13.25 has still ended up as 13.

To end up with what you say implies a certain randomness in the rounding process. So that in this idealised situation, statistically 3 of every 4 samples will be rounded down by 0.25, 1 in four will be rounded up by 0.75. This is really implying noise in the system is required for the principle to hold true.

And the gotcha is that 10 bit really makes most difference in a low noise signal. If the signal is noisy in the first place, all 10 bit may do is more accurately define the noise signal! I'm not saying that this "4 8-bit samples transforms to 1 10-bit" is altogether wrong, but there's a lot more to it than is often assumed.

Originally Posted by Jon Fairhurst

Assuming that the processing is done with good quality, there are no downsides to shooting at a higher resolution than you deliver, except that it might take more initial storage, more transfer and processing time and that you (obviously) lose resolution. On the other hand, if the processing is done badly, you could get aliasing, ringing, banding, and all sorts of artifacts. So use good quality tools. :)

Yes to most of that, but there are a couple of things to consider. First is datarate of the recorded 4K signal. It's highly likely to not be 4x that of an equivalent QFHD image with an equivalent compression system, which means that for any given block of x pixels square they must be more heavily compressed in 4K than QFHD, yes? That may not normally be an issue, as normally that block will be physically far smaller in QFHD than HD - assuming the same screen size in each case - but when you start to zoom in the picture........

The final thought to consider is aliasing. If we imagine starting off with a 1080 signal, then also assume it has blocks of lines corresponding to 1000, 900, 800 700, and 600 lpph. We downconvert to 720p and we hope it will keep the last two blocks intact, but the others must be more than a 720 system can handle. A perfect system would just make them grey blocks. The alternative is to cause aliasing. And in such the aliases "wrap round" to give lower frequency banding rather than plain grey. (I won't go into the theory, but take it from me that 800 will transform to 640, 900 to 540, and 1,000 to 440 lpph in all cases.)

Not good, and even worse with movement. So a good downscaler will filter out the (too) fine detail in the image BEFORE doing the actual downscale. (Though you may not be aware of it.)

[Jon Fairhurst]

David, that's a lot to respond to. :)

Regarding "throwing information away", that's not necessarily true. If you use a good filter, you take all of the information as the input. Of course, for downsampling, we will reduce the HF content, but as I showed before, that information shows up as more bit depth. Of course, if you immediately round or truncate back to 8-bits, then you are truly throwing the information away.

One place where this fails is when you have a perfectly flat field that is noise free and already has an error. An example would be a digital cartoon with a flat sky that was supposed to be a given color of cyan, but was uniformly rounded to a different, perfectly uniform value. In this case, the information is already lost and downsampling will never get it back.

But now consider a real sky with subtle variations, and noise in the system. (Or a synthetic sky with more bits where you added dither before rounding.) If the actual value was 13.285714..., then in two sevenths of the area, the rounded pixels will be 14 and in five sevenths of the area, the rounded pixels will be 13. Dithering like this is good practice anyway as without it, shallow ramps will show contouring artifacts. With dither or natural noise, the density of the "13s" will gradually reduce and the density of the "14s" will gradually increase as the ramp level slowly grows.

With such a signal, you do indeed gain two bits of information as you downscale UHD to FHD,

Now let's say you do some color correction using a deep bit depth system (as one should). In this case, you are indeed using these extra two bits in the math. After than process, round back to 8-bits.

Except... First dither. Let's now say that you have a flat field of 13.25 values after processing. When you dither and then round, a quarter of the values in that area will have a value of 14 and three quarters will be 13. Of course, if one doesn't need to go back to 8-bits, you just keep that information.

So yes, you are correct that this doesn't work when, a) the information is already lost (such as when a noise-free signal was truncated without dithering), b) or when the bits are thrown away after filtering without dithering. But for most non-synthetic signals and well-processed synthetic signals, one has the option of doing the conversion well.

One caveat: If the original signal was heavily compressed, it might have block artifacts. These artifacts are like undithered, rounded, flat fields. In this case, the information was already lost (there's no HF detail or bit depth information), so downconversion won't get that information back. Then again for a flat field, you haven't lost any detail due to downsampling either.

Regarding the filtering, yes, one loses resolution by design. If you simply throw away pixels, you get aliasing and phase shifts. By filtering too softly (like with a cheap, weighted average filter), one loses detail. By filtering too sharply, one can get ringing. But like Baby Bear's porridge, one can select a filter that is "just right", where the edges are sharp and the ringing just on the edge of being noticeable. If your tool doesn't have a good downconversion filter, one trick is to add or reduce sharpness in the original before downconversion to produce the desired result.

To summarize good practice:
* Use a higher bit depth setting when down converting.
* Dither before rounding.
* Use a good down conversion tool

[David Heath]

Originally Posted by Jon Fairhurst

Regarding "throwing information away", that's not necessarily true. ........ Of course, for downsampling, we will reduce the HF content, but as I showed before, that information shows up as more bit depth.

Oh, I think it is reasonably accurate to say "If you downsample you are almost inevitably throwing information away......"

As far as the bitdepth point goes, then I did say "I'm not saying that this "4 8-bit samples transforms to 1 10-bit" is altogether wrong, but there's a lot more to it than is often assumed", and I stick by that. I certainly think that when downsampling, what you lose in resolution information is never compensated for in extra bitdepth, even when the circumstances are such that it works well.

Of course, it depends what the original is. It's plausible to have a (say) 1080 signal, but the information contained in it to not make use of the full capacity of the signal. As example I'd quote a camera like the HVX200. It's 960x540 chipset had a limiting resolution of about 1150x650. It could be recorded as a 1080p signal - but that didn't affect the fundamental resolution limit. And downconvert to 720p (at 1280x720) and you wouldn't see any significant softening.

But start off with a 1920x1080 chipset and it's a different matter. Downconvert to 720 and information WILL be lost - how much depending on the image itself. (In the extreme, if say a pattern of very fine lines at 900lpph, the downscaled 720p should be just a uniform grey raster!! So ALL the information is lost! :) But that's an extreme example!)

As far as dithering goes, then whilst it may be used to disguise or mask banding, then isn't it almost equivalent to saying "adding noise"? So whilst it may make the pseudo 8 to 10 bit transform work in theory, the dither is introducing a certain randomness (noise) into the sample values - so the whole point of raising bitdepth (making the exact sample value more precise) is negated?

I'm not arguing that there may be no benefit in post producing in 10 bit, even if acquisition was 8 bit, but that is a separate argument to the whole subject of saying useful bitdepth can increase with downsampling.

[Seth Bloombaum]

Originally Posted by Seth Bloombaum

...Yadif is the open source method generally referred to as the best rescaler. Strangely, common NLEs don't use it. However, you can get it in the freeware Handbrake, which is a very good h.264/MP4 encoder, that uses the best available methods of rescaling, deinterlacing, and MP4 encoding.

Originally Posted by Bryan Worsley

Yadif is a deinterlacer - name derivation "Yet Another De-Interlace Filter"

Quite right Bryan. I managed to transpose the deinterlacer and the rescaler. Handbrake *does* use state of the art methods for both, but Lanczos is the scaler, and Yadif is the deinterlacer.

[Bryan Worsley]

There is also a modified version of Yadif - YadifMod - that takes it's spatial (interpolation) predictions from an external source - typically NNEDI3, which itself can be used as an upscaler....but not downscaler.

Yadifmod - Avisynth wiki

Nnedi3 - Avisynth wiki

I use them all the time in my AVISynth processing routines. I'm not sure if YadifMod has a counterpart in ffmpeg which is what Handbrake runs on. But the AVISynth version is actually a modification of a port of the original Yadif from Mplayer (ffmpeg).

I use Spline36 for downscaling 1080p to 720p, by the way. Although considered a "sharp" resizer, it is less prone to ringing artifacts than Lanczos, at least using AVISynth resize filters.

[Jon Fairhurst]

David, you are absolutely correct that high frequency information near and above the Nyquist frequency is thrown away. Poof. Gone.

But don't discount information theory when it comes to oversampling. 1-bit audio doesn't just work in some conditions. SACDs just plain work. You can in fact trade sampling frequency for bit depth.

https://en.wikipedia.org/wiki/Oversampling
"Oversampling improves resolution..." (In this case, they mean bit depth resolution.) https://en.wikipedia.org/wiki/Oversampling#Resolution