upsampling / supersampling at DVinfo.net

Frank Ladner · August 10th, 2003, 09:07 PM

I have been thinking on this for a while and wanted to see if you guys had any ideas.

After watching some footage on the Discovery channel of high-detail landscapes and backgrounds, I thought about how that is basically 720x480, which is what my camera captures at, but the detail is a lot better looking than a standard DV camera. Of course, this has a lot to do with the optics and how many pixels are being sampled up front. (Or if it is shot on film and transferred to video, it's automatically better.)

So I looked into the theory of upsampling images. Everyone knows that you can shrink a digital image with no problem, but when you blow one up, even using the best algorithms such as Lancoz(sp?), b-spline, and bicubic sampling, you'll still introduce a lot of softness to the image.

One approach for you Photoshop guys, is to take your 720x480 frame and upsize it in small increments, several times. This is better than doing it all in one step.

Now you do all your hue/saturation, curves adjustments and run the unsharp mask filter (which is easy to overdo, if you are running it on tye 720x480 image. ), since you would have more image data to manipulate.

Now when you take the final image and size back down to a 720x480 frame, you should have a better looking image.

Of course, this may be prohibitive since you would have to export your footage as a sequence of frames and run this one frame at a time using a Photoshop action script, but it may be worth it if you only use it on, say, landscapes.

This I have yet to try, but one thing I have done was to take a scanned 35mm picture, add grain, and render to a 720x480 avi, and it looked like footage. (Of course, you can't do this on waterfall scenes.)

Anyhow, I'm just throwing some ideas out there. Let me know what you think.

,Frank

Gints Klimanis · August 10th, 2003, 10:07 PM

>So I looked into the theory of upsampling images. Everyone >knows that you can shrink a digital image with no problem, but >when you blow one up, even using the best algorithms such as >Lancoz(sp?), b-spline, and bicubic sampling,

Actually, image shrinking suffers similar problems from aliasing which adds noise to the picture.

None of these upsampling algorithms listed above are particularly good. They are linear combinations of a small number of pixels and thus are popular because they are computationally simple for the job they do. Since few samples are required, the phase shift is also small. Larger filters will shift the image by 2 or more pixels, requiring extra processing to restore the result. Ideally, you would like to use a 2 dimensional sinc function sin(x)/x interpolation during the resampling, although this computationally intensive algorithm will take more time. However, at least 8 or so points are needed before the polynomial interpolators are bested.
With even more points, the polynomial interpolators fall behind
quickly.

>you'll still introduce >a lot of softness to the image.

If someone showed you the frequency response of these interpolators, you would gag. There will always be some softness as you zoom in because you are trying to view absent information in a higher resolution format. However, these algorithms readuce the original information.

Since I work in audio, I'll use an audio example. Say you have a telephone recording, which has a sampling rate of 8000 Hz,
a theoretical bandwidth of 4000 Hz. (However, the real bandwidth is somewhat smaller). Even if you record your telephone line on a 48000 Hz digital recorder and play it back on a fantastic stereo system, the recording will have no more high frequency content than that in the original recording. If you want to make it brighter, you can use an equalizer to boost frequencies
below 4000 Hz. The equalizer won't work above 4000 Hz because there are no frequencies present. However, if one of the bicubic methods were used to convert the 8000 Hz sampling rate telephone signal to a sampling rate of 48000 Hz, frequencies in the 2000-4000 range would be tapered off, resulting in a recording that is singificantly duller than the original. Although audio and video signals enter our bodies using different sensors,
the signals are processed in a similar manner.

>One approach for you Photoshop guys, is to take your 720x480 >frame and upsize it in small increments, several times. This is >better than doing it all in one step.

This is disturbing. If Photoshop users are doing multi-pass resampling, then Adobe should offer higher order interpolators so that only one step is needed. They are wasting your time.

Vladimir Koifman · August 11th, 2003, 05:20 AM

Can it be that Discovery channel cameras do not have optical anti-aliasing? Then what you see is many aliasing artifacts from the details that can not be completely resolved. Visually it could produce detail-reach picture, even though there is no actual resolution there.
On DV camera optical anti-aliasing basically removes all the details that can not be resolved by sensor. Mathematically such an approach is more correct, but it's percepted as a lack of details. Discovery might employ mathematically incorrect, but visulaly more pleasing effects.

Rob Lohman · August 11th, 2003, 05:34 AM

I can't imagine how upsampling an image and then down-sampling
it will INCREASE the quality. It may alter the look of the image to
something that you find more interesting.

I've seen some good results upsampling images using fractal
algorithms. Although these are very very time consuming
algorithms.

Frank Ladner · August 11th, 2003, 06:25 AM

Rob: I was thinking that not just upsampling, then downsampling, but upsampling, adjusting, then downsampling would be better, since curves adjustments, etc would have more pixels to work with when the image is larger. True, it probably wouldn't increase the quality, but maybe make it appear to look better.

Vladimir: I haven't heard a lot about optical anti-aliasing. I will look into this more.

Gints: Very thorough answer there. Thanks for explaining that!

Jeff Donald · August 11th, 2003, 06:31 AM

I've seen the technique Frank is talking about. It works very effectively in still photography. But it really is just tricking the mind and eye into perceiving more detail than what is actually in the image. I don't think it would be a viable tool for video production because of the length rendering times.

Frank Ladner · August 11th, 2003, 06:46 AM

Jeff: Yes, it would be very cumbersome to run this process on each frame. It would maybe be something you would do with just a particular piece of footage, like a landscape. Even that would be time consuming, though.

I was wondering - when DV footage is transferred to film, what sort of process do they run on the footage to make it look better when it is blown up?

Jeff Donald · August 11th, 2003, 07:04 AM

I would suspect they use unsharp mask to make the images appear sharper to the eye. There are some new sharpening filters (software) coming soon that should advance the quality and speed of operation.

The Discovery Channel also does a great deal of computer animation and the realism is amazing on some of the scenes.

Ron Johnson · August 11th, 2003, 09:54 AM

There are also fractal-based image 'magnification' techniques used in digital photography. I don't recall any specific products, but a few years ago when CCD pixels were very expensive, these programs seemed to be popular.

Frank Ladner · August 11th, 2003, 11:18 AM

I have heard complaints about how DVDs are being over sharpened, showing the white/black 'ringing' effect on the bigger TVs.

Jeff Donald · August 11th, 2003, 02:14 PM

Most viewers set the sharpness control on their TV's to the highest setting. This creates many visual problems for DVD's and other low resolution mediums. It should really be set to the lowest setting.

Gints Klimanis · August 11th, 2003, 02:58 PM

>I've seen the technique Frank is talking about. It works very >effectively in still photography. But it really is just tricking the >mind and eye into perceiving more detail than what is actually in >the image. I don't think it would be a viable tool for video >production because of the length rendering times.

Although I'm new at videography, I used to experiment with image processing. I noticed that some aliasing during resizing, expecially in black&white images, appeared preferable to none at all. The aliasing is most pronounced around edges (which are high frequencies), and so this noise results in an increase in image sharpness or more information around the edges.

Gints Klimanis · August 11th, 2003, 03:00 PM

For video, I suspect this aliasing noise around the edges would animate (like that found in line "jaggies") and thus be as undesirable.

Gints Klimanis · August 11th, 2003, 03:16 PM

>Rob: I was thinking that not just upsampling, then >downsampling, but upsampling, adjusting, then downsampling >would be better, since curves adjustments, etc would have >more pixels to work with when the image is larger. True, it >probably wouldn't increase the quality, but maybe make it >appear to look better.

Frank,

Provided the filters used to upsample and downsample are pretty good, more image fidelity would be preserved if upsampled, processed, then downsampled. Operating at higher sampling rates allow more bandwidth for out-of-band or aliased components. Non-linear effects such as color curves often introduce out-of-band components. Most video processors accept these artifacts as a fact of life. However, much of that could be reduced by processing at a higher samping rate to reduce the aliased energy by spreading it into the upper frequencies. These undesirable components which would then be filtered out when the signal is downsampled. Without the extra bandwidth, the aliasing components spill into the baseband and ccould never be eliminated. Generally, these aliasing components are part of random, animated grunge one sees at edges.

Audio signal processing has found significant quality improvements by operating at higher sampling frequencies.
Basic guitar distortion called waveshaping is quite similar to video's color curves. In Waveshaping, a non-linear curve is applied to the audio signal to remap it. The effect is harmonic generation and the results are usually easy to hear.

Frank Ladner · August 12th, 2003, 11:29 AM

Gints: That is very interesting!

In theory, if an image was converted to, say, 48 bits, manipulated, and then dropped to 24, would that work better than manipulating in 24 bit mode, or would the footage have had to originate in 48 bit mode?

I know a lot of the newer applications are taking advantage of 48 bit editing.

August 10th, 2003, 09:07 PM	#1
Frank Ladner Major Player Join Date: Aug 2003 Location: Poplarville, MS Posts: 453	upsampling / supersampling I have been thinking on this for a while and wanted to see if you guys had any ideas. After watching some footage on the Discovery channel of high-detail landscapes and backgrounds, I thought about how that is basically 720x480, which is what my camera captures at, but the detail is a lot better looking than a standard DV camera. Of course, this has a lot to do with the optics and how many pixels are being sampled up front. (Or if it is shot on film and transferred to video, it's automatically better.) So I looked into the theory of upsampling images. Everyone knows that you can shrink a digital image with no problem, but when you blow one up, even using the best algorithms such as Lancoz(sp?), b-spline, and bicubic sampling, you'll still introduce a lot of softness to the image. One approach for you Photoshop guys, is to take your 720x480 frame and upsize it in small increments, several times. This is better than doing it all in one step. Now you do all your hue/saturation, curves adjustments and run the unsharp mask filter (which is easy to overdo, if you are running it on tye 720x480 image. ), since you would have more image data to manipulate. Now when you take the final image and size back down to a 720x480 frame, you should have a better looking image. Of course, this may be prohibitive since you would have to export your footage as a sequence of frames and run this one frame at a time using a Photoshop action script, but it may be worth it if you only use it on, say, landscapes. This I have yet to try, but one thing I have done was to take a scanned 35mm picture, add grain, and render to a 720x480 avi, and it looked like footage. (Of course, you can't do this on waterfall scenes.) Anyhow, I'm just throwing some ideas out there. Let me know what you think. ,Frank

August 11th, 2003, 05:34 AM	#4
Rob Lohman RED Code Chef Join Date: Oct 2001 Location: Holland Posts: 12,514	I can't imagine how upsampling an image and then down-sampling it will INCREASE the quality. It may alter the look of the image to something that you find more interesting. I've seen some good results upsampling images using fractal algorithms. Although these are very very time consuming algorithms. __________________ Rob Lohman, visuar@iname.com DV Info Wrangler & RED Code Chef Join the DV Challenge \| Lady X Search DVinfo.net for quick answers \| Buy from the best: DVinfo.net sponsors

August 11th, 2003, 06:31 AM	#6
Jeff Donald Warden Join Date: Mar 2002 Location: Clearwater, FL Posts: 8,287	I've seen the technique Frank is talking about. It works very effectively in still photography. But it really is just tricking the mind and eye into perceiving more detail than what is actually in the image. I don't think it would be a viable tool for video production because of the length rendering times. __________________ Jeff Donald Carpe Diem Search DVinfo.net for quick answers \| Where to Buy? From the best in the business: DVinfo.net sponsors

August 11th, 2003, 07:04 AM	#8
Jeff Donald Warden Join Date: Mar 2002 Location: Clearwater, FL Posts: 8,287	I would suspect they use unsharp mask to make the images appear sharper to the eye. There are some new sharpening filters (software) coming soon that should advance the quality and speed of operation. The Discovery Channel also does a great deal of computer animation and the realism is amazing on some of the scenes. __________________ Jeff Donald Carpe Diem Search DVinfo.net for quick answers \| Where to Buy? From the best in the business: DVinfo.net sponsors

August 11th, 2003, 09:54 AM	#9
Ron Johnson Regular Crew Join Date: Jul 2002 Location: Portland, Oregon Posts: 103	There are also fractal-based image 'magnification' techniques used in digital photography. I don't recall any specific products, but a few years ago when CCD pixels were very expensive, these programs seemed to be popular. __________________ Ron Johnson Portland, OR

August 10th, 2003, 10:07 PM	#2
Gints Klimanis Inner Circle Join Date: Jun 2003 Location: San Jose, CA Posts: 2,222	>So I looked into the theory of upsampling images. Everyone >knows that you can shrink a digital image with no problem, but >when you blow one up, even using the best algorithms such as >Lancoz(sp?), b-spline, and bicubic sampling, Actually, image shrinking suffers similar problems from aliasing which adds noise to the picture. None of these upsampling algorithms listed above are particularly good. They are linear combinations of a small number of pixels and thus are popular because they are computationally simple for the job they do. Since few samples are required, the phase shift is also small. Larger filters will shift the image by 2 or more pixels, requiring extra processing to restore the result. Ideally, you would like to use a 2 dimensional sinc function sin(x)/x interpolation during the resampling, although this computationally intensive algorithm will take more time. However, at least 8 or so points are needed before the polynomial interpolators are bested. With even more points, the polynomial interpolators fall behind quickly. >you'll still introduce >a lot of softness to the image. If someone showed you the frequency response of these interpolators, you would gag. There will always be some softness as you zoom in because you are trying to view absent information in a higher resolution format. However, these algorithms readuce the original information. Since I work in audio, I'll use an audio example. Say you have a telephone recording, which has a sampling rate of 8000 Hz, a theoretical bandwidth of 4000 Hz. (However, the real bandwidth is somewhat smaller). Even if you record your telephone line on a 48000 Hz digital recorder and play it back on a fantastic stereo system, the recording will have no more high frequency content than that in the original recording. If you want to make it brighter, you can use an equalizer to boost frequencies below 4000 Hz. The equalizer won't work above 4000 Hz because there are no frequencies present. However, if one of the bicubic methods were used to convert the 8000 Hz sampling rate telephone signal to a sampling rate of 48000 Hz, frequencies in the 2000-4000 range would be tapered off, resulting in a recording that is singificantly duller than the original. Although audio and video signals enter our bodies using different sensors, the signals are processed in a similar manner. >One approach for you Photoshop guys, is to take your 720x480 >frame and upsize it in small increments, several times. This is >better than doing it all in one step. This is disturbing. If Photoshop users are doing multi-pass resampling, then Adobe should offer higher order interpolators so that only one step is needed. They are wasting your time.

August 11th, 2003, 05:20 AM	#3
Vladimir Koifman Regular Crew Join Date: May 2003 Location: Israel Posts: 115	Can it be that Discovery channel cameras do not have optical anti-aliasing? Then what you see is many aliasing artifacts from the details that can not be completely resolved. Visually it could produce detail-reach picture, even though there is no actual resolution there. On DV camera optical anti-aliasing basically removes all the details that can not be resolved by sensor. Mathematically such an approach is more correct, but it's percepted as a lack of details. Discovery might employ mathematically incorrect, but visulaly more pleasing effects.

August 11th, 2003, 06:25 AM	#5
Frank Ladner Major Player Join Date: Aug 2003 Location: Poplarville, MS Posts: 453	Rob: I was thinking that not just upsampling, then downsampling, but upsampling, adjusting, then downsampling would be better, since curves adjustments, etc would have more pixels to work with when the image is larger. True, it probably wouldn't increase the quality, but maybe make it appear to look better. Vladimir: I haven't heard a lot about optical anti-aliasing. I will look into this more. Gints: Very thorough answer there. Thanks for explaining that!

August 11th, 2003, 06:46 AM	#7
Frank Ladner Major Player Join Date: Aug 2003 Location: Poplarville, MS Posts: 453	Jeff: Yes, it would be very cumbersome to run this process on each frame. It would maybe be something you would do with just a particular piece of footage, like a landscape. Even that would be time consuming, though. I was wondering - when DV footage is transferred to film, what sort of process do they run on the footage to make it look better when it is blown up?

August 11th, 2003, 11:18 AM	#10
Frank Ladner Major Player Join Date: Aug 2003 Location: Poplarville, MS Posts: 453	I have heard complaints about how DVDs are being over sharpened, showing the white/black 'ringing' effect on the bigger TVs.

August 11th, 2003, 02:14 PM	#11
Jeff Donald Warden Join Date: Mar 2002 Location: Clearwater, FL Posts: 8,287	Most viewers set the sharpness control on their TV's to the highest setting. This creates many visual problems for DVD's and other low resolution mediums. It should really be set to the lowest setting. __________________ Jeff Donald Carpe Diem Search DVinfo.net for quick answers \| Where to Buy? From the best in the business: DVinfo.net sponsors

August 11th, 2003, 02:58 PM	#12
Gints Klimanis Inner Circle Join Date: Jun 2003 Location: San Jose, CA Posts: 2,222	>I've seen the technique Frank is talking about. It works very >effectively in still photography. But it really is just tricking the >mind and eye into perceiving more detail than what is actually in >the image. I don't think it would be a viable tool for video >production because of the length rendering times. Although I'm new at videography, I used to experiment with image processing. I noticed that some aliasing during resizing, expecially in black&white images, appeared preferable to none at all. The aliasing is most pronounced around edges (which are high frequencies), and so this noise results in an increase in image sharpness or more information around the edges.

August 11th, 2003, 03:00 PM	#13
Gints Klimanis Inner Circle Join Date: Jun 2003 Location: San Jose, CA Posts: 2,222	For video, I suspect this aliasing noise around the edges would animate (like that found in line "jaggies") and thus be as undesirable.

August 11th, 2003, 03:16 PM	#14
Gints Klimanis Inner Circle Join Date: Jun 2003 Location: San Jose, CA Posts: 2,222	>Rob: I was thinking that not just upsampling, then >downsampling, but upsampling, adjusting, then downsampling >would be better, since curves adjustments, etc would have >more pixels to work with when the image is larger. True, it >probably wouldn't increase the quality, but maybe make it >appear to look better. Frank, Provided the filters used to upsample and downsample are pretty good, more image fidelity would be preserved if upsampled, processed, then downsampled. Operating at higher sampling rates allow more bandwidth for out-of-band or aliased components. Non-linear effects such as color curves often introduce out-of-band components. Most video processors accept these artifacts as a fact of life. However, much of that could be reduced by processing at a higher samping rate to reduce the aliased energy by spreading it into the upper frequencies. These undesirable components which would then be filtered out when the signal is downsampled. Without the extra bandwidth, the aliasing components spill into the baseband and ccould never be eliminated. Generally, these aliasing components are part of random, animated grunge one sees at edges. Audio signal processing has found significant quality improvements by operating at higher sampling frequencies. Basic guitar distortion called waveshaping is quite similar to video's color curves. In Waveshaping, a non-linear curve is applied to the audio signal to remap it. The effect is harmonic generation and the results are usually easy to hear.

August 12th, 2003, 11:29 AM	#15
Frank Ladner Major Player Join Date: Aug 2003 Location: Poplarville, MS Posts: 453	Gints: That is very interesting! In theory, if an image was converted to, say, 48 bits, manipulated, and then dropped to 24, would that work better than manipulating in 24 bit mode, or would the footage have had to originate in 48 bit mode? I know a lot of the newer applications are taking advantage of 48 bit editing.