Comparison of NLE HD to SD downconversion quality

[Jason Livingston]

Lately I've been frustrated by very soft-looking SD and DVD downconverts from my HDV projects, so I decided to do a scientific test to determine where the problem was coming from and what is the best workflow to work around it.

I took a 1920x1080 still frame test chart consisting of various patterns of horizontal, vertical, and diagonal lines of increasing frequencies, compressed it directly to HDV, and imported it into a variety of NLEs and video processing programs, then used a couple different kinds of workflows to eventually output a 720x480 16:9 MPEG2 for DVD. I'll compare the relative quality of the results I obtained.

In this comparison, 10 means "as close as possible to the original HD file", 5 means "no loss or gain from using an HD source vs. SD" (i.e. looks like the chart was shot in SD), and 1 means "looks far worse than using SD in the first place."

The software I've tested so far is: Premiere Pro 2.0 with Adobe Media Encoder (AME), Edius 3.6 with Procoder Express (PCE), VirtualDub (forgot which version), and Tsunami MPEG Encoder (tmpgenc). I would love to test FCP or Avid or other programs but I don't know anyone who uses them in my area. Note that for this comparison, HD means 1440x1080 (anamorphic HDV) and SD means 720x480 (anamorphic for 16:9 DVD). All MPEG compressions were done at CBR 9mbps which is the maximum quality a DVD can use.

Here are my results. I would like to post sample pics but I'm waiting for permission to use this test chart publically, and I need to set up a web host anyway.

Quality Scores, best to worst:

10/10 : HDV Clip import into Virtualdub, resize from HD->SD via Resize filter (method Lanczos3, Use Interlaced), export via frameserving or uncompressed temp file to Tmpgenc. The quality via this method is excellent. All lines are sharp but without jaggies or noticeable sharpening artifacts. The contrast and color balance are unchanged from the original.

9.9/10: HDV Clip recompressed near-losslessly to Canopus HQ or Cineform codec (via either PP2 or Edius, identical results), imported into Virtualdub for resizing and output to Tmpgenc as above. Virtually identical quality to the above method, just requires more disk space and time. This is an excellent option if you want to output your PP2/Edius HD timeline project for conversion to SD/DVD.

6/10: HDV Clip on Edius DV (SD) timeline, export via PCE for DVD. The image has become noticably softer and slightly brighter than the original. Also there are now black bars on the sides of the image, about 8 pixels wide each, thus the original material is now squished horizontally, everything becoming slightly taller and thinner (aspect ratio bug?). However, the luma resolution is still slightly better and the chroma much better than encoding from DV.

5/10: Camera downconvert to DV on Edius DV timeline, export via PCE. Chroma and (to a lesser extent) luma resolution has definitely been lost. After DVD encoding, chroma res is only 180x240 (4:1:0) due to the 4:1:1 DV and 4:2:0 DVD subsampling, as opposed to the 720x480 (4:4:4) that is possible when converting HDV directly to DVD. Contrast has increased slightly making the whites look a little brighter.

4.5/10: Camera downconvert to DV on PP2 DV timeline, export via AME. As with Edius, chroma resolution is only 180x240. Contrast and saturation have decreased somewhat and there is a very slight green tinge, apparently this is caused by the Adobe Media Encoder's MPEG2 codec and not by Premiere itself.

2/10: HDV Clip on PP2 HDV or SD timeline, export via AME for DVD. With PP2 there seems to be no difference between starting with an HD or SD timeline when downsizing for DVD. The image has become extremely soft, looking almost out of focus. The camera downconvert version looks significantly sharper, except maybe for chroma subsampling due to DV compression. It is also slightly darker and with less color saturation and a slight green tinge.

2/10: HDV Clip on PP2 HDV or SD timeline, export to uncompressed 720x480. I did this test to determine if the bad image quality above was caused by PP2 itself or by AME. I exported to uncompressed 720x480, thus using PP2's resampling algorithms but bypassing any effects caused by AME. The image was just as soft as in the above test, indicating that the softness problem is caused by PP2 and not by AME. The loss of contrast and saturation was not apparent, so those seem to be caused by AME.

1/10: HDV Clip on Edius HD timeline, export via PCE for DVD. Compared to the Edius SD timeline method (see above), there is now extremely bad stair-stepping and aliasing effects on horizontal and diagonal lines. It appears that sending 1080i material into PCE is the problem; either PCE uses a different (awful) resampling algorithm or it has a serious bug. When Edius does the resizing (that is, using an SD timeline project setting, as seen above), the result is not nearly as bad. This has been a known Edius/PCE bug for a long time.

1/10: HDV Clip import into Tmpgenc. Apparently Tmpegenc uses a resizing algorithm similar to PCE. Extremely bad stairstepping and aliasing. You MUST let Virtualdub do the resizing.


Conclusions: Those using Edius can obtain satisfactory results by switching to a DV timeline and outputting MPEG2 from PCE. Under no circumstances should Edius users attempt to output SD MPEG2 from an HD timeline due to severe artifacts.

Those using Premiere Pro 2 are less fortunate. PP2's resampling algorithm results in an image so soft it looks more like out-of-focus DV, whether your timeline is HD or SD. Also the Adobe Media Encoder is not very good at all, as it noticably lowers the contrast and saturation, and introduces a slight green tinge.

By far, the best DVD downconvert can be obtained by letting Virtualdub do the resizing and output to a good MPEG2 encoder like Tsunami. This requires more disk space since you have to output your Edius/PP2 project as an intermediate Canopus HQ/Cineform file, but I bet a lot of editors do this anyway, and the quality increase is significant and immediately noticable.

Please feel free to post any questions or comments about my results. Keep in mind that I'm very picky when it comes to image quality, and what I consider unacceptable might seem ok to you. If you are satisfied with your workflow results, then don't throw a fit just because I called it unacceptable. However I suggest you first try the Virtualdub method and see if you like the results better before criticizing my test.

P.S. If anyone knows some special setting or patch that corrects the issues I had with PP2, please let me know and I'll re-do this test. But from what I read on the Adobe message boards, everyone else is having the same problem with no known workaround.

[Kevin Shaw]

That's right about the problems with encoding directly from an HD timeline to SD output using Edius 3.x with Procoder Express; it will be interesting to see if any improvement has been made with Edius 4 which began shipping recently. Another way around this problem is to capture and edit HD, render that out to an HD file for future use and then re-import that into a DV timeline for rendering to SD. That way you avoid problems switching back and forth from HD to SD resoluiton on the timeline, and end up with satisfactory HD and SD output.

[Thomas Smet]

I should do this test on Avid Liquid. Avid Liquid has many different ways of resizing. The classic mode can even use lancsos, gauss, box, and many other math formulas just like in Virtualdub.

[Trung Dau]

Great thread, thanks a lot Jasion

[Yletin Thuz]

I'm a newbie, so please forgives of my stupid question.

Just downloaded and Used for the very first time Virtualdub, i resize the HDV clip .AVI captured and compressed with cineform, resize it with Virtualdub to 720X480, Lanczos3 filter mode , with interlaced .
It gives me an uncompressed avi with an aspect ratio not 16:9.
O.K., i went ahead anyway converted it with TMPGEnc setting for 16:9, still the mpg output cames out with wrong aspect ratio.
Please, what did i do wrong during the resizing or encoding????
I have to admit that the output mpg stream is excellent compared with everything i went thru so far using PP2 , Vegas, CinemaCraft...

TIA

[Trung Dau]

Hi Jasion,

What Virtualdub version did u use?

[Jason Livingston]

Originally Posted by Yletin Thuz

I'm a newbie, so please forgives of my stupid question.

Just downloaded and Used for the very first time Virtualdub, i resize the HDV clip .AVI captured and compressed with cineform, resize it with Virtualdub to 720X480, Lanczos3 filter mode , with interlaced .
It gives me an uncompressed avi with an aspect ratio not 16:9.
O.K., i went ahead anyway converted it with TMPGEnc setting for 16:9, still the mpg output cames out with wrong aspect ratio.
Please, what did i do wrong during the resizing or encoding????
I have to admit that the output mpg stream is excellent compared with everything i went thru so far using PP2 , Vegas, CinemaCraft...

TIA

Make sure in Tmpgenc, under the Settings->Video tab, the aspect ratio is set to 16:9 Display, and in the Advanced tab, "Source aspect ratio" is set to 16:9 Display and "Video Arrange Method" is set to "Full screen." If your source footage is HDV interlaced then Field order should be Top Field First (or for NTSC DV its Lower Field First). That worked for me.

I don't think the settings or version of Virtualdub has any affect on the aspect ratio, but FWIW, I'm using Virtualdub-MPEG2 1.6.10 (somewhat older version).

[Jason Livingston]

My friend with Vegas (6.0c I think) ran the same test and I checked the output files for their quality relative to my tests above.

9/10: HDV clip on Vegas HD timeline, output direct to MPEG-2 for DVD, rendering quality "best." Looks nearly identical to the Virtualdub/Tmpgenc output, except for thin vertical black bars on each side, and very slightly more moire/aliasing effect, but you'd probably never notice this in real footage.

7/10: HDV clip on Vegas DV timeline, output direct to MPEG-2 for DVD, rendering quality "best." In contrast with Edius, where the DV timeline export looks much better than the HDV export, with Vegas the DV export is clearly inferior to its HDV export. Sharpness is ok, but there is a lot of moire and aliasing on the test patterns, and some stairstepping on diagonal lines. Still not too bad though.

He also tried the "good" rendering mode, but the results were noticibly inferior in both tests (subjectively scored around a 5 or 6). Sharpness was down and there were significant moire and aliasing artifacts. It might not be noticable on all footage, but to ensure the best quality, make sure your final render is set to "best" quality.

Conclusion: Vegas users who want the best DVD output possible should edit HDV/Cineform on an HD timeline and need to use "best" for the rendering quality. Done this way, the results are good enough that the extra step of using Virtualdub/Tmpgenc is probably not worth it. Edius and (especially) Premiere users should still consider the Virtualdub method because the quality is much better than what you can achieve straight from the NLE.

[Michael Y Wong]

Hi Jason

Sorry to dig up this thread but I have a question. I am giving this a try and you state:

10/10 : HDV Clip import into Virtualdub, resize from HD->SD via Resize filter (method Lanczos3, Use Interlaced), export via frameserving or uncompressed temp file to Tmpgenc. The quality via this method is excellent. All lines are sharp but without jaggies or noticeable sharpening artifacts. The contrast and color balance are unchanged from the original.

I am new to virtualDub but when u say import HDV clip to virtualDub, I output my final products to tape (m2t) and i tried importing to the m2t to virtualdub but it doenst work. When you say HDV do you mean output from pp2.0 as a 25mb/s mpeg2 file? What I did iinstead was took my final product m2t->cineframe hd (highest quality) then plugged that into virtualdub, which seemed to wrok fine.

For some reason I cant get frameserving to work (i am new to virtualdub/tmpgenc so I am just outputting to 720x480 avi (from virtualdub), I am assuming that this is the uncompressed temp file u are reffering to.

Am I on the right track? Thanks a LOT, this thread has been super helpful.

Michael

Also, is it just me or is virtual dub unforuntately NOT support dual core processors? I cant find anywhere in teh preferences so that dual core processing is exploited.

[Brent Graham]

It'd be interesting to see what After Effects does with this stuff. Any chance you'll put AE through the ringer???

[Ervin Farkas]

Quote: "10/10 : HDV Clip import into Virtualdub, resize from HD->SD via Resize filter (method Lanczos3, Use Interlaced), export via frameserving or uncompressed temp file to Tmpgenc. The quality via this method is excellent. All lines are sharp but without jaggies or noticeable sharpening artifacts. The contrast and color balance are unchanged from the original."

Jason is suggesting that frame serving is the way to go between VirtualDub and the encoder. Is there any other advantage to this other than saving hard drive real estate?

Thanks,

[Devon Lyon]

I just performed the import HDV into Virtualdub - resize and export via the methods explained above and then import into TMPGEnc and export MPEG2.

The footage is from a Canon XH-A1 at 1440x1080 and footage shot in the 24f mode. I do agree that the quality seems sharper than my normal Premier Pro 2.0 export via Adobe Media Encoder to MPEG2, but I swear I am still seeing jaggies.

Any advice on my settings? I am choosing 16:9 when exporting and have tried both interlaced and deinterlaced. Thoughts?

[Mark Miner]

Jason,

thanks for the tests and the 'recipe'. I used the 9.9 method with canopushqintermediate, vdub, and tmpgenc. This has resulted in an wonderfully nice 16:9 SD file. Much better than my old pd170.

Mark Miner

[Marc Plomp]

Hi,

I am new to this forum. I try to convert a HDV file captured in PP2 (CFHD file) with virtualdub , but Virtualdub didn't reconize the file.
How can I get this file into virtualdub?
I hope someone can help me because the qualiti from the downconversion in PP2 is bad.
Thanks

[Graham Hickling]

Virtualdub will need the VfW (Video for Windows) CFHD codec installed on your system....it comes as part of the AspectHD package so you might try (re)installing that?