How to upscale from SD to HD?

[Nikolaj Marquez von Hage]

Hi!
For my feature film, I shot some material on an ordinary, cheap consumer DV camera.
At night, to add to the damage! Anyway, to incorporate this material into my
project, I simply imported it into my native HDV project in Premiere, and zoomed in and
changed the scale horizontally somewhat to get the 4:3 image to fit the 16:9 format.

It only now occurs to me that there may exist more rigorous ways to upscale from SD to
HD. I mean, I don't know what Premiere does when I zoom in on an SD image.

Are there tools for using advanced interpolation and anti-aliasing techniques?
Can I avoid the obvious pixelation and overall less quality of my SD material as it
appears in my movie?

Thankful for any help on this.

Nikolaj

[Tripp Woelfel]

You're in a tough spot. There's no real way to create something from nothing and the low light footage will not help you.

There are tools, like Instant HD from Magic Bullet that make big claims but reports on the results are decidedly mixed. You might want to check out the current thread on Instant HD on this list.

Trying to stretch 4:3 to 16:9 will only make things worse. You'll be throwing away lines of resolution to get to widescreen, which you really cannot afford to do and retain a modicum of image quality. With iffy SD video, you might want to just pillarbox your footage.

[Nikolaj Marquez von Hage]

pillarbox - is that when you leave black areas to the left and right of the 4:3 image?

[Ervin Farkas]

Yes, that's pillarboxing. The other way around, when you import wide screen source into your 4x3 project, is called letterboxing.

If you are the type of person who doesn't mind spending some time experimenting, you may want to check out VirtualDub - it has a variety of filters, different resizing algorithms. Some work better for downscaling, some for upscaling.

After Effects has a decent grain removal filter.

[Christopher Drews]

Originally Posted by Nikolaj Marquez von Hage

Are there tools for using advanced interpolation and anti-aliasing techniques?
Can I avoid the obvious pixelation and overall less quality of my SD material as it
appears in my movie?

Adobe After Effects CS3 is your ticket. For our feature I up-converted via AE before the editing process, much of the material was low light and it did a pretty good job minimizing the distortion. I ended up crushing the blacks to loose the distortion in the black details.
-C

[Robert Martens]

Scaling up to 1080 is a bit much with my method of choice, but 720 results are certainly what I'd call respectable: http://www.youtube.com/watch?v=yDMp7O9TbsE

It is, however, fairly complicated. You'd install two applications (VirtualDub, as Ervin mentioned, and an older version of Avisynth) and more than a few plugins for them, then assemble a simple script to deinterlace and scale the footage. You'd load that script into VDub at which point you could render a final version to use in your film. Even with the multi-threading plugin--which is the reason we need an old Avisynth, the plugin hasn't been updated for the newest version just yet--I was only able to render at five or so frames per second on my quad core system.

I didn't want to waste everyone's time with a huge, detailed explanation if you don't want to bother with something this involved, but I'd be happy to detail the steps I took if you're interested. If you had a sample clip uploaded somewhere, I could try out the upscale and give you an idea what kind of quality you'd get before you commit to installing all this extra stuff.

[Tripp Woelfel]

Originally Posted by Christopher Drews

Adobe After Effects CS3 is your ticket. For our feature I up-converted via AE before the editing process, much of the material was low light and it did a pretty good job minimizing the distortion. I ended up crushing the blacks to loose the distortion in the black details.
-C

Would you mind sharing your workflow? This year I'll be producing some motor racing in HD but the in-car cameras will be SD. I'm looking for something that will upscale the 16:9 SD footage and make it look respectable next to the HD video. I have InstantHD (which I've not yet used) and CS3 (which is my constant companion).

Thx.

[Samuel Ko]

Jesus can turn water into wine.

[Nikolaj Marquez von Hage]

Robert, I'm intrigued. There's just one thing: I'm a newbie as to "uploading" stuff. If you'd give me instructions on that I'll be happy to upload my DV raw footage. It's PAL, 50i.

What I did so far is blending together three adjacent images to lessen the grain. Of course,
that makes all motion more soft and blurry...

Ervin, thanks for the tip about grain removal filter.

[Robert Martens]

If your file is less than three hundred megabytes, you can upload it to Sendspace without even signing up. Registered users can track their files' downloads, but for a quick single file upload all you'll need to do is use the simple interface you find right on that page. Click Browse to find the file, then check the box that says you agree to the terms of service, and enter robert.martens@gmail.com in the "Recipient's email" field. I get truckloads of spam as it is, I hardly mind posting my email address directly on a message board.

If it's over three hundred megs, well, Googling "free large file upload" reveals no end of file transfer services. I've never used any of them before, so I can't make any recommendations, but I'm sure one of them would work.

PAL 50i footage should be no problem; all of my stuff is NTSC 60i, but the tools I'm using require only a few simple parameter changes to work with PAL. As soon as I get the footage I can run it through and send the original back to you.

[Nikolaj Marquez von Hage]

Robert, I did as you instructed. I'm looking forward to see what can be done.
Just to be explicit: I'd like to know about a near-optimal workflow for turning
this crappy SD footage into 1440x1080 HD. If noise can be reduced in the process
that's a big bonus!

If your workflow includes de-interlacing, that is OK. My final product is 25p.

Regards,
Nikolaj

[Robert Martens]

Excellent! I just got the file and was about to ask if you had a specific format in mind for the finished product. The workflow does include deinterlacing, by way of the TempGaussMC script for Avisynth, which for the record includes denoising.

Once I finish I'll upload the file, and if you like the result I'm prepared to walk you through installing and configuring all the software necessary.

Quick edit: The file's going up to Sendspace right now, you should get the email presently. First things first, you may need to grab the XviD codec (xvid.org) to decode it. Of all the formats VirtualDub supports, XviD video in an AVI file was the best looking option at the smallest size that I could think of. Second, since it's 1440x1080, and the AVI container doesn't properly use the display aspect ratio setting in the video stream, you'll need to set the ratio of the footage yourself in whatever software you use. 16 by 9 display ratio, or 1.3333 pixel aspect.

[Nikolaj Marquez von Hage]

I have now done a frame-by-frame comparison. All I can say is WOW! That IS a greater improvement than I was really counting on. I would very much like to know how you achieved this. Thank you for sharing.
Regards,
Nikolaj

[Robert Martens]

Glad to hear it! Watch this space, because I'll edit this post with my instructions as soon as they're prepared. You'll need to give me a while, though; scatter brained as I am it sometimes takes me an hour or more to collect my thoughts and organize them so they make some kind of sense. There's more than a few steps, too, so bear with me.

Here we go! Right off the bat are the two most important pieces of this puzzle. Grab the newest version of VirtualDub and the next-to-newest version of Avisynth. VirtualDub 1.8.8 is the latest stable release, and Avisynth 2.5.7 is the latest version that will allow us to work at semi-reasonable speeds (I'll explain that, don't worry). Grab that exe installer, we don't need the source code here.

You can go ahead and install those now, because next come all the plugins we'll need to make Avisynth do what we want it to. The biggest one, that isn't a "plugin", strictly speaking, is what I linked up above, doom9 forum member Didée's TempGaussMC. There's also a "TempGaussMC_beta1mod" available, but we'll avoid that for now for reasons I'll get to later. Inside the RAR file is an Avisynth script which you'll extract to a location of your choice.

The rest are actual DLLs, are as follows, and get placed in your Avisynth 2.5\plugins directory:

MVTools 1.11.4.5 All the way at the bottom; a new version 2.0 is under development, but the TGMC script doesn't use it yet.

RemoveGrain and Repair This archive contains several files, but all we need are Repair and RemoveGrain. There are several versions of each, choose the ones appropriate for your CPU, in my case RepairSSE3 and RemoveGrainSSE3. The RAR file I've linked is "pre-release", as per this page, but is the latest version I'm aware of and fixes an SSE3 bug present in earlier releases.

MaskTools 2 Latest version is 2.0 alpha 36, but feel free to keep an eye on http://manao4.free.fr/ for updates (pay close attention to the specified dates, otherwise it's easy to get confused by the filenames). All we want from inside the package is mt_masktools-26.dll.

NNEDI 1.3 This is the deinterlacer I've chosen to use with the script, though there are others (EEDI2 and Yadif, but we won't get into all that, this is convoluted enough as is). Grab this zip file and extract nnedi.dll.

MT A combination of a plugin and a modified Avisynth DLL. It adds the ability to either spatially or temporally multi-thread Avisynth filters, by either splitting each frame into pieces or working on several frames at a time, respectively. We'll be using the latter. Install by extracting MT.dll to your plugins directory, and overwriting your existing avisynth.dll (in your windows\system32 folder) with the one in this archive. If you want, you can first rename the existing avisynth.dll to something else, maybe avisynth.old, so you can easily revert versions if you ever have problems. I never have, but it's always nice to be safe. This plugin is the reason we're using Avisynth 2.5.7. Since part of it involves a modified Avisynth, the author needs to recompile it for each version that's released. AS 2.5.8 is less than two weeks old, and that hasn't happened yet, so we stick with the previous version. It's a similar situation with the modified TGMC script that's available. There's a multi-threaded version of MVTools that can be taken advantage of by TempGaussMC_beta1mod, but I gain many more frames per second with the MT plugin using the settings I detail below, and the multi-threaded MVTools doesn't play well with the MT plugin, so I stick with the original TempGaussMC_beta1.

Once all of that is finished, the hard stuff is done. Just create an Avisynth script (regular text file with a .avs extension) that contains these lines:

SetMTmode(2,0)

Import("C:\TempGaussMC_beta1.avs")

LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\mvtools.dll")
LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\RepairSSE3.dll")
LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\mt_masktools-26.dll")
LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\nnedi.dll")

AVISource("directory\subdirectory\filename.avi")

ConvertToYV12(interlaced=true)
AssumeBFF()
TempGaussMC_beta1(EdiMode="NNEDI")
SelectEven()
Lanczos4Resize(1440,1080,0,72,720,432)

All directories, naturally, should be changed appropriately depending on where you put all the files. The first filter is the last one we installed. The SetMTmode command must be the first line in your script, and the syntax is SetMTmode(mode,threads). Mode 1 is the fastest, but doesn't work with all the plugins we're using, so mode 2 is our best choice. Setting threads to zero will use as many processors or cores as are reported by Windows.

The second, Import, is builtin to Avisynth, and allows you to bring the contents of another script into the one you're using. The next five lines may not be necessary (I haven't done extensive performance testing with this), but it never hurts to explicitly load the plugins you're using, if only for your own reference should you ever come back to this script.

Now we get to the good stuff. AVISource does exactly what it says, no surprises there. Next, since the TGMC script uses plugins that require the YV12 colorspace we need to convert the video we're loading. It's interlaced, so we make sure to say so (the conversion isn't done correctly if we forget this argument). AssumeBFF() is there since all DV footage, NTSC and PAL alike, is bottom field first, and we want to be sure it's defined as such. Another "might-not-be-necessary" line, since Avisynth usually does a good job figuring out field order, but I like to be careful.

Calling the deinterlacer is as simple as you see here, with the one change that we use NNEDI instead of the default EEDI2. I get the best results with that, though if you want to try the other options feel free.

TGMC is a bob deinterlacer, however, which means it turns fields into frames. You'll have 50 frames per second after it's finished, and since your project is 25p, the SelectEven() filter will grab every other frame and give you that. There's also a SelectOdd(), but since frames in Avisynth are numbered starting from zero, SelectEven will give you every other frame starting from the very first one.

The final piece of the puzzle is the resize. Lanczos4Resize is only one of many resizers available, and is my favorite for upscaling, but don't take my word for it. Read the Avisynth docs for more details. The syntax of this statement seems odd, I'm sure, but it's really not that bad. It effectively combines these two separate lines:

Crop(left,top,width,height)
Lanczos4Resize(width,height)

Into Lanczos4Resize(targetwidth,targetheight,cropleft,croptop,sourcewidth,sourceheight)

All AS resizers feature the expanded syntax I'm using in my script, and using it is preferable in this case. The technical details are sort of confusing, but it works differently than a standalone Crop filter, in a way that's beneficial to us. Again, the manual has all that info.

The dimensions I chose should be relatively easy to understand; we want an end result of 1440 by 1080, so we enter those numbers. Next, since the four by three square pixel equivalent of PAL's 720x576 resolution is 768x576, the sixteen by nine equivalent would be 768 by 432. That means 144 lines need to be removed from the source image, and in this case I've chosen to crop 72 off the top and 72 off the bottom. I didn't specify the bottom crop, but the combination of top crop with target picture height will implicitly define that. The Resize filter comes after the SelectEven only so we're not wasting time resizing frames we won't end up using.

That's it for the script! Save it, then open it in VirtualDub. Under the Video menu, choose Fast Recompress (since we're not using VDub for anything beyond compression, we avoid a color space change by using this option), and in the same menu choose Compression to set your compression. I used XviD for the file I sent you, set to single pass, 8000kbps.

Finish up by hitting the F7 key or choosing Save as AVI from the File menu. It'll render--slower than we'd all like, but it'll get done--and you'll have your file! If you want to render out an image sequence, you'll need to go to the Video menu again and return to Full Processing Mode, then Video->Color Depth and make sure 24 bit RGB is selected as the output format. Head over to File->Export->Image sequence, choose your file format, name, and numbering scheme, then render.

Ultimately, of course, don't forget to set the proper aspect ratio in your editing/compositing software. That should do it! The TempGaussMC_beta1.avs file has instructions inside of it that detail all of the various parameters you can change, so start reading to see how it works.

Any questions or problems, don't hesitate to ask; I'll do my best to help as soon as I can.

[Nikolaj Marquez von Hage]

I'll be patiently waiting. After working with the post for this movie on my free time for over a year, I guess a couple of hours (or days) more is something I can live with...