Hi!
For my feature film, I shot some material on an ordinary, cheap consumer DV camera.
At night, to add to the damage! Anyway, to incorporate this material into my
project, I simply imported it into my native HDV project in Premiere, and zoomed in and
changed the scale horizontally somewhat to get the 4:3 image to fit the 16:9 format.

It only now occurs to me that there may exist more rigorous ways to upscale from SD to
HD. I mean, I don't know what Premiere does when I zoom in on an SD image.

Are there tools for using advanced interpolation and anti-aliasing techniques?
Can I avoid the obvious pixelation and overall less quality of my SD material as it
appears in my movie?

Thankful for any help on this.

Nikolaj

You're in a tough spot. There's no real way to create something from nothing and the low light footage will not help you.

There are tools, like Instant HD from Magic Bullet that make big claims but reports on the results are decidedly mixed. You might want to check out the current thread on Instant HD on this list.

Trying to stretch 4:3 to 16:9 will only make things worse. You'll be throwing away lines of resolution to get to widescreen, which you really cannot afford to do and retain a modicum of image quality. With iffy SD video, you might want to just pillarbox your footage.

pillarbox - is that when you leave black areas to the left and right of the 4:3 image?

Yes, that's pillarboxing. The other way around, when you import wide screen source into your 4x3 project, is called letterboxing.

If you are the type of person who doesn't mind spending some time experimenting, you may want to check out VirtualDub - it has a variety of filters, different resizing algorithms. Some work better for downscaling, some for upscaling.

After Effects has a decent grain removal filter.

Are there tools for using advanced interpolation and anti-aliasing techniques?
Can I avoid the obvious pixelation and overall less quality of my SD material as it
appears in my movie?


Adobe After Effects CS3 is your ticket. For our feature I up-converted via AE before the editing process, much of the material was low light and it did a pretty good job minimizing the distortion. I ended up crushing the blacks to loose the distortion in the black details.
-C

Scaling up to 1080 is a bit much with my method of choice, but 720 results are certainly what I'd call respectable: http://www.youtube.com/watch?v=yDMp7O9TbsE

It is, however, fairly complicated. You'd install two applications (VirtualDub, as Ervin mentioned, and an older version of Avisynth) and more than a few plugins for them, then assemble a simple script to deinterlace and scale the footage. You'd load that script into VDub at which point you could render a final version to use in your film. Even with the multi-threading plugin--which is the reason we need an old Avisynth, the plugin hasn't been updated for the newest version just yet--I was only able to render at five or so frames per second on my quad core system.

I didn't want to waste everyone's time with a huge, detailed explanation if you don't want to bother with something this involved, but I'd be happy to detail the steps I took if you're interested. If you had a sample clip uploaded somewhere, I could try out the upscale and give you an idea what kind of quality you'd get before you commit to installing all this extra stuff.

Adobe After Effects CS3 is your ticket. For our feature I up-converted via AE before the editing process, much of the material was low light and it did a pretty good job minimizing the distortion. I ended up crushing the blacks to loose the distortion in the black details.
-C

Would you mind sharing your workflow? This year I'll be producing some motor racing in HD but the in-car cameras will be SD. I'm looking for something that will upscale the 16:9 SD footage and make it look respectable next to the HD video. I have InstantHD (which I've not yet used) and CS3 (which is my constant companion).

Thx.

Jesus can turn water into wine.

Robert, I'm intrigued. There's just one thing: I'm a newbie as to "uploading" stuff. If you'd give me instructions on that I'll be happy to upload my DV raw footage. It's PAL, 50i.

What I did so far is blending together three adjacent images to lessen the grain. Of course,
that makes all motion more soft and blurry...

Ervin, thanks for the tip about grain removal filter.

If your file is less than three hundred megabytes, you can upload it to Sendspace (http://www.sendspace.com/) without even signing up. Registered users can track their files' downloads, but for a quick single file upload all you'll need to do is use the simple interface you find right on that page. Click Browse to find the file, then check the box that says you agree to the terms of service, and enter robert.martens@gmail.com in the "Recipient's email" field. I get truckloads of spam as it is, I hardly mind posting my email address directly on a message board.

If it's over three hundred megs, well, Googling "free large file upload" reveals no end of file transfer services. I've never used any of them before, so I can't make any recommendations, but I'm sure one of them would work.

PAL 50i footage should be no problem; all of my stuff is NTSC 60i, but the tools I'm using require only a few simple parameter changes to work with PAL. As soon as I get the footage I can run it through and send the original back to you.

Robert, I did as you instructed. I'm looking forward to see what can be done.
Just to be explicit: I'd like to know about a near-optimal workflow for turning
this crappy SD footage into 1440x1080 HD. If noise can be reduced in the process
that's a big bonus!

If your workflow includes de-interlacing, that is OK. My final product is 25p.

Regards,
Nikolaj

Excellent! I just got the file and was about to ask if you had a specific format in mind for the finished product. The workflow does include deinterlacing, by way of the TempGaussMC (http://avisynth.org/mediawiki/TempGaussMC) script for Avisynth, which for the record includes denoising.

Once I finish I'll upload the file, and if you like the result I'm prepared to walk you through installing and configuring all the software necessary.

Quick edit: The file's going up to Sendspace right now, you should get the email presently. First things first, you may need to grab the XviD codec (xvid.org (http://www.xvid.org/)) to decode it. Of all the formats VirtualDub supports, XviD video in an AVI file was the best looking option at the smallest size that I could think of. Second, since it's 1440x1080, and the AVI container doesn't properly use the display aspect ratio setting in the video stream, you'll need to set the ratio of the footage yourself in whatever software you use. 16 by 9 display ratio, or 1.3333 pixel aspect.

I have now done a frame-by-frame comparison. All I can say is WOW! That IS a greater improvement than I was really counting on. I would very much like to know how you achieved this. Thank you for sharing.
Regards,
Nikolaj

Glad to hear it! Watch this space, because I'll edit this post with my instructions as soon as they're prepared. You'll need to give me a while, though; scatter brained as I am it sometimes takes me an hour or more to collect my thoughts and organize them so they make some kind of sense. There's more than a few steps, too, so bear with me.

Here we go! Right off the bat are the two most important pieces of this puzzle. Grab the newest version of VirtualDub (http://www.virtualdub.org/) and the next-to-newest version of Avisynth (http://www.avisynth.org/). VirtualDub 1.8.8 (http://prdownloads.sourceforge.net/virtualdub/VirtualDub-1.8.8.zip?download) is the latest stable release, and Avisynth 2.5.7 (http://sourceforge.net/project/showfiles.php?group_id=57023&package_id=72557&release_id=366702) is the latest version that will allow us to work at semi-reasonable speeds (I'll explain that, don't worry). Grab that exe installer, we don't need the source code here.

You can go ahead and install those now, because next come all the plugins we'll need to make Avisynth do what we want it to. The biggest one, that isn't a "plugin", strictly speaking, is what I linked up above, doom9 forum member Didée's TempGaussMC (http://home.arcor.de/dhanselmann/_stuff/TempGaussMC_beta1.rar). There's also a "TempGaussMC_beta1mod" available, but we'll avoid that for now for reasons I'll get to later. Inside the RAR file is an Avisynth script which you'll extract to a location of your choice.

The rest are actual DLLs, are as follows, and get placed in your Avisynth 2.5\plugins directory:

MVTools 1.11.4.5 (http://avisynth.org.ru/mvtools/mvtools.html) All the way at the bottom; a new version 2.0 is under development, but the TGMC script doesn't use it yet.

RemoveGrain and Repair (http://home.arcor.de/kassandro/prerelease/RemoveGrain-1.0.rar) This archive contains several files, but all we need are Repair and RemoveGrain. There are several versions of each, choose the ones appropriate for your CPU, in my case RepairSSE3 and RemoveGrainSSE3. The RAR file I've linked is "pre-release", as per this page (http://avisynth.org/mediawiki/Removegrain), but is the latest version I'm aware of and fixes an SSE3 bug present in earlier releases.

MaskTools 2 (http://manao4.free.fr/masktools-v2.0a36.zip) Latest version is 2.0 alpha 36, but feel free to keep an eye on http://manao4.free.fr/ for updates (pay close attention to the specified dates, otherwise it's easy to get confused by the filenames). All we want from inside the package is mt_masktools-26.dll.

NNEDI 1.3 (http://forum.doom9.org/showthread.php?t=129953) This is the deinterlacer I've chosen to use with the script, though there are others (EEDI2 and Yadif, but we won't get into all that, this is convoluted enough as is). Grab this zip file (http://bengal.missouri.edu/~kes25c/nnedi_v1.3.zip) and extract nnedi.dll.

MT (http://avisynth.org/mediawiki/MT) A combination of a plugin and a modified Avisynth DLL. It adds the ability to either spatially or temporally multi-thread Avisynth filters, by either splitting each frame into pieces or working on several frames at a time, respectively. We'll be using the latter. Install by extracting MT.dll to your plugins directory, and overwriting your existing avisynth.dll (in your windows\system32 folder) with the one in this archive. If you want, you can first rename the existing avisynth.dll to something else, maybe avisynth.old, so you can easily revert versions if you ever have problems. I never have, but it's always nice to be safe. This plugin is the reason we're using Avisynth 2.5.7. Since part of it involves a modified Avisynth, the author needs to recompile it for each version that's released. AS 2.5.8 is less than two weeks old, and that hasn't happened yet, so we stick with the previous version. It's a similar situation with the modified TGMC script that's available. There's a multi-threaded version of MVTools that can be taken advantage of by TempGaussMC_beta1mod, but I gain many more frames per second with the MT plugin using the settings I detail below, and the multi-threaded MVTools doesn't play well with the MT plugin, so I stick with the original TempGaussMC_beta1.

Once all of that is finished, the hard stuff is done. Just create an Avisynth script (regular text file with a .avs extension) that contains these lines:

SetMTmode(2,0)

Import("C:\TempGaussMC_beta1.avs")

LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\mvtools.dll")
LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\RemoveGrainSSE3.dll")
LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\RepairSSE3.dll")
LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\mt_masktools-26.dll")
LoadPlugin("C:\Program Files\Avisynth 2.5\plugins\nnedi.dll")

AVISource("directory\subdirectory\filename.avi")

ConvertToYV12(interlaced=true)
AssumeBFF()
TempGaussMC_beta1(EdiMode="NNEDI")
SelectEven()
Lanczos4Resize(1440,1080,0,72,720,432)

All directories, naturally, should be changed appropriately depending on where you put all the files. The first filter is the last one we installed. The SetMTmode command must be the first line in your script, and the syntax is SetMTmode(mode,threads). Mode 1 is the fastest, but doesn't work with all the plugins we're using, so mode 2 is our best choice. Setting threads to zero will use as many processors or cores as are reported by Windows.

The second, Import, is builtin to Avisynth, and allows you to bring the contents of another script into the one you're using. The next five lines may not be necessary (I haven't done extensive performance testing with this), but it never hurts to explicitly load the plugins you're using, if only for your own reference should you ever come back to this script.

Now we get to the good stuff. AVISource does exactly what it says, no surprises there. Next, since the TGMC script uses plugins that require the YV12 colorspace we need to convert the video we're loading. It's interlaced, so we make sure to say so (the conversion isn't done correctly if we forget this argument). AssumeBFF() is there since all DV footage, NTSC and PAL alike, is bottom field first, and we want to be sure it's defined as such. Another "might-not-be-necessary" line, since Avisynth usually does a good job figuring out field order, but I like to be careful.

Calling the deinterlacer is as simple as you see here, with the one change that we use NNEDI instead of the default EEDI2. I get the best results with that, though if you want to try the other options feel free.

TGMC is a bob deinterlacer, however, which means it turns fields into frames. You'll have 50 frames per second after it's finished, and since your project is 25p, the SelectEven() filter will grab every other frame and give you that. There's also a SelectOdd(), but since frames in Avisynth are numbered starting from zero, SelectEven will give you every other frame starting from the very first one.

The final piece of the puzzle is the resize. Lanczos4Resize is only one of many resizers available, and is my favorite for upscaling, but don't take my word for it. Read the Avisynth docs for more details. The syntax of this statement seems odd, I'm sure, but it's really not that bad. It effectively combines these two separate lines:

Crop(left,top,width,height)
Lanczos4Resize(width,height)

Into Lanczos4Resize(targetwidth,targetheight,cropleft,croptop,sourcewidth,sourceheight)

All AS resizers feature the expanded syntax I'm using in my script, and using it is preferable in this case. The technical details are sort of confusing, but it works differently than a standalone Crop filter, in a way that's beneficial to us. Again, the manual has all that info.

The dimensions I chose should be relatively easy to understand; we want an end result of 1440 by 1080, so we enter those numbers. Next, since the four by three square pixel equivalent of PAL's 720x576 resolution is 768x576, the sixteen by nine equivalent would be 768 by 432. That means 144 lines need to be removed from the source image, and in this case I've chosen to crop 72 off the top and 72 off the bottom. I didn't specify the bottom crop, but the combination of top crop with target picture height will implicitly define that. The Resize filter comes after the SelectEven only so we're not wasting time resizing frames we won't end up using.

That's it for the script! Save it, then open it in VirtualDub. Under the Video menu, choose Fast Recompress (since we're not using VDub for anything beyond compression, we avoid a color space change by using this option), and in the same menu choose Compression to set your compression. I used XviD for the file I sent you, set to single pass, 8000kbps.

Finish up by hitting the F7 key or choosing Save as AVI from the File menu. It'll render--slower than we'd all like, but it'll get done--and you'll have your file! If you want to render out an image sequence, you'll need to go to the Video menu again and return to Full Processing Mode, then Video->Color Depth and make sure 24 bit RGB is selected as the output format. Head over to File->Export->Image sequence, choose your file format, name, and numbering scheme, then render.

Ultimately, of course, don't forget to set the proper aspect ratio in your editing/compositing software. That should do it! The TempGaussMC_beta1.avs file has instructions inside of it that detail all of the various parameters you can change, so start reading to see how it works.

Any questions or problems, don't hesitate to ask; I'll do my best to help as soon as I can.

I'll be patiently waiting. After working with the post for this movie on my free time for over a year, I guess a couple of hours (or days) more is something I can live with...

Robert, the link for TempGaussMC links to a forum discussion where potential memory leak problems are discussed. Did I misunderstand you somehow or has the page you refer to been changed somehow? In that case, can I get my hands on TempGaussMC elsewhere? Regards,
Nikolaj

Sorry, Nikolaj, I forgot that page links to one particular post in that discussion that itself has a link to the file:

http://home.arcor.de/dhanselmann/_stuff/TempGaussMC_beta1.rar

I thought I proofread my post, guess I let one mistake slip by. I'll go fix that right away!

On the subject of deinterlacing footage without rescaling, one can simply remove the line with the Lanczos4Resize (or comment it out by adding a # to the beginning of the line).

You'll still need the SelectEven() to get the same framerate as the original source, otherwise you'll get a fields-to-frames conversion. A very good one, but it may not be what you want.

Why bother bob deinterlacing to 50p first? Why not go directly to 25p from the source footage? Because Didée's TempGaussMC only allows for bobbing, and I've yet to find another deinterlacer that works nearly as well, or with as little parameter adjustment. It produces stable images, preserves much of the original detail, and denoises the source in the process.

Other software exists that works faster, and avoids the step of generating one frame for every field only to throw half of them away later, but I've never seen anything that looks this good.

Resurrecting a thread more than year dead would normally be poor etiquette, I know, but I have something to add here.

First things first, this ignorance deserves to be corrected:

Other software exists that works faster, and avoids the step of generating one frame for every field only to throw half of them away later, but I've never seen anything that looks this good.

Some time after I posted that I actually got around to reading the Avisynth manual in depth, and it turns out that since scripts are interpreted output-to-input (back to front, bottom to top, however you want to describe it), any frames a calling application never requests are never generated. So with the procedure I detailed on the last page, you're not generating 60p and throwing away half the frames, you're only generating every other frame in the first place. No wasted CPU cycles.

But I wouldn't bother replying to such an old thread just for that. The real news is that I've designed a custom Avisynth script that will take care of most of the questions surrounding how to crop and scale DV footage to produce certain kinds of SD and HD output. It uses updated, much more stable versions of the software listed in my giganto-post from the previous page, and addresses a concern Nikolaj brought to my attention shortly after this thread first popped up: motion blur. To quote from my own description of this new script:

"TGMC is a bob (fields-to-frames) deinterlacer. Normally one wouldn't use such a tool to create same rate output, but since TGMC handles diagonals so well--and denoises so well, and does everything else so well--I've elected to employ it for that task. The problem comes in the motion blur. Producing a bobbed stream and simply following it up with a SelectEven() or SelectOdd() will give you, using NTSC as an example, 30 images per second, each of which has a shutter speed of 1/60th of a second. This is undesirable to most people, as I've been told, where the preferred result is 30 images per second that each have a 1/30th shutter speed.

I've built a compromise into this script that will bob deinterlace the input video in all cases, but when shtrfix is true pairs of frames will subsequently be merged to create a same-rate output that simulates (it's not perfect, of course) the look of footage shot with the proper exposure time. I nonetheless recommend those questing for the film look Holy Grail try setting both this and drate to false. The staccato motion rendering thus provided will give something approaching the visual impact provided by 24 frame per second material shot with a 1/48th shutter."

As I say there, the result isn't mathematically identical to footage that's actually shot 30p at 1/30th, but it looks rather convincing, and should help alleviate the stuttery nature of my prior approach, if you so desire. If you like the sound of that, take a look at the SimpleSlug (http://www.gyroshot.com/simpleslug.htm) page, or follow along with the associated tutorial (http://www.gyroshot.com/upscale1.htm) if you're not comfortable with Avisynth usage just yet.

You sir, are my hero.

Could the process be done on a win7 64bit machine?

I have no 64 bit OS with which to test any of this, but I've had reports from users that it works. I think--but don't quote me on this--you'd use the Program Files (x86) directory for installing AviSynth and the various 32 bit versions of the plugins. If you're following the tutorial on my site, the step where you replace avisynth.dll, in your WINDOWS\system32 folder, might instead require you to go to Windows\SysWOW64. I really don't know that for sure, though, so please be careful.

That's all there is to it, as far as I know, unless you're the type comfortable experimenting, in which case you might want to take a look at a 64 bit version of AviSynth 2.5.8 MT (http://forum.doom9.org/showthread.php?t=152800). No warranties from me, of course, I've got no way to toy with that. I'd stick with the 32 bit for the time being, at least to get up and running.

On another note, I didn't want to bother people with more updates to this thread, but since I'm responding I'll note that SimpleSlugUpscale has been significantly improved over the past four months, and can now turn any size and shape input into any size and shape output. It's still "simple", if you're just interested in the basic functionality, and it's still a "slug", if you use my default deinterlacing plugin, but strictly speaking it's no longer just for upscaling. Nor does it require you to know all the details of the output you desire.

The basics are introduced in the documentation (included in the script itself (http://www.gyroshot.com/files/simpleslug/SimpleSlugUpscale.avsi)), but to demonstrate, I've uploaded a few sample images. For starters, let's say you wanted output that was 640 pixels wide, and had a display aspect ratio of 2.35, but you're not sure what vertical dimension is required for that. You'd just use


SimpleSlugUpscale(outwidth=640,DARout=2.35)


and your output would be the final attachment, "outwidth640DARout235.jpg". The overlaid text is just for the purpose of demonstration, and isn't applied by my script.

Update: I only just noticed a tiny mistake in my script that renders the following italicized passage incorrect! 272 is a multiple of 16, and will now be the resultant dimension for the arguments given; at a couple of points in the script I used a Ceil(), which always rounds up, instead of a Round(). It helped certain cases, but it's not ideal for most, so it's gone. The script has silently been replaced with a corrected version, but I thought I should leave this here for reference. You'll note the vertical is 288, instead of the expected 272, and that's because by default, all output dimensions are mod 16, for most efficient compression (most video codecs, as I understand it, still use 16 pixel macroblocks). I thought that would be more important to people than perfect aspect ratio, but if you want more accuracy, you can change the 'modw' and 'modh' parameters as you see fit. Keep in mind that the minimums for progressive YV12 color are 4 and 2, respectively. If you want single pixel accuracy, you'll need to set both modw and modh to 1, then feed SimpleSlug RGB input.

The rest of the attachments demonstrate the various effects one can achieve by way of my automatic letter/pillarboxing, and are subtitled with both a description of the input and the arguments passed into the function to get each result.

Ty for another awesome reply, sir

I'd better copy this thread in case it ever disappears!

Robert, tell me something please

Why in the world is it not possible for some bright mind to combine all this stuff into one stand-alone program? I've been reading about the wonders of Avi Synth for ages, but the complexity of it frightens me; besides, I hate to install 25 pieces of individual software and make a big mess out of my computer...

Is it really not possible?

Thanks,

It's called StaxRip (http://www.stax76.bplaced.net/index.php?slab=staxrip), actually. There are many, many others (http://forum.doom9.org/forumdisplay.php?f=78), but StaxRip comes bundled with an enormous variety of utility applications, including AviSynth and a number of plugins (and, though slightly off topic, x264, the hands-down, no-contest best H.264 encoder you'll find anywhere). All packed into a directory structure, no registry changes, just extract the whole .7z archive into a folder and run Staxrip.exe.

Regarding the overall fear of AviSynth, I don't mean to sound condescending, but "suck it up and get over it" is the advice I would give myself from five years ago. I'd also give myself a new pair of glasses and a haircut, but that's beside the point.

Believe me, I had the same reaction as you to AviSynth for the longest time, but it's really nothing to be worried about. All the pieces of this puzzle take a lot more effort to describe than they do to use, and although AviSynth is powerful enough to let people write scripts as complex as SimpleSlug (many even more so), you can get results with an .avs file as barebones as:


AVISource("C:\Video\amusingfilename.avi")
AssumeBFF()
SeparateFields()
BilinearResize(1440,540)
Weave()


Just a bunch of human readable words in a text file. Here, we load a clip, assume bottom field first field order, separate the fields, resize them, then weave them back together. It's a contrived, trivial example (and will look awful from DV source, for example, since it uses only the most rudimentary scaling technique), but I think it demonstrates the point. You can make things as complex as you need to, or you can stick to the basics; the abundance of user-made scripts and plugins (http://avisynth.org/mediawiki/External_filters) makes it that much easier.

As far as making a mess of a system, AviSynth plugins are just DLLs you drop into a folder. AviSynth itself does add some registry entries upon installation, like any other software, but it uninstalls itself cleanly, and should cause no problems. Take your time, read all the instructions carefully, and you'll be fine. Try it on a home computer first if you're worried there's some risk to a production machine.

In general, I can only beg people to please make their best effort to conquer their trepidation surrounding open source software like this. I skipped over all this stuff for years thanks to the same concerns ("oh, those adorable kids and their computers; they're not real experts, like us video guys" -- I was an arrogant pain in the backside), and when I finally got around to giving it an honest shot I was red faced. We're doing ourselves and our projects a tremendous disservice by ignoring AviSynth and its contemporaries; NLEs are great at editing, compositing packages are great at pulling keys, painting mattes, wire removal and object tracking, but for preprocessing (deinterlacing, scaling, denoising, etc.) there are what should not be surprisingly powerful solutions available as plugins and scripts for AviSynth, many of which do as good or better a job than the most expensive plugins you could find for Premiere, After Effects, or any other big name suite.

I'll go put my soapbox away now, but do yourself a favor and block out an hour or two next weekend, download AviSynth and start reading the manual. You'll be over the anxiety in no time.