Compressing h264 videos for archiving at DVinfo.net

David St. Juskow · April 23rd, 2010, 06:24 PM

So I've been fooling around with crunching these quicktimes down for archiving, to see how small I can get them without any noticeable compression. "Noticeable" is subjective, of course, so I'm putting out what I've figured so far in case people want to chime in.

Now, I realize that the professional answer is "how can you compress your raw footage? That's insane!" And, yes, for certain uses, I would never do that- but the reality is that this stuff takes up SPACE and it fills drives up quickly. So until I can get a 400 TB drive for $10, I'm trying to find a way to crunch the less-than-vital footage I've shot and will be shooting. The fact is, a good chunk of HD video will not be blown up to a 50 foot screen or be used in high-end compositing and green screen effects, stuff where this compression starts to really matter.

Basically, I'm trying to find the perfect balance between quality and economy. If your comment is "don't compress," please provide a really great reason other than the obvious one, such as, compressing these files even slightly will cause Earthquakes in Uganda.

My limited results, using footage shot with the 50mm 1.4 canon lens, so far-

Original File - 47 mbs - looks great
Approx 22 mbs - looks great. The loss is so miniscule, it's work the 50% space saving.
Approx 12 mbs- looks really, really good. Most people would never notice. In fact, I only noticed after frame by frame comparisons. Playing the video, you'd never tell. Tempting at 1/4 of the space!
9 mbs - still looks great but I start to see the softness pretty easily. But it's such a nice softness!
6 mbs - looks amazing for the size, softer, but still impressive, that algorithm!
4 mbs - impressive for the size but it's getting a little too soft even for me.

I haven't tried something around 35 mbs, but that 22 mbs is so great, I'm curious what others think. Of course, other footage, lenses, lighting scenarios could provide much different results, this shot was using natural light indoors during daytime, the lens was pretty wide open but with a low ISO, shooting closeups going in and out of focus with the shallow depth of field.

Thoughts? Yea or Nea? It's amazing, to me, that even at 10% the already-compressed footage looks as good as it does. Okay, shoot!

James Donnelly · April 24th, 2010, 06:50 PM

I am glad you posted on this, I am mulling over exactly the same issue. I am shocked how quickly I fill up those 8GB cards now! I have a Canon HF100, which has a max bitrate of 17mb/s, and looks great to me.

The majority of my footage is personal/family stuff, so the image quality / file size trade off is less critical. For almost all of this type of stuff, 12-17 mb/s is a level I think I'm happy to archive with.

Incidentally, I am encoding with Mainconcept/mp4. I have tried to get Vegas Pro 9 to use the VFW x264 codec, but I can't get it to work. What are you using?

David St. Juskow · April 24th, 2010, 07:25 PM

I'm using Compressor, which comes with Final Cut Pro. I used to experiment a lot with encoding back, what, 10 years ago now, using Media Cleaner Pro, trying to do the same thing, but I hadn't had much need for that in the interim, so technology zipped by me. Now I'm back, and I've tried a couple of compression programs like Sorensen Squeeze, but Compressor is pretty good and, of course, is part of FCP so I haven't found need to try something else as of yet. I use it to convert the files to Apple ProRes for editing as well, it can add visual timecode to clips if people need to watch as dailies for notes, etc. It's pretty versatile.

Anyway, good to hear that bitrate seems to work for you, too. Curious about others...

Pedanes Bol · April 24th, 2010, 08:13 PM

Since the original files are already compressed and there are no RAW files, I chose to keep the originals. I copy them onto DVD-Rs and store them as data files. One DVD-R accommodates ~4.3GB or roughly 13 minutes of 1080p video. That is the maximum size of an individual clip you can generate in these cameras anyway. If we consider one DVD-R costs $0.20 and accommodates 4GB, then 1TB of data storage costs ~$50.

Bryan McCullough · April 24th, 2010, 09:48 PM

And with 1TB drives hardly more than that I just save them on a hard drive and skip the hassle of generating all those Dvds.

James Donnelly · April 25th, 2010, 03:54 AM

A hard drive for archiving? Aren't you worried about if an when it will fail?

I have some mirrored external 500gb drives for storing work in progress, but I would need about 20 of these to store everything!

Waiting for the price of Blu-ray to come down. Quality media costs about £2 a piece bought in 10's, and the cheapest writer is about £150. So, £2 for 25Gb = 6 DVD-R's. A bit more money than DVD for a lot less hassle.

Hmm, seems I'm talking myself into another purchase!

George Angeludis · April 25th, 2010, 04:16 AM

Quote:

Originally Posted by Pedanes Bol

Since the original files are already compressed and there are no RAW files, I chose to keep the originals. I copy them onto DVD-Rs and store them as data files. One DVD-R accommodates ~4.3GB or roughly 13 minutes of 1080p video. That is the maximum size of an individual clip you can generate in these cameras anyway. If we consider one DVD-R costs $0.20 and accommodates 4GB, then 1TB of data storage costs ~$50.

My experience all of those years as for archiving is to follow this one. DVD-R's and HDD at the same time. I keep all of my important projects like this. A copy at a HDD and then to a DVD-R or a Dual Layer. BD's in a while will make this even better.

Bryan McCullough · April 25th, 2010, 10:20 AM

Quote:

Originally Posted by James Donnelly

A hard drive for archiving? Aren't you worried about if an when it will fail?

We keep backups of all our hard drives. I've had more burned disks go bad after a few years than I've had hard drives fails. No backup system is perfect, but I've found archiving to hard drives and having redundancy backups is the way that works best for us.

Now I can see eventually also having blu-ray backups of those drives since 20 disks per TB hard drive would be much easier to deal with than 200 disks.

David St. Juskow · April 25th, 2010, 12:17 PM

Forget the cost of media- I don't know where you keep all that storage. Maybe being a New Yorker has made me space-conscious, but that a LOT of whatevers {hard drives, dvd spindles, etc.) that will pile up fast using this camera, even if they're still functional in 20 years. I know there's no perfect system yet and we all pick our battles, but I guess that's why I'm trying to pack in as many clips onto the same amount of space, to have less crap in my home!

Bryan McCullough · April 25th, 2010, 01:12 PM

Yeah, I'm in Texas. Space ain't a concern. :)

Manus Sweeney · April 25th, 2010, 03:36 PM

Quote:

Originally Posted by George Angeludis

My experience all of those years as for archiving is to follow this one. DVD-R's and HDD at the same time. I keep all of my important projects like this. A copy at a HDD and then to a DVD-R or a Dual Layer. BD's in a while will make this even better.

my projects generally run over the 100gig mark, often a lot more.. dont you guys find the same??

i personally wouldnt be a fan of sitting at home all day burning 30 discs! So far I'm archiving everything on cheap(er) and slow usb 2.0 drives with 2 copies of all my active or semi active work..

Jonathan W. Hickman · May 1st, 2010, 08:16 PM

After a month of shooting interviews and my little movie review show, I think I'm up to about 200 gigs. Of course that's in Pro Res after log and transfer or transcoding with Streamclip. And that's everything, finished shows have been running about 5 gigs or so depending on show segment length (max around 9 minutes before a short commercial break).

I've got a cheap usb 500 gig drive that I'm going to fill up and then I'm planning on buying a new one every few months. And a bud of mine told me to keep the box that the drive comes in, because you put the drive in the box and store it when done. Even use the plastic bag that's in the box when storing it.

I've got stacks of dual layer dvds that I've archived shows to over the years. And that takes so much time. Right now, hard drive is the only real solution. I spoke with a guy at CNN who said that they had been using some kind of tape system until recently. I suspect that Blu-Ray will be great when the price comes down and speed increases for burning. There's just something comforting about having a hard copy on disc.

Rusty Rogers · May 2nd, 2010, 01:46 PM

Quote:

Originally Posted by Pedanes Bol

Since the original files are already compressed and there are no RAW files, I chose to keep the originals. I copy them onto DVD-Rs and store them as data files. One DVD-R accommodates ~4.3GB or roughly 13 minutes of 1080p video. That is the maximum size of an individual clip you can generate in these cameras anyway. If we consider one DVD-R costs $0.20 and accommodates 4GB, then 1TB of data storage costs ~$50.

I've used this method for single project archive using WinRAR to make easily burned DVD5's.

My primary backup is my DroboPro. Not super quick, but dual redundancy is nice.
Being able to edit directly off the DroboPro is possible in a pinch, but not for day-to-day work.

April 23rd, 2010, 06:24 PM	#1
David St. Juskow Regular Crew Join Date: Jan 2010 Location: new york city Posts: 113	Compressing h264 videos for archiving So I've been fooling around with crunching these quicktimes down for archiving, to see how small I can get them without any noticeable compression. "Noticeable" is subjective, of course, so I'm putting out what I've figured so far in case people want to chime in. Now, I realize that the professional answer is "how can you compress your raw footage? That's insane!" And, yes, for certain uses, I would never do that- but the reality is that this stuff takes up SPACE and it fills drives up quickly. So until I can get a 400 TB drive for $10, I'm trying to find a way to crunch the less-than-vital footage I've shot and will be shooting. The fact is, a good chunk of HD video will not be blown up to a 50 foot screen or be used in high-end compositing and green screen effects, stuff where this compression starts to really matter. Basically, I'm trying to find the perfect balance between quality and economy. If your comment is "don't compress," please provide a really great reason other than the obvious one, such as, compressing these files even slightly will cause Earthquakes in Uganda. My limited results, using footage shot with the 50mm 1.4 canon lens, so far- Original File - 47 mbs - looks great Approx 22 mbs - looks great. The loss is so miniscule, it's work the 50% space saving. Approx 12 mbs- looks really, really good. Most people would never notice. In fact, I only noticed after frame by frame comparisons. Playing the video, you'd never tell. Tempting at 1/4 of the space! 9 mbs - still looks great but I start to see the softness pretty easily. But it's such a nice softness! 6 mbs - looks amazing for the size, softer, but still impressive, that algorithm! 4 mbs - impressive for the size but it's getting a little too soft even for me. I haven't tried something around 35 mbs, but that 22 mbs is so great, I'm curious what others think. Of course, other footage, lenses, lighting scenarios could provide much different results, this shot was using natural light indoors during daytime, the lens was pretty wide open but with a low ISO, shooting closeups going in and out of focus with the shallow depth of field. Thoughts? Yea or Nea? It's amazing, to me, that even at 10% the already-compressed footage looks as good as it does. Okay, shoot!

April 24th, 2010, 08:13 PM	#4
Pedanes Bol Regular Crew Join Date: May 2007 Location: Boston MA Posts: 167	RAW Files? Since the original files are already compressed and there are no RAW files, I chose to keep the originals. I copy them onto DVD-Rs and store them as data files. One DVD-R accommodates ~4.3GB or roughly 13 minutes of 1080p video. That is the maximum size of an individual clip you can generate in these cameras anyway. If we consider one DVD-R costs $0.20 and accommodates 4GB, then 1TB of data storage costs ~$50.

April 24th, 2010, 06:50 PM	#2
James Donnelly Major Player Join Date: Apr 2010 Location: Chelmsford England Posts: 287	I am glad you posted on this, I am mulling over exactly the same issue. I am shocked how quickly I fill up those 8GB cards now! I have a Canon HF100, which has a max bitrate of 17mb/s, and looks great to me. The majority of my footage is personal/family stuff, so the image quality / file size trade off is less critical. For almost all of this type of stuff, 12-17 mb/s is a level I think I'm happy to archive with. Incidentally, I am encoding with Mainconcept/mp4. I have tried to get Vegas Pro 9 to use the VFW x264 codec, but I can't get it to work. What are you using?

April 24th, 2010, 07:25 PM	#3
David St. Juskow Regular Crew Join Date: Jan 2010 Location: new york city Posts: 113	I'm using Compressor, which comes with Final Cut Pro. I used to experiment a lot with encoding back, what, 10 years ago now, using Media Cleaner Pro, trying to do the same thing, but I hadn't had much need for that in the interim, so technology zipped by me. Now I'm back, and I've tried a couple of compression programs like Sorensen Squeeze, but Compressor is pretty good and, of course, is part of FCP so I haven't found need to try something else as of yet. I use it to convert the files to Apple ProRes for editing as well, it can add visual timecode to clips if people need to watch as dailies for notes, etc. It's pretty versatile. Anyway, good to hear that bitrate seems to work for you, too. Curious about others...

April 24th, 2010, 09:48 PM	#5
Bryan McCullough Major Player Join Date: Oct 2003 Location: Richardson, TX Posts: 842	And with 1TB drives hardly more than that I just save them on a hard drive and skip the hassle of generating all those Dvds.

April 25th, 2010, 03:54 AM	#6
James Donnelly Major Player Join Date: Apr 2010 Location: Chelmsford England Posts: 287	A hard drive for archiving? Aren't you worried about if an when it will fail? I have some mirrored external 500gb drives for storing work in progress, but I would need about 20 of these to store everything! Waiting for the price of Blu-ray to come down. Quality media costs about £2 a piece bought in 10's, and the cheapest writer is about £150. So, £2 for 25Gb = 6 DVD-R's. A bit more money than DVD for a lot less hassle. Hmm, seems I'm talking myself into another purchase!

April 25th, 2010, 12:17 PM	#9
David St. Juskow Regular Crew Join Date: Jan 2010 Location: new york city Posts: 113	Forget the cost of media- I don't know where you keep all that storage. Maybe being a New Yorker has made me space-conscious, but that a LOT of whatevers {hard drives, dvd spindles, etc.) that will pile up fast using this camera, even if they're still functional in 20 years. I know there's no perfect system yet and we all pick our battles, but I guess that's why I'm trying to pack in as many clips onto the same amount of space, to have less crap in my home!

April 25th, 2010, 01:12 PM	#10
Bryan McCullough Major Player Join Date: Oct 2003 Location: Richardson, TX Posts: 842	Yeah, I'm in Texas. Space ain't a concern. :)

May 1st, 2010, 08:16 PM	#12
Jonathan W. Hickman Regular Crew Join Date: Mar 2009 Location: United States Posts: 81	After a month of shooting interviews and my little movie review show, I think I'm up to about 200 gigs. Of course that's in Pro Res after log and transfer or transcoding with Streamclip. And that's everything, finished shows have been running about 5 gigs or so depending on show segment length (max around 9 minutes before a short commercial break). I've got a cheap usb 500 gig drive that I'm going to fill up and then I'm planning on buying a new one every few months. And a bud of mine told me to keep the box that the drive comes in, because you put the drive in the box and store it when done. Even use the plastic bag that's in the box when storing it. I've got stacks of dual layer dvds that I've archived shows to over the years. And that takes so much time. Right now, hard drive is the only real solution. I spoke with a guy at CNN who said that they had been using some kind of tape system until recently. I suspect that Blu-Ray will be great when the price comes down and speed increases for burning. There's just something comforting about having a hard copy on disc.