HDV: How much can I compress?

[Brian Boyko]

Here's a question: I've got 28 hours of HD footage shot in 24p. After it goes through reverse telecine, each hour of footage takes up about, oh, 30GB or so. I have 28 tapes, and while I have a 1TB drive, I'd rather set it up as 2 500GB RAID drives for backup.

Would it be possible to compress all the video to H.264/best and edit using that? What kind of quality loss am I looking at? Can I save space WITHOUT losing quality by inverse telecining to Apple Intermediate Codec instead of Photo-JPEG?

What would you recommend?

[Andy Wilkinson]

28 hours on a 1GB drive! Now that's what I call compression!!!! :=)

[Rick L. Allen]

Three things;
1. 2 - 500GB drives set up as Raid 0 still equals 1000GB (1 TB) of storage.
2. Do a web search. There are several free widgets and programs that help you compute storage space vs. codec.
3. ProRes422 uses up approximately 115 GB/hr. Neither Photo -JPEG or AIC will look as good as ProRes.

You can either apply heavier compression (and lose picture quality) or buy more storage.

[Ray Bell]

Each time you compress and then un-compress and compress again you loose quality with video footage... its not like compressing a text file...

compression is not a good work flow for archiving video footage...

[Brian Boyko]

Originally Posted by Andy Wilkinson

28 hours on a 1GB drive! Now that's what I call compression!!!! :=)

Should be 1TB.

[Seun Osewa]

Avoid recompression. How come you need 28 hours of footage?

[Allen Plowman]

Originally Posted by Rick L. Allen

Three things;
1. 2 - 500GB drives set up as Raid 0 still equals 1000GB (1 TB) of storage.

he said raid for backup. a raid for backup is a raid 1, and would write identical info on both drives

[Brian Boyko]

Originally Posted by Seun Osewa

Avoid recompression. How come you need 28 hours of footage?

Making a documentary, with a shooting-to-film ratio of about 20 to 1.

[Marcus Marchesseault]

I'm confused. Your title mentions HDV but you have 30GB/hour HD footage from reverse telecine. That means it can not be HDV. What type of tapes do you have? What is the encoding format? What will be the final output format?

[Brian Boyko]

Originally Posted by Marcus Marchesseault

I'm confused. Your title mentions HDV but you have 30GB/hour HD footage from reverse telecine. That means it can not be HDV. What type of tapes do you have? What is the encoding format? What will be the final output format?

I'm recording to DV tape in HDV format 1080/24p via the Canon HV20.

The files that come off the DV tape are much smaller than the files that result from the reverse telecine in photo-Jpeg.

The final formats will be:

Theatrical Distribution (maybe), H.264 1080/24p Blu-Ray, DVD NTSC, DVD PAL, and possibly H.264 1080/24p Online through iTunes.

My big concern is that I want to get a copy of this footage over to my movie-making partner in New Zealand. What I'm probably going to have to do is buy another 1TB drive and just ship it straight to her with the files on board.

[Brian Boyko]

There's another factor. Alot of this footage is over multiple tapes, so what I'm doing is I'm putting them together into a full interview - about 30-60 minutes long, and saving as uncompressed Quicktime.

Maybe I should save those files as MPEG-2? I mean, HDV is already in MPEG2, so...

[Marcus Marchesseault]

I still don't understand something. You are doing a reverse telecine "filmout" BEFORE you edit? Why are you converting to photo-jpeg? It doesn't make sense to me to convert to a larger format then re-compress for editing. What exactly are you trying to accomplish?

It seems to me that you should have edited in HDV or an intermediate format like the Cineform codec. After all the editing is done seems like a more sensible time to convert to a format for reverse telecine. If you edit in HDV, you can simply mail a copy of the tapes to your partner. It is also possible to edit in a low-res intermediate then substitute the HDV files later with an edit decision list but you need to make sure your timecode is all good or the edits won't sync. I have not used that process so somebody else will need to explain that process.

Again, the big issue I see is that you are doing a conversion to a reverse telecine format before you have even edited the project. Have you consulted the house that will be doing the conversion to film so you know exactly how they want it done?

[Brian Boyko]

Originally Posted by Marcus Marchesseault

I still don't understand something. You are doing a reverse telecine "filmout" BEFORE you edit? Why are you converting to photo-jpeg? It doesn't make sense to me to convert to a larger format then re-compress for editing. What exactly are you trying to accomplish?

I use the Canon HV20. It's a consumer-level camera that records HDV in 24p. However, it records it in a pseudo 60i format on the HDV tape - AND it doesn't put in the required flags to mark the interlaced frames.

To get around this, I have to download the material as that "pseudo 60i" format, and use a program called JES Deinterlacer which will detect the cadence breaks and perform the inverse telecine to make it a normal 1080/24p .mov file. This file uses Photo-Jpeg, although I could use QuickTime to compress it out of the box.

It seems to me that you should have edited in HDV or an intermediate format like the Cineform codec. After all the editing is done seems like a more sensible time to convert to a format for reverse telecine.

If I had edited the movie first, then used reverse telecine, it would have messed up any transitions, possibly messed up color corrections, and may, indeed, not have worked at all.

In short, my raw footage is therefore in Photo-JPEG.

If you edit in HDV, you can simply mail a copy of the tapes to your partner.

Unfortunately, if I record the Photo-JPEG files back to HDV tape, it will turn back into that psuedo 60i format. Otherwise that would have been my first choice.

Again, the big issue I see is that you are doing a conversion to a reverse telecine format before you have even edited the project. Have you consulted the house that will be doing the conversion to film so you know exactly how they want it done?

I'm not converting it to film.

---------------------------------------------------------------

This may sound like a bad idea to many, but I've decided that I'm going to convert all the files to H.264 1080/24p. Yes, I'm adding compression early in the process, but I've thought about it and it just simply makes sense.

The H.264 compression is just as good as if I had recorded in AVCHD rather than HDV - better actually because I'm using a higher bitrate than AVCHD. I'm recording on consumer level equipment to begin with. Yes, the HV20 is an awesome camera but it's not the same level as the Panasonic HVX, and the quality loss is minimal and the speed at which I'll be able to work is greatly increased by having files that I can send over to New Zealand by burning them onto DVDs and mailing them. It also means more room on my scratch disc, that I can co-locate them (It's easier and less expensive to find a hosting provider for 200GB of material rather than 1TB) to prevent data loss.

Additionally, I really, absolutely positively, cannot tell the difference in quality even when going over the pixels with a fine-tooth comb. Whatever compression artifacts H.264 introduces, it's not as bad as the MPEG-2 compression already on the tape.

[Marcus Marchesseault]

I'm glad you cleared that up. The 24p workflow is a bit mysterious.

I assume you are doing this?:

"This file uses Photo-Jpeg, although I could use QuickTime to compress it out of the box."

Are you able to select a different compression in JES Deinterlacer than the huge format you had before or are you just running a re-compression on the files in Quicktime? It is probably best to have the fewest operations performed on the data but if you can't tell a difference it should be okay as long as you keep other compression generations to a minimum.

[Brian Boyko]

Originally Posted by Marcus Marchesseault

I'm glad you cleared that up. The 24p workflow is a bit mysterious.

I assume you are doing this?:

"This file uses Photo-Jpeg, although I could use QuickTime to compress it out of the box."

Are you able to select a different compression in JES Deinterlacer than the huge format you had before or are you just running a re-compression on the files in Quicktime? It is probably best to have the fewest operations performed on the data but if you can't tell a difference it should be okay as long as you keep other compression generations to a minimum.

Indeed, I'm choosing Photo-JPEG out of JES Deinterlacer because that's an uncompressed format. I then take the uncompressed footage, bring it into FCP, string the disperate footage together into one single file, and color correct. Then I export that as an uncompressed .MOV file, and use Compressor to convert that .MOV file to h.264.

The real quandary is if I can delete those uncompressed .mov files (around 30 GB each) to save space, because the H.264 data (around 5 GB each) contains the same information.