422 vs XDCam Export

[Dan Brazda]

So I've done a lot of research on this (much debate out there) but have finally gotten down to doing a practical experiment. Here are the parameters:

-shoot XDCam HD 35mbps VBR
-edit same but set render to ProRez 422 (some say stay XDCam all the way)
-export as self-contained quicktime:
*using current settings ( ie- XDCam HD 35mbps VBR)
OR
*as ProRez 422 HQ 1920x1080

I'm in the midst of this test but would love to hear from those of you who have already done it.

I am confused about "trancoding" (might not be the proper terminology) in general. In my mind, whenever you start with a certain format, codec...whatever... you want to stick to that format, codec...all the way from start to finish to avoid "transcoding" along the way. I have heard many say that if you are using color correction and other things that you want to go 4:2:2 and stay there even if you start with 4:2:0. Huh! How do you make it better than what it started with? It reminds me of the people who used to think that you could take an inferior format like Super-8 and blow it up to 16 or 35 mm and it would look better.

Help me out with what I am missing here!

[Perrone Ford]

The idea of transcoding immediately to a better codec is not that it increases what you started with, but it keeps the image from degrading as you modify it in the timeline. You modify colors and make shades that were not there in the original footage. You add effects, titles, fades, flashes, etc. If you left the footage in the original (lossy) format, you'd keep losing quality as you modified the footage. By moving to a codec that can handle all the changes you are making without degrading, you can end up in a MUCH better position at the end of editing.

Then you render out to your finishing format which is nearly always lossy (Mpeg2/Mpeg4/etc.). You can also render to a lossless format for mastering.

[Dan Brazda]

Originally Posted by Perrone Ford

The idea of transcoding immediately to a better codec is not that it increases what you started with, but it keeps the image from degrading as you modify it in the timeline. You modify colors and make shades that were not there in the original footage. You add effects, titles, fades, flashes, etc. If you left the footage in the original (lossy) format, you'd keep losing quality as you modified the footage. By moving to a codec that can handle all the changes you are making without degrading, you can end up in a MUCH better position at the end of editing.

Then you render out to your finishing format which is nearly always lossy (Mpeg2/Mpeg4/etc.). You can also render to a lossless format for mastering.

So you're saying that you setup a 422 timeline in the first place? What I've been doing is I setup my timeline for the native format (XDCam HD 35mbps vbr) but set the render control to ProRez 422. Assuming that's what you are talking about, my question is after having done all titles and color correction and I want to export my master, am I gaining anything by exporting it as ProRez vs XDCam?

[Perrone Ford]

Well, I don't know what editor you're using. On mine it makes zero difference what I set up the timeline as. It does everything at 4:4:4 regardless of how I set the timeline.

On your's it might matter. And if it does, then set the timeline to you master format. What's the point of setting the timeline to limited color space, having it truncate all the data as you edit, and then exporting to a higher quality codec when there's no data there to take advantage of it.

It's like mixing the ingredient for a cake in quart pans and then pouring the final product in a 10 gallon container.

[Dan Brazda]

I guess this is a Final Cut Pro specific question. Any FCP/ XDCam HD users out there that would care to chime in?

[Eelco Romeijn]

I'm using Premiere CS3 with MatroxRTX2. Latest everything.
Most HD edits we start with converting the XDcamHD (35Mb/s) tot Matrox Iframe 4:2:2 codec in 1440x1080 at 100 Mb/s (with Vegas). However this takes some more time before you can actually start editing the rendered results are visibly better than in native XD. There is no difference if you just edit your shots but as soon as there's a graphics layer, colorcorrection or animations involved the image quality increased. Even chromakeys work a lot better from the converted 422 file. It seems to me that this would not be possible in theory but in practice it does.

I guess Prorez does more or less the same as Matrox iFrame.
In my opinion Mpeg longgop is an aquisition codec to get HQ images at relatively low bitrates on de disk. It's not an ideal editing codec.

I'm interested in the outcome of your tests.

[Rick L. Allen]

I'm currently working on a an HD project that was shot on XDCAM HD, HDV and DVCAM. My timeline is set up as ProRes422(HQ) and final playout will be to HDCAM via Kona LHe. Since all my footage was decompressed on input (via HD/SDI) I'm working with the purest video I can without editing uncompressed video (and the storage issues that go with uncompressed). Using this workflow "opens up" the mpeg container and makes it easier to use without having to decompress/recompress the XDCAM container for each edit. Cuts the processor load which can be used elsewhere for additional tracks etc.

Using ProRes422(HQ) allows me to make multiple changes, color correction, etc. while still maintaining quality and minimal generation loss. Short answer, work in the highest resolution your system and storage allows. It's a lot easier to go down in quality when it's time for final output. I'm doing QC using a JVC broadcast HD monitor and the ProRes422(HQ) is almost indistinguishable from the original footage and looks fabulous. Even my upconverted DVCAM looks pretty amazing.

[Dan Brazda]

So preliminary tests are done but it just makes me want to do more testing. I shot a concert in XDCam HD 35mbps VBR, set up my Final Cut Pro timeline for same but set render Control to ProRez 422. I did A LOT of color correction and then exported the final in two different forms:

-"using current settings" (ie- XDCam HD 35mbps VBR)

AND

-ProRez 422 HQ 1440x1080

The first gave me a finished Quicktime File of aprox 15GB. The 2nd gave me a finished file of aprox 65GB. After studying literally dozens of side-by-side frames ( as well as simul-rolling the two Quicktimes on side-by-side HD monitors) I honestly see no difference AT ALL.

My first reaction was- why bother with ProRez as an export master. BUT- after thinking about it, I'm wondering if in Final Cut when you set your render codec to ProRez and then export as "using current settings", that is actually the same thing as exporting as ProRez 422 (given the render control settings I described). That would explain why it looks identical I guess.

So, FCP folks- am I right in thinking that the best workflow is to set your timeline to the native format, set your render control to ProRez 422 (assuming you are doing extensive CC and Titling) and then export "using current settings" or "ProRez 422 1440x1080"- both of which will yield the same results?

[Perrone Ford]

Dan, either I am not communicating well, or you've chosen to ignore what I said in my first post.

Of COURSE the files don't look any different when you rendered them out. You threw away all the data when you set the timeline to an inferior spec.

Let me try to explain another way.

Assume for a moment that the EX1 could record a million colors (just hypothetical). And you set your timeline to only handle 256 colors. When you bring the EX1 file into the timeline, your 1 million colors will be truncated to 256 colors. You edit, and do all the things you want to do in that timeline, but it can only understand 256 colors.

Then you export it two ways. One that understands only 256 colors, and the other that can understand 1 million colors. The problem is that your placing 256 inside a space that can hold a million. So naturally it's going to look exactly like the other. All the extra data was thrown away at the outset.

Contrast this with setting up a timeline that can understand 1 million colors, and dropping the file onto this timeline. You do your editing, and so forth and all the colors are preserved. Now you render out to the two formats. One that holds 1 million colors and preserves the colors, and one that only holds 256 colors, in which case it's going to look vastly different.

This is a grossly simplified example to be sure, but it explains your results.

-P

Originally Posted by Dan Brazda

So preliminary tests are done but it just makes me want to do more testing. I shot a concert in XDCam HD 35mbps VBR, set up my Final Cut Pro timeline for same but set render Control to ProRez 422. I did A LOT of color correction and then exported the final in two different forms:

-"using current settings" (ie- XDCam HD 35mbps VBR)

AND

-ProRez 422 HQ 1440x1080

The first gave me a finished Quicktime File of aprox 15GB. The 2nd gave me a finished file of aprox 65GB. After studying literally dozens of side-by-side frames ( as well as simul-rolling the two Quicktimes on side-by-side HD monitors) I honestly see no difference AT ALL.

My first reaction was- why bother with ProRez as an export master. BUT- after thinking about it, I'm wondering if in Final Cut when you set your render codec to ProRez and then export as "using current settings", that is actually the same thing as exporting as ProRez 422 (given the render control settings I described). That would explain why it looks identical I guess.

So, FCP folks- am I right in thinking that the best workflow is to set your timeline to the native format, set your render control to ProRez 422 (assuming you are doing extensive CC and Titling) and then export "using current settings" or "ProRez 422 1440x1080"- both of which will yield the same results?

[Dan Brazda]

Originally Posted by Perrone Ford

Dan, either I am not communicating well, or you've chosen to ignore what I said in my first post.

Of COURSE the files don't look any different when you rendered them out. You threw away all the data when you set the timeline to an inferior spec.

Let me try to explain another way.

Assume for a moment that the EX1 could record a million colors (just hypothetical). And you set your timeline to only handle 256 colors. When you bring the EX1 file into the timeline, your 1 million colors will be truncated to 256 colors. You edit, and do all the things you want to do in that timeline, but it can only understand 256 colors.

Then you export it two ways. One that understands only 256 colors, and the other that can understand 1 million colors. The problem is that your placing 256 inside a space that can hold a million. So naturally it's going to look exactly like the other. All the extra data was thrown away at the outset.

Contrast this with setting up a timeline that can understand 1 million colors, and dropping the file onto this timeline. You do your editing, and so forth and all the colors are preserved. Now you render out to the two formats. One that holds 1 million colors and preserves the colors, and one that only holds 256 colors, in which case it's going to look vastly different.

This is a grossly simplified example to be sure, but it explains your results.

-P

I'm hoping your actual tone is different than it comes across in your response so I'll just write that off as an internet thread anomoly. Regarding whether or not I understood your initial post- I did indeed. I could be wrong but I have a feeling you are not using Final Cut Pro.

Again, I set my timeline for the NATIVE format (an "inferior spec" as you call it) of XDCam HD 35mbps VBR initially only because I can start editing right away with no rendering of any kind. HOWEVER, as I said, I set my RENDER CONTROL to ProRez 422. Therefor, any color correction or anything requiring rendering will be rendered as ProRez 422. How can you say I am "throwing away all the data" when I am simply matching up what was put on disc to the timeline? What am I throwing away? I would only be causing degredation if I remained in a 420 color space when doing color correction and other effects. Your analogy of 256 colors vs 1 million implies that by transcoding XDCam HD 420 footage into ProRez 422 footage from the start I am somehow making the original better than it was in the first place. Sorry but I don't buy it. I do buy the fact that if I want to put my original 420 footage through color correction and other laborious tasks I should do it in a more robust color space- which is what I think I am doing by setting my render control to ProRez.

All that being said, I will happily change my workflow and transcode all of my original XDCam HD footage into ProRez 422 BEFORE editing if someone else can dive in here and say that is what common knowledge is when working in FCP.

[Perrone Ford]

Yea,

Tone over the internet sucks for sure. Sorry if it came across bad. Wasn't meant that way. So let me lay some groundwork.

1. You are absolutely correct. I am not using FCP.
2. Glad you got my meaning in my post.
3. You are also correct that things cannot be "improved" by moving to data to a new codec.

Ok, so let me see if I can work this from a different way.

I do not understand FCP well enough to say whether or not how you set your render parameters holds for what is on the timeline. I'd feel a LOT better if someone who know FCP well enough would answer this.

For the time being, lets assume you are right, and the re-render on the timeline is actually pro-res 4:2:2. Your comment that you did a lot of color correction would indicate to me that you have done significant shifting in the available color space. If indeed you are working in the 4:2:2 space this should all be just fine. If you are not, then certainly this is not preserving all the color information it should.

This is an interesting problem and really does go to the heart of how FCP works. Again, I apologize if my tone caused an issue. I do tend to "jump in with both feet" on these kinds of problems. What I don't "get" is if the timeline is set to one spec, and you start changing colors around, how FCP could smoothly deal with that before rendering. I know that in Vegas, my editor of choice, the timeline is always 4:4:4 so all color information is preserved before rendering.

Let's hope someone with a lot of FCP backround jumps in here and educates us both!

Originally Posted by Dan Brazda

I'm hoping your actual tone is different than it comes across in your response so I'll just write that off as an internet thread anomoly. Regarding whether or not I understood your initial post- I did indeed. I could be wrong but I have a feeling you are not using Final Cut Pro.

Again, I set my timeline for the NATIVE format (an "inferior spec" as you call it) of XDCam HD 35mbps VBR initially only because I can start editing right away with no rendering of any kind. HOWEVER, as I said, I set my RENDER CONTROL to ProRez 422. Therefor, any color correction or anything requiring rendering will be rendered as ProRez 422. How can you say I am "throwing away all the data" when I am simply matching up what was put on disc to the timeline? What am I throwing away? I would only be causing degredation if I remained in a 420 color space when doing color correction and other effects. Your analogy of 256 colors vs 1 million implies that by transcoding XDCam HD 420 footage into ProRez 422 footage from the start I am somehow making the original better than it was in the first place. Sorry but I don't buy it. I do buy the fact that if I want to put my original 420 footage through color correction and other laborious tasks I should do it in a more robust color space- which is what I think I am doing by setting my render control to ProRez.

All that being said, I will happily change my workflow and transcode all of my original XDCam HD footage into ProRez 422 BEFORE editing if someone else can dive in here and say that is what common knowledge is when working in FCP.

[Dan Brazda]

Originally Posted by Perrone Ford

Yea,

Tone over the internet sucks for sure. Sorry if it came across bad. Wasn't meant that way. So let me lay some groundwork.

1. You are absolutely correct. I am not using FCP.
2. Glad you got my meaning in my post.
3. You are also correct that things cannot be "improved" by moving to data to a new codec.

Ok, so let me see if I can work this from a different way.

I do not understand FCP well enough to say whether or not how you set your render parameters holds for what is on the timeline. I'd feel a LOT better if someone who know FCP well enough would answer this.

For the time being, lets assume you are right, and the re-render on the timeline is actually pro-res 4:2:2. Your comment that you did a lot of color correction would indicate to me that you have done significant shifting in the available color space. If indeed you are working in the 4:2:2 space this should all be just fine. If you are not, then certainly this is not preserving all the color information it should.

This is an interesting problem and really does go to the heart of how FCP works. Again, I apologize if my tone caused an issue. I do tend to "jump in with both feet" on these kinds of problems. What I don't "get" is if the timeline is set to one spec, and you start changing colors around, how FCP could smoothly deal with that before rendering. I know that in Vegas, my editor of choice, the timeline is always 4:4:4 so all color information is preserved before rendering.

Let's hope someone with a lot of FCP backround jumps in here and educates us both!

Thanks Perrone.

"What I don't "get" is if the timeline is set to one spec, and you start changing colors around, how FCP could smoothly deal with that before rendering."

You nailed it with that sentence. That's what prompted me to post in the first place. I wish I could find the original thread (may not have been this forum) where there was extensive debate on this subject. I'll eagerly await FCP user responses along with you. Tomorrow I'll compare original footage frames with my final output frames and see if my workflow is causing any degradation. This may be difficult since my output frames are color corrected but I'll look for digital noise rather than overall image quality perception.

By the way, I've heard Sony Vegas is awesome to work with when shooting XDCam HD. I wish they made a version for the Mac.

[Perrone Ford]

You know, I thought of something.

When I was testing Cineform I loaded up color bars, gradients, and some other tests on the timeline. Then rendered them various ways at various resolutions.

This might give you some insight into what you are working with. The loss of data should be immediately apparent on the waveform monitor and vectorscope. Move some stuff around in the timeline, see what changes, then do a few renders and bring them into the timeline. The scopes should tell the tale definitively.

Oh, and by the way... Vegas is the best thing I have ever cut on. But it has it's issues, believe me. Titling is poor at best. There are times when I'd REALLY love to have After Effects. I may install it just to get around some of these issues.

-P

Originally Posted by Dan Brazda

Tomorrow I'll compare original footage frames with my final output frames and see if my workflow is causing any degradation. This may be difficult since my output frames are color corrected but I'll look for digital noise rather than overall image quality perception.

By the way, I've heard Sony Vegas is awesome to work with when shooting XDCam HD. I wish they made a version for the Mac.

[Christopher Drews]

I'm no expert Dan but I do use FCP and the hardware associated with it. The only way you'd do better is if you had a Kona board / AJA ioHD / Blackmagic and played out via HD-SDI (I assume you're acquiring via EX1) from your camera and recorded to Pro-Res / Uncompressed 10 bit directly. This of course is a 60i signal, so if your source were 23.98 you'd have to work on it. Yet, garbage in / garbage out still applies (as was said previously). What this does do though, is use the hardware converters in the board to control your sample rate / color data - all while ditching your 35mbps Long GOP codec. I've been surprised at this comparison before. When you can, use hardware to do the heavy lifting. This is true for decks also. Here is a good example that I know to be true - If I have a Digi-Beta Deck with an upconvert board (to 1080i) and I send that signal to a FCP ingesting at HD Pro-Res, the quality would look decidedly better than simply digitizing the Digi-Beta into FCP Pro-Res SD and then upconverting to 1080i Pro-Res. Does this make sense?

So in my mind an ideal workflow would be - EX1 HD-SDI > HD-CAM SR (Master created) HD-SDI OUT > FCP PRO-RES HQ or 10 Bit Uncompressed HD (Fiber Channel or SCSI required for 10 Bit). At the end of the day, FCP is smart enough to work with any codec but always try to use one which won't create the "red line of doom" when you change your sequence - and the best codec to minimize rendering seems to be Pro-Res.

Hope this helps,
-C

[Dan Brazda]

Originally Posted by Christopher Drews

I'm no expert Dan but I do use FCP and the hardware associated with it. The only way you'd do better is if you had a Kona board / AJA ioHD / Blackmagic and played out via HD-SDI (I assume you're acquiring via EX1) from your camera and recorded to Pro-Res / Uncompressed 10 bit directly. This of course is a 60i signal, so if your source were 23.98 you'd have to work on it. Yet, garbage in / garbage out still applies (as was said previously). What this does do though, is use the hardware converters in the board to control your sample rate / color data - all while ditching your 35mbps Long GOP codec. I've been surprised at this comparison before. When you can, use hardware to do the heavy lifting. This is true for decks also. Here is a good example that I know to be true - If I have a Digi-Beta Deck with an upconvert board (to 1080i) and I send that signal to a FCP ingesting at HD Pro-Res, the quality would look decidedly better than simply digitizing the Digi-Beta into FCP Pro-Res SD and then upconverting to 1080i Pro-Res. Does this make sense?

So in my mind an ideal workflow would be - EX1 HD-SDI > HD-CAM SR (Master created) HD-SDI OUT > FCP PRO-RES HQ or 10 Bit Uncompressed HD (Fiber Channel or SCSI required for 10 Bit). At the end of the day, FCP is smart enough to work with any codec but always try to use one which won't create the "red line of doom" when you change your sequence - and the best codec to minimize rendering seems to be Pro-Res.

Hope this helps,
-C

Well said. My workflow is acquisition to XDCam HD disc at 23.98 fps. So until I can acquire differently (ie- via the HD-SDI port on the PDW-F350 directly to a dockable Hard Disk) I am stuck with XDCam HD 35mbps VBR as my origination codec. The key is keeping that 420 color space from breaking up along the post process so hence the original thread.