And in THIS case, I COMPLETELY agree that saving the BPAV folder makes total sense. Most people won't need the thumbnails in archiving their master files, but broadcast is a totally different animal.

I work with broadcast news and archive the BPAV to RAID (one BPAV folder per story). My reason basically beeing I get these wonderful thumbnails, and all the metadata easy to read. Saves me a lot of time when I quickly need archive footage.

Perrone,

Thanks, that was incredibly clear and helpful.

Where does Apple Pro Res fit into this schema. Is it similar to Cineform?

Can you add a touch more that brings "Long GOP" into the picture as well?

- Lenny


Sure Leonard.

Apple's ProRes fits into phase 2 and phase 3. It's a robust codec that can stand up to editing (it was built for it) and is good enough to master in. There is also ProResHQ which is another step up, but is a bit "heavy" for most machines to use for editing. If you are cutting footage off RED or other "film" based footage it might make sense. ProRes is a bit lossy, but perfectly acceptable for a few generations of rendering.

The idea of long GOP is simple in theory.

Let's say we are shooting 30 progressive frames per second on our camera. It's on a tripod, and it's on a sidewalk aimed at a building. Every half second we record 15 frames of video. An intraframe camera like the HVX or like true film, will record 15 frames that are complete. It's like taking 15 rapid pictures with a digital still camera and that gets recorded to the media. In pictures that don't change much, this is an incredibly wasteful way to store data.

In the same scenario, Long GOP takes a full frame on frame 1. Then on frame 2 it takes another full frame and compares it to the first. It then only records the difference between the two and writes that to the media. It does the same for the next 13 frames. Well at least the Sony does because it uses a 15 frame GOP structure. Other cameras like the JVC use a 7 frame GOP structure which would handle rapid motion better.

Where Long GOP falls down is if there is a large amount of change between the frames in it's structure. At some point it can't keep up with how quickly things are changing. The software is usually pretty smart. It's making predictions of what will change from one frame to the next to speed up things. When that begins to fail, the picture quality begins to show artifacts. You can see exactly the same thing in your DVDs even if you start with intraframe sources because Mpeg2 which is used for DVDs (and the EX1) use the same technology.

Newer Long GOP codecs like the improved EX1 mpeg2 codec do a better job at this stuff than the older (DVD and HDV) mpeg2 based recorders. Better still are the new mpeg4/AVCHD recorders like the new Panasonic HMC150. The problem is that their efficiency at writing the data better comes at the cost of your computer working harder to DECODE the same stuff later. So you either need a faster computer for the new stuff, or you need to transcode to something easier to handle like Cineform or ProRes.

The internet community usually rails against Long GOP because it is harder on computers in editing, and the codec can be overwhelmed with fast action. What is nice is that the EX1/EX3 allows professional users and soon amateur users to bypass the long GOP codec. The SDI port on the side allows you to take the information off the sensors uncompressed. Just like in the $100K+ cameras. And you can record it any way you want. The downside is that you have to pay extra for that recorder. But in cameras without an SDI port, this option simply does not exist. Why this is exciting to pro users of the EX1/EX3 or amateurs with deeper pockets is that the EX1/EX3 give you the image sensors of a far more expensive camera, and with SDI based recording, you can now write that data stream. And you can do it for tens of thousands of dollars less than you could to do it 2 years ago.

Make sense?

Perrone,
Chapeau :-)

Thank's again Perrone, learning a lot here.

I also take my finished story cut in FCP and export it as XDcam MP4, import it to ClipBrowser and put it on my RAID. This way I can easily put it on the SxS-card in the camera, and play it out via the SDI.

If time is short I also use this way (SDI out) to feed a story via satellite instead of compressing and ftp.

As you point out, Perrone, Sony's long GOP codec is an interesting animal. At 35mbps(ala EX1), images are crystal clear, until you get full frame motion, like in a pan. Then the image breaks down, due to the compression artifacting at 35mbps.

While I still don't have my hands on a Convergent Design SDI to CF recorder(have a nanoflash on order)all the reports I've seen show a much reduced motion artifacting. The C-D CF recorders also use the Sony codec, storing data at a selectable 50 or 100mbps. This is a good compromise, since there's not a lot of portable equipment that can capture an HD-SDI signal.

Thanks again Perrone,

Your replies are now saved on my desktop. I sort of know most of this but it helps to see it all at once.
Now last questions -

1 - what's the diff between using Cineform or ProRes.

2 - I'm on a Macbook Pro - should I bypass ProRes HQ and just stick with regular ProRes?

3 - I know a lot of people who edit HD using Panasonic DVCPro100 codec. Often these are guys who are used to shooting P2 or with varicam etc. I've been trying to convince them to move over to an Ex-1 but they are freaked out by long GOP and the fact that its a different codec.

Can they convert Ex-1 XDCAM footage to DVCPro100?

I guess its kind of a pain for them because they need to convert all of it up front otherwise they are constantly rendering.

I've been told that the easiest way to cut EX-1 footage is to use and XDCam timeline but set for renders to be in ProRes. Can you do the same for DCPro100 and would that solve the problem? Just render once when dropping the stuff in and then it is always ProRes?

Sorry if this a bit afield of the rest of the thread here.

Lenny

As you point out, Perrone, Sony's long GOP codec is an interesting animal. At 35mbps(ala EX1), images are crystal clear, until you get full frame motion, like in a pan. Then the image breaks down, due to the compression artifacting at 35mbps.

Must be something wrong with your camera as my pictures don't break down on pans.

Perrone's statement that AVCHD is better than MPEG2 is not one I agree with. Certainly long GOP AVCHD can produce smaller files. AVCHD is a member of the Mpeg4 family and was really designed for very low bandwidth use such as web streaming and in that application Mpeg4 is very good, but I have yet to see any long GOP AVCHD or Mpeg 4 that has artifacts down at the level of XDCAM HD.

Perrone's description of the GOP sequence is simplified .Mpeg 2 not only forward predicts the motion in the picture but also backwards predicts, so your only ever 7 frames from your full reference frame, the Bi-directional predicted frames are call B frames, forward Predicted frames are P frames and the reference full frames are I frames. The frame sequence looks like this:
IBBPBBPBBPBBPBBIBBPBBPBBPBBPBBP

Must be something wrong with your camera as my pictures don't break down on pans.

It could be the camera... or the operator. In my early days of film camera operating, we were told not to do pans of more then 30 degrees per second with the camera with a 90 degree angle to the picture pane. A 180 degree pan therefore took 6 seconds. To not to pan too fast we counted loudly... and it worked. Faster pans always resulted in visibly stuttering pictures on the screen.
That may be old craftmanship many an operator now smiles mildly upon, nevertheless video follows the same psychovisual laws as old perforated film (especially with progressive frame rates).

If I really need a fast pan, I either do it verrry slowly and speed that take up during post pro or I open the shutter as far as it gets with that specific camera, sometimes even adding a light horizontal motion blur in post pro before speeding the pan up to the desired value.

Thanks again Perrone,

Your replies are now saved on my desktop. I sort of know most of this but it helps to see it all at once.
Now last questions -

1 - what's the diff between using Cineform or ProRes.

2 - I'm on a Macbook Pro - should I bypass ProRes HQ and just stick with regular ProRes?

3 - I know a lot of people who edit HD using Panasonic DVCPro100 codec. Often these are guys who are used to shooting P2 or with varicam etc. I've been trying to convince them to move over to an Ex-1 but they are freaked out by long GOP and the fact that its a different codec.

Can they convert Ex-1 XDCAM footage to DVCPro100?

I guess its kind of a pain for them because they need to convert all of it up front otherwise they are constantly rendering.

I've been told that the easiest way to cut EX-1 footage is to use and XDCam timeline but set for renders to be in ProRes. Can you do the same for DCPro100 and would that solve the problem? Just render once when dropping the stuff in and then it is always ProRes?

Sorry if this a bit afield of the rest of the thread here.

Lenny

Leonard, I can't fake it. I know very little about editing on FCP. However, something you have pointed out here is one of the major reasons I never will. On to your questions:

1. Both Cineform and ProRes are terrific codecs. ProRes is apple specific and you cannot share data for editing with anyone on a PC using any program. Period. I am not certain if Cineform is PC only, but I've only seen it used on PCs.

2. Unless you are originating with very high quality footage (2k or higher), I'd just stick with ProRes.

3. DVCProHD is 1280x1080. Taking footage from a full raster camera like the EX1 to DVCProHD throws away a bunch of information. Your 1920x1080 becomes 1280x1080. The Panasonic guys edit in this codec because their cameras SHOOT in this codec. They don't record anything better in the first place, so no need to shift to a 1920x1080 codec in post.

The DVCPro HD codec is superior to the EX1 codec in a couple of ways. First it is intraframe and not long GOP. This has clear advantages during phase 1. Second, it captures marginally more color information per frame of footage. The big loss of the DVCPro HD codec is that it's horizontal resolution isn't nearly as good as XDCAM EX. And the XDCAM EX is terrible for phase 2 or 3. It's a phase 1 codec only (or primarily).

My beef with FCP is that you need to have all of your sequences match. I am SO used to Vegas where I can drop ANYTHING on the timeline, cut it, and render out. No need to convert anything at all. I have dropped EX1 footage, miniDV, and RED stuff (2k .mov) onto a timeline, cut it like it was native, and rendered out to Windows Media for streaming. It's all seamless. FCP really needs to get on the ball with this. For a pro level NLE, this is a really stifling limitation.

Perrone's statement that AVCHD is better than MPEG2 is not one I agree with.


Fair enough. We can agree to disagree.


Certainly long GOP AVCHD can produce smaller files. AVCHD is a member of the Mpeg4 family and was really designed for very low bandwidth use such as web streaming and in that application Mpeg4 is very good, but I have yet to see any long GOP AVCHD or Mpeg 4 that has artifacts down at the level of XDCAM HD.


Watching mp4 encodes of 2k or 3k footage, with 40+ Mbps rates, I am inclined to disagree. I cannot speak to the footage quality of the HMC-150, but I have seen high bitrate AVCHD on BluRay, and it looks DARN good. Even with very high motion content.


Perrone's description of the GOP sequence is simplified .Mpeg 2 not only forward predicts the motion in the picture but also backwards predicts, so your only ever 7 frames from your full reference frame, the Bi-directional predicted frames are call B frames, forward Predicted frames are P frames and the reference full frames are I frames. The frame sequence looks like this:
IBBPBBPBBPBBPBBIBBPBBPBBPBBPBBP

Yes, I simplified. Thanks for filling in the blanks for a more full description.

"Must be something wrong with your camera as my pictures don't break down on pans. "

hmmm..this is a well know characteristic. You guys sure you look very closely at your footage? I suggest you do a search on this forum, for EX1 compression artifacting.

AFAIK, DVCPRO HD(100mbps) is not as good as DNx100. BUt, again, only my opinion, you will allow me to have one, yes? We are in the realms of esotericism.

Well, as usual, this thread is degenerating into insults and BS with you guys. Time for me to move on.

1. Both Cineform and ProRes are terrific codecs. ProRes is apple specific and you cannot share data for editing with anyone on a PC using any program. Period. I am not certain if Cineform is PC only, but I've only seen it used on PCs.

Only one point I can comment: ProRes is available both for Mac and Win. So it is not plattform depended as you think, posted August 28, 2008. Have a try here: Apple ProRes QuickTime Decoder 1.0 for Windows (http://support.apple.com/downloads/Apple_ProRes_QuickTime_Decoder_1_0_for_Windows)

I also take my finished story cut in FCP and export it as XDcam MP4, import it to ClipBrowser and put it on my RAID. This way I can easily put it on the SxS-card in the camera, and play it out via the SDI.

This is very interesting Fredrik--could you elaborate on the steps here or point me to another place as I've missed how to do this? I just did Export > Sony XDCAM, had to recompress to XDCAM HD (my sequence was uncompressed), and this generated an MXF file which Clip Browser would not import. Thanks in advance!

Only one point I can comment: ProRes is available both for Mac and Win. So it is not plattform depended as you think, posted August 28, 2008. Have a try here: Apple ProRes QuickTime Decoder 1.0 for Windows (http://support.apple.com/downloads/Apple_ProRes_QuickTime_Decoder_1_0_for_Windows)

That only works in Quicktime. It does not allow me to take a ProRes file into an editor. So yes, unfortunately, it is as platform dependent as I think.

That only works in Quicktime. It does not allow me to take a ProRes file into an editor. So yes, unfortunately, it is as platform dependent as I think. Perrone, as I understand it, Mac OS is a plattform and Win is a plattform. Quicktime to my understanding however is a format/wrapper. If this is not correct, we have a different view of technical terms and I do apologize.

Perrone, as I understand it, Mac OS is a plattform and Win is a plattform. Quicktime to my understanding however is a format/wrapper. If this is not correct, we have a different view of technical terms and I do apologize.

Peter no need to apologize. If you read my post about platforms, it specifically mentions not being able to share files for editing. Playing the videos is another matter altogether. What Apple has done is allow a PC user to play a video created in FCP with the native codec. This has extremely limited use and I don't know ANYONE who'd offer up ProRes quicktime files to a PC user except for checking color or other things.

Right now, when I work with Mac users I typically move files back and forth as PNG quicktimes, or uncompressed. What I had hoped was that Apple would make an implementation of ProRes that would allow me to encode using it, so I could edit and pass work seamlessly off to FCP users in their native codec. Or that I could recieve files from FCP and drop them into my editor. But this is not the case. It's a "view-only" deal that Apple has given us.

Perrone, I can clearly see your point now. All in all this turns out to be a very interesting topic I'm also very much interested in also from my professional life. Perhaps we have to move on with our discussion to somewhere else, like the NLE forum. To be honest, I'm too tired right now but I'll touch base in about 10 hours time if that is ok with you. P.

Hi Perrone. You've touched a very interesting topic with your last post here. I'm currently establishing an external video server for our company to archive all of our media assets (video and telecined film) there in the future. Our external video service providers mostly work on win plattforms (Liquid and Avid) while internally we work on Macs (Final Cut Studio). We are searching for a format/codec that runs natively on Mac and on Win machines so that every one both internally and externally can upload a video's last edit for later download somewhere else and to work on that video without having to render first. We work in SD PAL and intend to switch to HD in about two years time.
Our short list of formats: MXF D10, DV50 and AVID Medien Compressed.
Uncompressed is no-no.

Any thoughts?

Leonard, I can't fake it. I know very little about editing on FCP. However, something you have pointed out here is one of the major reasons I never will. On to your questions:

My beef with FCP is that you need to have all of your sequences match. I am SO used to Vegas where I can drop ANYTHING on the timeline, cut it, and render out. No need to convert anything at all. I have dropped EX1 footage, miniDV, and RED stuff (2k .mov) onto a timeline, cut it like it was native, and rendered out to Windows Media for streaming. It's all seamless. FCP really needs to get on the ball with this. For a pro level NLE, this is a really stifling limitation.

I regularly mix HDV, XDCAM EX 1080, XDCAM EX 720, MiniDV on the same timeline in FCP. I then edit normally and render it out to whatever format I need. It´s been like this for a long time

I regularly mix HDV, XDCAM EX 1080, XDCAM EX 720, MiniDV on the same timeline in FCP. I then edit normally and render it out to whatever format I need. It´s been like this for a long time

I read an article about FCP last night about this. Some things still seem weird, but it certainly supports what you've said here.

I read an article about FCP last night about this. Some things still seem weird, but it certainly supports what you've said here.

Sometimes weired things happen though. I'll give you that. It's usually because a clip has been rendered into a different format ones before. Seems to happen to dv clips

You do get problems with many NLE's if you mix DV footage with non DV footage. The problem is that for some reason the field order in DV material is the opposite to everything else. DV footage displays the lower field in each frame before the upper field, while every other codec that I have ever used and TV transmission all display the upper field first. I have no idea why this is but certainly FCP does not like mixing DV and non DV in the same timeline. It works, but the field order gets muddled.

Alister, the mixup because with the inverse line order of DV is due to technical requirements
I won't explain here in detail. No fingerpointing - ok?

To circumvent the problem it is advisable to de-interlace the DV source material with DVMaker,
Fieldskit, JES de-interlacer, MagicBulleFrames and the likes. Correctly applied you get two identical
fields that do not pose problems with standard interlace video in upper field first format.

It is also possible to change the field order (upper field first instead of lower field first) after
changing the the line numbering scheme: line 2 becomes line 3 and line three becomes line 4 and so forth...

Let me know if you want me to search for the apps.

I know a situation when I need the BPAV file set, when I'm using the freeze-mix funtion of the EX3.
See: http://www.dvinfo.net/conf/sony-xdcam-ex-cinealta/135507-freeze-mix-funtion-ex3.html
Best ArildP

I know a situation when I need the BPAV file set, when I'm using the freeze-mix funtion of the EX3.

Actually not any more, now that we can put edited video back onto SxS (at least from FCP). It's been noted elsewhere, but I'd missed it until an earlier post in this thread. XDCam Transfer 2.8 allows one to export from FCP an MP4 file that you can put back on SxS in Clip Browser, recreating the BRAV structure. This makes me very happy, and not just for freeze mix. I don't have SDI out from my Mac, and now don't need it to get to tape.

Yes, it works with FCP Rob. Great!
Best ArildP