With MXF export files, is it really necessary to keep BPAV files?

[Jay Gladwell]

Originally Posted by Bill Ravens

Long form GOP(eg Sony mxf) is designed to be a camera storage medium. It is NOT an edit medium. Sure, you can edit with it, but, that comes with its own set of problems. There's lots of folks who will claim they have no problems editing mxf, and I won't argue. But, at a minimum, they are incurring multi-generational losses on a grand scale.

Bill, so far (as we've discussed) I haven't edited in HD and rendered to HD (haven't gotten that far yet), just SD.

Do I understand you to say that editing with the Cineform files eliminates recompression when working totally in HD?

[Bill Ravens]

Hi Jay...
Using CFHD doesn't totally eliminate recompression losses, but, if you visit their site and look at their examples, 10 generations and the image still holds up. CFHD is a very dependable and predictable codec...no surprises, no failures. Cineform has frequent updates. They seem to be a wee bit sloppy, making software mistakes as they go thru all these evolutions, but, all in all, well worth working with Cineform as a routine part of my workflow. One of their options is to convert to SD when you transcode. For my money, it's too early (i.e. before editting) to downconvert, but, I've heard positive results from their downrez algorithms.

[Peter Kraft]

Originally Posted by Bill Ravens

There are many ways to skin this cat. Exporting to Sony mxf is absolutely identical to the original mp4, no reason to save one or the other, as long as you archive one. You can also export directly to Avid AAF, but, AAF files lose their link to the source material in Avid, so you have to be really careful to backup your data with the BPAV info.

Jerry's initial question was: With MXF, do I have to archive BPAV?
Your second sentence is the answer: NO. Period.
And BTW, the original is MP2 (in an MP4 wrapper), not MP4. Don't confuse BPAV with AVCHD.

IMHO, the absolute best way is to do a Cineform conversion directly from the BPAV folder, then archive the CFHD.AVI file. Sony .mxf will not import into all NLE's. CFHD will. Within Avid, a CFHD file does not need to be transcoded to DNxHD. It can be imported to whatever DNx resolution you select...very convenient. CFHD also imports as 601/709, every time. Certain NLE's, like Vegas will play games with the superwhites that you have to be very careful about wrangling in order to keep your images undistorted.

AVI is just another wrapper like Quicktime and MXF but with lots of probs coz Microcrap did not get it for years how to write a good wrapper. Why this additional wrapper without additional benefit and why another codec?
Plus: Avids do read MXF.
And concerning Vegas: It does not make sense to run in circles just because some software authors play games. Better quit them and choose a product that fulfills essential requirements. Let your purse vote. Professional results require professional tools. Nothing less. If you count the time to run circles, it costs you lots of coins more than what the app did cost you less.

Long form GOP(eg Sony mxf) is designed to be a camera storage medium. It is NOT an edit medium. Sure, you can edit with it, but, that comes with its own set of problems. There's lots of folks who will claim they have no problems editing mxf, and I won't argue. But, at a minimum, they are incurring multi-generational losses on a grand scale.

That used to be so - in the early days of MP2. With the advent of far more powerful PCs, lots of RAM, things have changed fundamentally and MP2 Long GOP has grown to a full scale edit format. Take HDV or XDCAM on Avids, Liquids, Macs to name but a few.
On the other hand, MXF is a wrapper, no format (or codec), don't mix it. MXF is worldwide THE interchange format for video between broadcast organizations. So the question is, can the NLE read a certain MXF file AND does it understand the content in that MXF file, can it de-cipher the data in that file and is the required codec available?

I can therefore not understand why you talk about generation losses. A digital copy is as good as the original. If you change the codec and have to re-render, than you get a generation loss. Your above mentioned CFHD.AVI incurs a generation loss, not MXF with MP2 Long GOP content.

I just can't imagine the logistic nightmare of saving BPAV folders. Totally unnecessary and cumbersome.

I could not agree more. But IMHO MXF is THE way to skin the cat.

With all due respect. P.

[Bill Ravens]

Peter...

Thanx so much for expressing your point of view, however, you have some misinformation.
First of all, if you want to playback or decode the mp4 file in the BPAV folder, you need an AVC codec, wrapper not withstanding.

Secondly, mxf is not a universal format. there are several versions which are incompatible with one another, depending on the OEM. Sony mxf, for example, is not the same as Avid mxf. As you said, mxf is just a wrapper. What's contained in the wrapper is as variable as avi or mov.

Lastly, CPU processor speed has nothing to do with generational losses of a video codec. Long form GOP is still a interframe compressed mpeg2 codec. It will suffer a loss every time you go thru the decompress/recompress process, and it doesn't matter how fast the processor does this. CFHD is an intra-frame comression scheme. If I can quote cineform: " None of these formats were designed for post-production – they were either designed for capturing video on tape (DVCPRO HD), distributing video (MPEG), or still images (JPEG). Their application in a post-production environment often introduces unwanted characteristics including: i) generally poor multi-generational quality, ii) 8-bit arithmetic processing limitations, iii) specialized hardware, and iv) lack of a compatible forward path to higher resolutions (Cinema 2K/4K) or arithmetic depth (10-bit, 12-bit). In most cases these codecs are not worthy of use in a Digital Intermediate workflow, and is the reason many have felt that "compression" within DI is a "bad word". " While the penalty you pay is larger file sizes, the silver lining is that you don't suffer compression/decompression losses. A digital copy is NOT an exact duplicate if it has been decompressed/recompressed. May I refer you to this URL: http://www.cineform.com/technology/C...termediate.htm

I still read about people having problems playing back long form GOP codecs. They see a bit of stuttering, simply due to pipeline roadblocks, I/O limitations, whatever is taking up CPU cycles. This doesn't happen with CFHD.

With all due respect.

[Peter Kraft]

Originally Posted by Bill Ravens

First of all, if you want to playback or decode the mp4 file in the BPAV folder, you need an AVC codec, wrapper not withstanding.

Bill, sorry... Wrong. The video format of the EX1 is MPEG2. See picture below, EX1 OpsManual Page 124. MP2 in an MP4 wrapper. No AVC codec needed, but an MPEG2 codec.
SP Mode files are basically HDV files which after rewrapping for example to QT run natively in Final Cut Pro or any other app with the proper (Calibrated) codec.

Secondly, mxf is not a universal format. there are several versions which are incompatible with one another, depending on the OEM. Sony mxf, for example, is not the same as Avid mxf. As you said, mxf is just a wrapper. What's contained in the wrapper is as variable as avi or mov.

I did not want to say that MXF were a universal format but it is a worldwide acclaimed and proven container that fulfills what professionals in the video and broadcast industry need and require. Mainly for cross plattform file exchange. Notwithstanding the fact that MXF comes in several "flavours".
As a matter of fact, when re-wrapping a BPAV file to MXF it keeps its entire MP2 content, therefore I made intentionally the distinction between being able to read a file (container) and to understand or decode its content (codec).
So the Sony MXF is as good as the corresponding BPAV file.

Lastly, CPU processor speed has nothing to do with generational losses of a video codec.

Where did I say that?

Long form GOP is still a interframe compressed mpeg2 codec. It will suffer a loss every time you go thru the decompress/recompress process, and it doesn't matter how fast the processor does this. CFHD is an intra-frame comression scheme. If I can quote cineform: " None of these formats were designed for post-production – they were either designed for capturing video on tape (DVCPRO HD), distributing video (MPEG), or still images (JPEG). Their application in a post-production environment often introduces unwanted characteristics including: i) generally poor multi-generational quality, ii) 8-bit arithmetic processing limitations, iii) specialized hardware, and iv) lack of a compatible forward path to higher resolutions (Cinema 2K/4K) or arithmetic depth (10-bit, 12-bit). In most cases these codecs are not worthy of use in a Digital Intermediate workflow, and is the reason many have felt that "compression" within DI is a "bad word". " While the penalty you pay is larger file sizes, the silver lining is that you don't suffer compression/decompression losses. A digital copy is NOT an exact duplicate if it has been decompressed/recompressed. May I refer you to this URL: CineForm Technology

Again: The question was not which codec is best for post-pro but does Jerry have to archive the complete BPAV folder structure or is MXF as good as BPAV. And I repeat:
MXF is identical to BPAV, not need to bother with both.

And again: CFHD is one generation behind the BPAV original or its identical twin Sony MXF.

If you transcode from MXF for post pro, take DVCProHD, CFHD, ProRes as whichever other interframe codec you prefer. But that is a different topic.

Best P.

[Bill Ravens]

huh????
must be a language problem 'cuz, I don't understand

[Rob Collins]

Originally Posted by Bill Ravens

I just can't imagine the logistic nightmare of saving BPAV folders. Totally unnecessary and cumbersome.

I've saved all mine for almost a year, and I wouldn't call it a nightmare. I just drag the foler to two external drives, and then import QT from one of them (I back up the QT to DVD).

Based on the disagreements in this thread, I don't plan on changing my workflow any time soon...

[Leonard Levy]

You guys have totally left me in the dust here.

Can anybody recommend a site that will explain MFX, BPAV, wrappers, AVI, AVC, Cineform , CFHD etc.

[Perrone Ford]

Originally Posted by Leonard Levy

You guys have totally left me in the dust here.

Can anybody recommend a site that will explain MFX, BPAV, wrappers, AVI, AVC, Cineform , CFHD etc.

You don't need a site. Let's try it this way:

Most cameras record a type (format of video). In the case of the EX1 it records a file using mpeg2 compression. This kind of compression is also used for DVDs. Sony's implementation of the mpeg2 codec is newer than most. This could present a problem for some trying to place that file into their editors. So the mpeg2 file is "wrapped" into a more standardized format called MXF or "Media Exchange Format". This is supposed to make it easier to share that file from place to place and make it more recognized by more programs.

When the EX1 writes the files to the SxS card, it writes them in a folder structure. That structure is called BPAV. If you open up a BPAV folder on your computer you'll find that it contains a specific folder structure with files in them. The Sony Clipbrowser software takes this proprietary file structure, and extracts an mpeg2 file wrapped up in an MXF file container. So the filename has the structure xxxx.MXF.

There are three fairly common file "wrappers" out there and some smaller known ones. On Macs the most common is the Quicktime wrapper. It has the extension .MOV. On the PC, the most common wrapper is the .AVI. Inside of those wrappers you can have all kinds of files. When you open your NLE and tell it to open a file, it looks at the wrapper to see if this is a wrapper it knows about. If so, it allows you to put the file on the timeline. If the compression method used to create the file inside that wrapper is known to your NLE, then it displays the video and audio. If it doesn't know the compression method (codec) then the file opens on the timeline but may be missing the video or audio portion, or maybe both.

In some cases, the codec is so specific that it doesn't need a wrapper. An example of this is Windows Media Files. These are created by Microsoft encoders and always have a .WMV extension. It is it's own file type inside it's own wrapper. No other wrapper is needed. Another is .mp3 audio. It's a very specific type of audio that needs no wrapper.

Now let's talk about codecs. There are generally 3 types.

1. Codecs used to capture video. These codecs need to take a lot of raw data being thrown at them by the camera, and write them quickly out to tape or media card. They are far less concerned with how easy it is to DECODE the information they ENCODE. These are also known as acquisition codecs. The Sony EX1 uses Mpeg2, the Panasonic HVX uses DVCProHD.

2. Editing or intermediate codecs. Once we get the video back to the editor, we need something that will hold up while editing. It would be nice if they were easy on the computer CPU so that they would play in real time even as we make changes to them. Very few codecs are truly great at this. An exception to this is the Cineform codec. It holds up great while editing, and is very fast. When I convert my HD files to this codec for editing, they move like miniDV files on the timeline.

3. Mastering or finishing codecs. These codecs are used to store all the information we want saved once we are done editing. In some cases they are very specific like mpeg2 for writing to DVDs, or AVCHD/Mpeg4 for writing to BluRay. In some cases, they are more generic like when we want a lossless codec to make sure that we have an EXACT representation of our finished work in case we want to produce something from it later.

In some cases, the codecs can pull double or triple duty. In the case of the Panasonic cameras, their DVCProHD codec is often used in all three phases. It's not necessarily ideal for all 3, but the time saved in not having to transcode from one thing to another is a real help when speed is of the essence. This is one reason these cameras have taken such a stranglehold of broadcast news. You can shoot, edit, and master without ever having to change codecs and it saves a lot of time.

Cineform works great as an editing codec,and it's not bad as a mastering one either. I've mastered some of my HDV projects this way. It produces fairly small files for as high a quality as they are. CFHD is shorthand for CineformHD.

Other times, I'll want to really master to something that is totally lossless. and when I want that, I may render with the PNG Lossless codec inside a quicktime container. Or use the Blackmagic lossless codec inside the AVI container.

Does this make things clearer for you?

[Peter Kraft]

Thx Perrone for your clear and very explanatory words.
Wish I could write like you do. P.

[Jay Gladwell]

Originally Posted by Perrone Ford

... I may render with the PNG Lossless codec inside a quicktime container.

How is that done?

[Bill Ravens]

Perrone...

well said!

Jay...

If you have Quicktime Pro, one of the export options is PNG. PNG is an uncompressed format like like HuffyUV, but, more oriented to single frame image sequences. It works well, but, makes HUGE files.

[Perrone Ford]

Originally Posted by Jay Gladwell

How is that done?

Jay, yes, Bill is right. However, there are several types of PNG. On my implementation, I can select Best (uncompressed), or adaptive, or several others. Adaptive seems to give me very little visual loss with smaller files.

[Jay Gladwell]

Thanks, Bill and Perrone.

I have what is probably a stupid question (once again exposing my ignorance of such things): Why can't someone--the NLE makers, the codec people--come up with a codec that renders a bit-for-bit copy of the original without any loss? Why is that so difficult?

<buttoning up my trench coat>

[Bill Ravens]

Jay...

It's called "uncompressed". And the drawbacks are gigantic file sizes and most computer systems can't playback because the image thruput is too high. The bottleneck is the playback speed of the storage media. If you stop to think about it, a 1920x1080 image is about 1.2Gbps at 24fps( I think I did that calc right...;o)). Modern hard drives(non-RAID) can do about 700mbps. Codecs get around this by compressing the bits and offloading some of that thruput on the CPU to regenerate the compressed images into visual images. The compression process does so by throwing away data bits that can never be recovered, only estimated. A compression scheme that doesn't lose the initial data doesn't exist(yet). There are near to perfect compression schemes, but, they incur very large file sizes and thruputs. Can't get something for nothing. On the positive side, codecs like Avid's DNxHD at 220 Mbps are nearly perfect. 220Mbps is a 2K format. 440Mbps is a 4K format. Super16 is a 2k format. 35mm is a 4K format...just to put things in perspective.

I think it's funny that film resolution has become the defacto standard for image quality. People complain that video, because of its sharpness, looks too videolike and not film like. Oh well, I'm drifting off topic.