EX Workflow - do you keep BPAV files

[Olof Ekbergh]

Craig,

The way I understand it is that XDcam is really an MPEG2 fileformat.

ProRes is Inra frame therefore the larger file size and easier editing, it is also much less lossy over generations. Am I wrong Craig?

Also HD/BluRay storage is really cheap these days, that is how I archive, I dont mind the increased size.

Right now XDcam footage is less than 30% of the file format I use for acquisition for projects I work on. I am trying to use a uniform format for all kinds of media, and so far QT has been it.

I see the BPAV folder as an added complexity I don't need.

Craig I am not arguing I want to learn, if there is a better way than the one I am using now I want to know.

[Craig Seeman]

XDCAM is MPEG2 GOP and the EX variant is wrapped in MP4.
Certainly ProRes is larger but IntraFrame is only part of the picture. After all DV is IntraFrame as well and it's smaller than XDCAM.
ProRes is not lossless like 8 or 10 bit Uncompressed but it's actually smaller (more efficient) and for most practical purposes functions almost like lossless in that in can hold up well to compositing and grading and other mild generational degradation.

XDCAM EX BPAV(MP4) is the same size and XDCAM EX MOV. The former though can easily be used natively in some systems or rewrapped for other systems that use MXF or MOV.

XDCAM EX MOV does not have the same flexibility of the above although one can certainly work around it. Rewrapping XDCAM EX MOV back to BPAV MP4 or to MXF is NOT readily available.

An XDCAM EX BPAV folder is the same size as an XDCAM EX MOV so both can take the same storage space. In fact, Sony ClipBrowser allows BPAV to be split and remain viable for recording to such things as Blu-ray. Key though is that XDCAM EX BPAV is far more portable, easy to port, to something any NLE can use . . . and these days many more NLEs can handle the MP4 contained within directly. That is NOT the case with XDCAM EX MOV. As noted above I think Apple themselves may move in that direction with Final Cut Studio.

A BPAV folder is not more complex than the cassette that holds a DV tape for Beta tape. You don't need to think about or otherwise investigate the contents. Given what ClipBrowser shows, there's metadata in there that does not go into the MOV though. Going forward the BPAV will be even easier to handle if Apple supports it directly (not that that will mean a change in support for XDCAM EX MOV).

Personally, being future thinking in my archiving, I want a container (BPAV) that I can hand to any client, use on a future project on a future NLE, without risking a potential future constraint that may come to XDCAM EX MOV. To me, that's the main point of an "archive."

Often enough I have to pull something from projects over 5 years old. 5 years is a lifetime in Computer and NLE worlds. I am every much an FCS "fanboi" having used it for 10 years, but also having used Avid 10 years before that . . . along with Adobe Premiere, Radius EditDV, CMX 6000 NLE and a couple of others in there. I want to know that 5 years from now I know I can EASILY use the material I've shot 2 and a half years ago when I bought my EX1.

In particular EX in MOV wrapper is such a proprietary combination, so not universal, that some future series of turns or decisions at Apple and Sony can impact that specific combination.

I just don't see any advantages to EX MOV over EX BPAV and I see many current and future disadvantages to EX MOV.

Additionally I think Apple might be looking at something more efficient that ProRes to come up with a more efficient codec so that there may be something to convert things to that will be both space efficient and portable. That MIGHT be an alternative if it happens.

[Olof Ekbergh]

Craig, I am looking at the big picture. Surely you are not suggesting I take all my files from different cameras as well as all my legacy stock, and convert it to XDcam. Remember XDcam is only a part of the files I work with.

As far as meta data is concerned. I don't personally use it. It does not tell me what filters, lighting etc. I was using. OK so it tells me the profile, but that is not very important to me at least. The lighting is far more important than profile in my opinion.

I just like to have all my different shots be the same easy to deal with format, and not to have to launch a special app or plugin just to deal with some of my files. To me the BPAV system is a pain, and I think it would be great if QT could deal directly with the mp4's. Like the files from the Canon 5DmkII etc.

I have a searchable DB I created myself and it contains the pertinent data I want about each shoot, or stock shot I have. It links to the files on a number of different (and redundant) HD's and optical discs. To use my files I just plug the HD in and import the ready to work with file into M100, AE, FCP or whatever I need to work on it. I don't want to have to launch another app just to rewrap every XDcam file. I want all my files to be ready to work on.

[Craig Seeman]

Surely you are not suggesting I take all my files from different cameras as well as all my legacy stock, and convert it to XDcam.

No, camera masters should be archived in their original form to maximize portability going into the future. The 2nd choice would be a universal codec (should such exist) that does not result in loss of resolution.

I don't personally use it. It does not tell me what filters, lighting etc. I was using.

It will tell you camera number, good for tracing issues.
Lens model which can be important if you need to match that in the future.
Acquisition Date
Gamma mode you used as part of Picture Profile.
Gain
Color Temperature setting
Shutter settings
ND Filter setting
CC Filter
F stop
Zoom setting
Focus indicator

I just like to have all my different shots be the same easy to deal with format, and not to have to launch a special app or plugin just to deal with some of my files.

But if you ever need to use anything in an non FCS program you may well need plugins or some way to convert that EX XDCAM MOV file.

[Craig Seeman]

Example of some of the EX metadata attached.

[Marcus Durham]

So basically the meta information is a load of stuff that I can't see a use for? Ergo it isn't worth preserving.

[Enrique Orozco Robles]

I totally agree with Graig ... if you have some important shot, just keep the original masters (BPAV folder) in a safe place... it is your master "tape" no matter what platform or NLE you use today or in the future...
with DV tapes we used to "keep" valuable original tapes and "capture" or transcode files in order to work (mov in MAC or AVI in PC) .... I feel the same with EX original masters .... my 2 cents...

[Olof Ekbergh]

Personally I find that metadata not important.

I can always tell the camera used from the file name. And I almost always use TOD as my TC, jam synced to all the cams. And I slate all pertinent data on each shoot and then use my database to put in all the data I want. This makes for really easy retrieval years later.

As I mentioned I use the NanoFlash most of the time now when shooting. The really nice thing about this is that you decide in the NanoFlash menu if you want a mxf or .mov wrapper. And it uses the XDcam codec from Sony, or an intraframe codec. I use the 100mb/sec XD codec most of the time and it is a very nice timesaver not to have to rewrap after shooting. Just copy the files.

I don't understand what Craig has against keeping all your archives and projects in one format. In my studio this makes a great deal of sense. I guess if most of your work is passed on to other facilities the BPAV makes sense.

I keep digitized copies of all my old Umatic, BetaSP, DVcam, DVCpro etc. files. Some going back to the 80's. The tapes from then have lost a lot of their quality. But the digital files are in great shape, and I can retrieve them instantly with no need to digitize over again. That is what I love about the digital domain. I am actually considering trashing a lot of old tape as the video has faded so badly. They take up a lot of space, especially the old 3/4 Umatic..

So I am just making the case for a Database system linked to digital files that can be viewed instantly using in my case QT. And it is no trouble at all to hand off footage to other production houses. I do this from time to time, most of the time to TV stations that need some of my clients footage and then I simply put it on tape, DVD, HD or upload to their FTP site in whatever format they want usually h.264 these days. Naturally I charge for this and my clients love this service.

I just don't understand what is wrong with this approach. I sell my footage on Getty and they want ProRes these days, so that is how I deliver to them. The same format I use, there must be a reason they use this format. They don't want me to send BPAVs.

[Craig Seeman]

As noted before, Pro Res is the one thing that trumps others as a "universal" codec. I suspect Apple is about to improve on that.

That metadata is important for matching shoot conditions. When a client wants to update a 5 year old project with a few new inserts I can match the shoot conditions. The above metadata tells me stuff I could never match by eye years later. Things such as DOF, color temp and how I got there (3200k will look different with preset vs white balance card). It tells me the gamma setting I used. I can get close to the look of a shot so the insert doesn't look out of place.

When my clients ask for master clips, ProRes is too large, EX MOV is not compatible, BPAV gives them complete flexibility. EX MOV is fine for FCP users but most of my clients are not. Corporate clients are primarily on Windows (as is most of the world FWIW). I can move from BPAV to ProRes just as easily as from EX MOV to ProRes. I see many advantages to BPAV and very few for EX MOV. It's the reason most NLEs now use native files in native wrappers and why I believe Apple/FCP will soon follow suit.

[Olof Ekbergh]

So Craig how would you handle NanoFlash files?

Also I really doubt that I will be using the same cameras 5 years from now, I now use mostly LED and fluorescent lights, I used to use HMI and tungsten. I really have very little problem matching old footage with new. People age too. Many times I update a corporate video and the spokes person has changed as well as the company logo. I update legacy projects all the time mixing BetaSP with XDcam from the NanoFlash and currently XD EXcams.

I think in the near future we will all be using large chip cameras, hopefully with RAW codec, that will blow todays cameras out of the water, much better low light and greater dynamic range, just like the XDcams blew the old formats (DVcam DV HDV etc.) away. I think the 5DmkII started a revolution, it will take a while but it is coming. And I also believe QT will support all these new codecs.

[Craig Seeman]

NanoFlash I'd keep in whatever native form you recorded them to. That would probably be .mov extension although I'd love to seem them use the more universal .mp4 rather than .mov or .mxf. It's funny how wrappers change. In world full of non standards .mp4 really is becoming universal.

Even Apple has played games with this. Note that .m4v used in video for iDevices is really just .mp4 but calls iTunes for example on opening. You can actually change .m4v to .mp4 as the metadata internal to the wrapper is the same. That can not be done with .mov (it can't be change to .mp4 or .m4v since some of the "internals" are in different places). This is another indicator that Apple is really looking to support .mp4 even if the wrapper extension may appear different.

Yes cameras are going to go to large chips very soon as Sony indicates with their $2000 interchangeable lens camera . . . which records to AVCHD. I'd expect the "pro" version by NAB next year or sooner. It won't be RAW though as the data rates are still too high for most recording media. I can't guess where Sony is going, whether such camera will be AVCHD, AVC Intra or, MPEG 50 4:2:2 or an MPEG I frame format.

Granted there are many ways to match settings are accept that they can't match but the metadata is definitely helpful to me. Even with different cameras it quantifies a lot about the "look" of the shot. In some respects it could even be more critical since that info can help match things. Knowing the narrow nature of the focus and the focal point of the subject can give me an idea where to go to set up a similar shot. Obviously it won't mean anything if you're not trying to do that . . . but I have and I am.

BTW given the movement in cameras and codecs it's all the more reason I think EX MOV is the "odd bird." It's way to proprietary (tied basically to one NLE) for me to consider that "archive."

As to the new codecs, I'm not sure if they'll be "direct" support in Quicktime as you now think of it. I think Apple will move to native support although it may be through the "tricks" that make the system think it's Quicktime as Calibrated Software does.

[Steve Kalle]

Craig and Olof: PLEASE just agree to disagree on this subject :) I know Olof has been doing this for decades and he has been successful; thus, he knows what works for him and his company. Craig has been obviously doing this for years and knows what works best for him and his company. It sounds as though Olof will never ever move from FCP whereas it sounds like Craig might but is waiting for FCP 4; thus. his trepidation about QT/MOV and native support.

I have been doing this for exactly 1 year and I don't even have a solidified workflow yet. So far, all I do is download the cards through my Siig expresscard to PCIe adapter, save all cards from one event to a single folder with a naming convention like this: xx VIDEO/2010/August/08-03_National_Night_Out_Cam1
After all cards are downloaded, I copy the new folder to both of my 2TB e-sata drives. Because I use Premiere, I never transcode and always keep the BPAV. Furthermore, I might add an Avid workstation for a 2nd editing workstation so I need to keep all options open.

For edited audio, stock video/audio and images, I copy and paste to another pair of 2TB drives (1 internal and 1 external). For everything else, I use Acronis Backup & Recovery 2010 and I use a "Grandfather" scheme - it backs up 3 days a week, once every week and once every month. It keeps the last 3 daily backups, last 3 weekly backups and last 3 monthly backups. This works great for going back to a prior version of a Premiere or AE project file, which I have only done with AE so far. Also, I have separate backups for my PPro files, AE files and Cinema 4D files.

Like I said before, another benefit to the BPAV folder is being able to use Flash Band removal plus whatever other features might be added to ClipBrowser down the road.

Olof, now that I think about it more, I think Craig is trying to convince you that FCP 4 will move to native support; therefore, you won't need to keep the MOV files because FCP 4 will be able to use the BPAV or MP4 at the very least.

[Craig Seeman]

Steve, you bring up some additional good points. ClipBrowser may introduce or improve features. FlashBand is a good example, Imagine, when the time comes to reuse the source copied pre FlashBand, to improve on that.

Regarding FCP, one doesn't know what changes are in store for it on many levels. Think of Media 100's history for example . . . all the ownership changes and some of the derivative systems and features.

Even though my approach is different than Olof (BTW I go back to 2" tape dub days) I think readers may want to see why we've arrived at our different places so they can evaluate their own directions. We've seen that in other parts of this forum where the veterans have significant reasons for very different workflows. Better to know a lot about all the options than not enough.

BTW I generally think it's no fun switching edit systems but I've had to do that at times in my life for reasons out of my control. There may come a time when the hardship of switching may be easier than the hardship of staying. This is especially so when a system goes away. EditDV and Discreet Edit* come to mind (although I never used the latter the possibility had come up). I certainly was THRILLED to go from CMX (how could you ever switch from a leading company) 6000 to this "start-up" called Avid in 1989. About 12 years later I was eyeing FCP closely although it had too many weaknesses at that point. By FCP 3 is was much easier to pry my hands away from Avid. For me, I have learned that "forever" generally only exists in the present tense.

[Matt Davis]

I've sat on my hands whilst lurking on the thread, but have finally jumped in.

Craig and I have 'agreed to disagree' over on Vimeo - we both agree the other person has valid points, but our workflows differ too widely. For my part, I love the way XDCAM Transfer enables me to batch name movie files and embed metadata that shows up in Final Cut Pro - I can find footage using CDfinder, I can see metadata useful to me (who is this interviewee? What is the tummy tag info I need? Who do they work for?). When other shooters give me BPAVs to download using badly setup FTP sites, CalibratedQ comes to the rescue. When shooters give me BPAVs to download with no log notes, slates or even audio cues, I thank them dearly for letting me know they shot at f4 and at a colour temperature of 2960 Kelvin. So, when making a compilation of Mr X, top executive of whateverco, it's often the time it was shot that helps me, late at night, piece together who is who and what they're called.

Now, IF they had used XDCAM Transfer, AND they had typed in the info, I would get metadata that is USEFUL to... AN EDITOR. Heard of these guys? They get loads of footage flung at them, and as time goes by, the concept of 'logs' and 'slates' seem to have been forgotten because of 'metadata'.

But I will mention this: Craig is right to keep an eye on the future, as the future has this really tricky habit of becoming the past really quickly. I have material from the mid 1990s and earlier, stored as Macromedia Director 4 and (shudder) Accelerator movies that cannot be viewed now. They weren't kept up to date.

I worry slightly less about the movies I have in QuickTime format as I do worry about the edits I have in FCP1 format. Movies I made in the early 1990s are in MPEG1 format and they still play. The various format files (PCTS?) that some were archived in don't work now - and the cost of conversion aren't worth the cost of getting a consultant to work out how to unpick them, sadly.

The trouble is, in 30-50 years time, who will be able to unpick a FCP file, or even a BPAV or MOV for that matter? I foresee having to translate formats every 10-15 years on proper 'archive' stuff. PhotoJPEG QT was good for a while, but what will take over from that?

Meanwhile, I will remain naughty and stick with the 'keep your MOV, bin your BPAV - unless it's worth something in 10-15 years, in which case back it up in lots of formats' camp. And don't forget to make an XML EDL of your FCP edits! :)

[Alister Chapman]

Don't mix up MP4 and mp4, they are slightly different things.

.mp4 is an extension normally used on video files encoded using the mpeg4 codec also known as H264

.MP4 is a multimedia container defined by the MPEG group (MPEG 4 Part 14) that can be used to simultaneously contain any number of media streams including video, audio and data such as subtitles and still images. It is actually based on the Apple Quick Time .mov container but includes extra information about what is contained within the wrapper. The video and audio streams it can hold may be compressed using almost any kind of mpeg from mpeg 1 to mpeg 4 plus a few others as well. It's similarity to .mov is one of the reasons why it's easy to convert XDCAM EX files to .mov

This makes the MP4 wrapper very versatile and cross platform compatible.