What's the big scare about copying BPAV?

[David Morgan]

Relatively new to the Ex-1r and the file workflow. Would like some feedback regarding copying, archiving and the somewhat confusing file designations.

Seems that some of us are casual about using the Mac OS finder or Windows to simply copy the BPAV folders from the memory cards to a hard drive. I've seen warnings from Sony and some editors that the initial copy process should be done with either Sony's clip browser software or an application like Shot Put pro. Shot Put has the ability to copy the cards to multiple locations simultaneously.
So far, I've separated (in my mind anyway), the archive and copy process from the editing. Once the archive is done, then there are multiple ways to import the clips into whatever NLE your using.
I see references to file types such as MP4, MXF etc.... Part of my confusion is how the clips get, or if and when they get transcoded. I thought the XDCAM format was mpeg 2 anyway.

I am very familiar with tape based workflow. I have a very uneasy feeling about this whole file workflow process as I feel lucky when the clips somehow manage to import in my NLE (final cut pro). However, I'm looking for a deeper understanding as to what's going on at the various stages of the process.
Anyone have a link to a better understanding? I've already purchased the Vortex DVD's.

thx

[Perrone Ford]

Just a few things.

The Sony camera places it's XDCamEX codec (which you rightly stated is Mpeg2 based) inside a file with an MP4 extention. That file structure does not have the ability to carry metadata. So the metadata is stored separately inside the BPAV along with an index and other information.

Doing an OS file copy is usually fine. But there is no verification on that process and if any small glitch happens, it can make the video file corrupt, it could show up in the video or audio, or you may see no effect at all. Programs like ClipBrowser do a verification process on the files that are copied to make certain they are bit for bit accurate and can be relied on.

Some NLEs cannot use the BPAV structure and so clipbrowser allows you to re-wrap the video/audio files into a .MOV file for use on the Mac, or a .MXF for use on the PC. Not all NLEs need this done. And if it is done, you lose the metadata. There is no transcode happening here.

I am unfamiliar with how Final Cut does what it does, but Sony Vegas can use either the BPAV structure or the MXF files. It does little with the metadata though. Avid on the other hand can use either the BPAV or the MXF fles, and if using the BPAV has access to ALL the metadata it seems. Very handy.

There really is a lot less to go wrong here than there is with tape. There is not verification process other than with your own two eyes when coming off tape. Tape also cannot carry metadata. Well at least not at our level. I can't even begin to tell you how nice it is to import clips into the NLE and sort by good takes automatically.

[David Morgan]

thx, explanation is pretty clear. How about renaming? Can you rename clips at any point? Or, should you rename after injest into the NLE? If you do that, it seems that going back to the original archive files with the renaming would make finding clips a small nightmare?

[Perrone Ford]

DO NOT RENAME THE CLIPS...

If you want to create a mess, this is about the fastest way to do it. Again, Metadata is your friend. I don't know what FCP gives you, but you should be able to do anything you need with the metadata and not touch the actual clip names.

This is actually one thing I love about Avid. You CAN rename the clips inside Avid to anything you like and Avid just doesn't care at all.

[Walter Brokx]

Originally Posted by Perrone Ford

This is actually one thing I love about Avid. You CAN rename the clips inside Avid to anything you like and Avid just doesn't care at all.

Just like Premiere... It's a great way to organize inside your project.

[Olof Ekbergh]

If you want you can rename the clips in FCP as well, it does not change the actual filenames. I tend not to do this but use comment fields for notes and folders/bins for organizing in FCP.

FCP is actually a nice database and quick search program as well as an editor, like I am sure Avid is as well.

I usually drop all the clips from a project into one FCP project and keep it in the same folder as the archived clips. It is much better IMHO than XDcamTrans for making notes and viewing clips, it will play back through my AJA Kona and Matrox MX02, so you can see what you really have.

Personally I do not keep my BPAV's once I have rewrapped to .movs. Because most of my projects are shot with a lot of different cameras and formats, so keeping everything as .movs is easiest for me. I use the nanoFlash a lot and it does not make BPAV's, it uses a higher bitrate EXcodec.

So personally I don't see much use for the BPAV file structure.

But before you have established your system of archiving and working, I would definitely keep the BPAV's. It also makes it easier to send clips to someone not using a Mac. Although that is easy by just trans coding the .movs in any number of different programs like Squeeze or Compressor.

I have personally never had a problem just drag copying BPAV's, over 800 hrs worth I think. But it probably is safer to use Clipbrowser or Shotput with verification. I always copy to two HD's and check a couple of the files by playing them back, before wiping the SxS cards.

[Steve Gibbons]

For what it's worth...

We've been doing straight Windows OS file copies of BPAV folders from SxS cards to network archive storage since the PMW-EX1 came out and we have never had a corrupt file problem or anything similar; it's proven to be very reliable.

[Leonard Levy]

I've been told by a friend who is a a major computer geek that if you use the finder to drag & drop and check that the size of the file is exactly the same by checking with get info then you are safe. He says if the bits are the same it can't be corrupted.

I have had Clip Browser not see some files that were fine in the past but supposedly that's fixed tho it loeft me wary. I personally found the shot Put interface not simple enough and easy fro someone I gave the job to to screw up. Nice thing about Clip browser is that it will put many cards in one BPAV folder this connecting spanned cards. I always use it for spanned cards.

Someone else told me that Carbon Copy Cloner was also reliable because t double checked copies.

[Duncan Craig]

I use Toast to 'compare' the BPAV master to the two copies I make after I've done a drag and drop copy using the OS. It takes twice as long but ensures your copies are perfect. This is CRC checking.

Using any other system like Shotput or Clip Transfer will take just as long, the only way to make it faster is to use faster hard drives and a faster card reader.

[Dean Sensui]

Computers have error checking as a routine part of the file copy process. That goes for the Mac or PC.

If it weren't for that, you'd have all sorts of problems arising from making backups, installing programs, etc.

So you can use the clip browser for duplication and transfer, or you can just drag-and-drop, and save yourself a lot of additional effort.

[Andy Taplin]

Dean's right. There is no issue using drag and drop to copy files otherwise data would always be getting corrupted - it never does - as long as the files copy across completely the'll be fine.

Of course you must NEVER mess around with the file structure or rename files in the BPAV folder.

[Doug Jensen]

I also move BPAV folders around all the time with the Finder and have never had a problem. Clip Browser and XDCAM Transfer are important parts of may workflow, but that doesn't mean BPAV folders can't simply copied from one location to another without causing problems.

[John Peterson]

Ditto,

John

[Keith Moreau]

I have FCP and an EX1.

I use Calibrated Software's Quicktime Plugin "EX XDCAM MP4 Import" with native EX1 Files. This excellent software makes Quicktime think that the EX1 files are Quicktime and allows you to play them without transcoding anywhere Quicktime is used, including Final Cut Pro. I am really grateful that Calibrated created this plugin. It's saved me terabytes of uselessly wasted hard drive space and weeks of wrapping / transcoding time.

Finder Copies vs another tool:

Although the Finder or any OS based copy is quite reliable in normal copying, if one bit is off or corrupted in transit or if the source flash card destination hard drive is on the edge, you can get errors that are not reported in a Finder copy. It's also possible that a 'get info' would not report a difference in size. This is very rare but in fact I have experienced it a few times. Using a more fail safe method to copy, then compare the copy with the original, is more reliable. For EX1 I use the freely-supplied Sony EX Clip Browser software to perform the copies. It's easy, reliable, creates a destination directory with a unique name based on the date, and if you set the preferences correctly you can get what is called a CRC copy which ensures that the destination is the same as the original. You can then browse and playback the resultant copied EX video with it and rename the top level directory if you wish. Alternative methods, such as Shotput, or Chronosync with verification (which I use to transfer non EX flash media) is also possible, but not necessary because the Clip Browser is fast and reliable and has other browsing features.

Backing up entire directory vs other methods:

I would backup the entire flash drive to the hard drive, preserving the structure and meta data. Then you can always have the media in it's original form if you want to use it and other apps need that meta data to interpret the EX1 files properly, such as the FCP plugin or Calibrated's plugin. You can then import the files to FCP and perform the 'selects' there. I like to use 'subclips' to organize my FCP clips rather than renaming the clips. The file info is stored in the 'clip' and not altered but the subclip mechanism works well for me.

Alternatively, you can use Premiere CS5, which works natively with EX files, is faster than FCP. I am using that increasingly over FCP because it can natively work with pretty much every file type you throw at it.

Hope this helps.

[Shaun R Walker]

I use Finder to copy my BPAV's all the time. Clip Browser is too slow for my purposes. However, I have had several instances where only the video files have copied and not the two metadata folders. Luckily I always check the files after copying and if I have time also open up the folder in XDCAM Transfer and check that they copied ok, so disaster was averted. I would recommend you always check the copied files before reformatting your cards.