Archival File Format - What's Best?

[Christopher Brown]

What is the best file format in which to archive EX1 footage?

My inclination is to store all data as .MOV files because I edit in FCP on the Mac. Rather than have archival files in the Sony MP4 format and a second file set in .MOV to store with the FCP files, it makes sense to me to simply trash the original .MP4 files and archive only the .MOV files.

My ideal archival system would have two sets of .MOV files. Set 1 would be all original footage simply converted from .MP4 to .MOV and archived. Set 2 would be only the .MOV files used with the project, and any related FCP files (titles, etc.).

In the world of still photography, the original RAW image files are kept in their original file format (which is proprietary for each camera manufacturer). This data is kept with all final output files (usually RGB TIFF) on optical disk and hard drive. The risk is that as RAW file converters change with the times, the original file formats may become unreadable (this is Adobe's motivation for the .DNG file format).

Any comments on this would be appreciated.

[George Kroonder]

Chris, I am also pondering archival methods...

You are correct in that 'everything computer' changes, so uncompressed video is likely to be the most cross-systems-and-time 'future proof' way. At a cost of diskspace and possibly disk speed issues in post.

I am considering an intermediate codec, specifically Cineform, to standardise on for post and archiving. If I was capturing some of the footage to Cineform directly (via HD SDI) that would make even more sense.

Although files would be around 3x as big as the EX1's mxf's, I'm not worried about diskspace as such, that is cheap enough.

I haven't figured out an economical way for archiving or duplicates (off site) for projects >>100GB. I would need qute a few BlueRay datadisks; maybe a (external/disconnected) harddisk would be a possibility. I'm not sure how reliable that last option would be though.

George/

[Rob Collins]

My understanding is that if you trash the original MP4 data you make it more difficult to edit on other platforms and versions in the future.

My workflow at the moment is to drag the BPAV folder in the finder from SxS to an external drive. I then use XDCAM transfer to extract the .mov files to my internal RAID for editing, and back up the .mov files to DL-DVD.

[Michael H. Stevens]

The BEST & ONLY format is the original! Keep clone copies of the .mpg4. Don't go archiving something that has been converted or processed. This is a basic general rule for all data security.

[Craig Seeman]

If you hand somebody the XDCAM .MOV files created on a Mac with Sony XDCAM Transfer Tool, they can use MPEGStreamclip (even on Windows) to convert those files to another format.

Yes it's an extra step and yes it's not as simple has handing them a copy of the BPAV folder but it can work.

[Alister Chapman]

You really should keep the original MP4/BPAV folders. The XML files contain information about when the footage was shot and some of the camera settings which could prove useful in the future. By saving the original files you will be able to work with the native material across many Mac and PC platforms without any processing. If you save as something else, then if you move to another platform you may have to re-encode or process the data which will lead to degradation. It doesn't matter how good the codec is, if you have to decode and the re-encode you will introduce errors (artifacts). These may be small and may be invisible at first but if you then have to convert to something else (DVD, Uncompressed, whatever) you will add errors on top of the errors from your first pass and very quickly you will see a degradation of picture quality. Concatenation is the enemy of video production and while adding an extra encoding stage at the beginning may not show any immediate visible effects you may end up regretting it later. Once you have added an extra encoding stage you can never get back to your original quality, so my advice is to always keep the native media in it's original state whenever possible.

[Christopher Brown]

Thanks for the responses, I really appreciate it. The consensus is to keep the original MP4 data and convert it to whatever is needed for editing.

I agree, but there could come a time when the original data is not accessible. Keeping the MP4 files requires the use of Sony's Transfer utility, and this assumes that the program will be available from Sony in future versions that will run on new operating systems and new hardware.

[Dean Harrington]

http://www.bitjazz.com/en/products/sheervideo/

Graeme Nattress review on sheerVideo codec.

http://www.lafcpug.org/reviews/review_sheer_video.html

[Alister Chapman]

But as I said no matter how good the codec by re-compressing you WILL introduce additional artifacts. Say you use the sheer codec, at the end of the edit you render back to MPEG2 for HD-DVD or Blu-Ray. Your material will have gone through at least 2 stages of encoding to and from different codecs. If you keep the MP4's you can go straight from the MP4 wrapped MPEG2 at 35Mb to you chosen bit rate of MPEG2 for output. 1 Pass, 1 encode step, only one codec so no additional errors introduced through cross codec conversion.

[Greg Boston]

Originally Posted by Chris Brown

Thanks for the responses, I really appreciate it. The consensus is to keep the original MP4 data and convert it to whatever is needed for editing.

I agree, but there could come a time when the original data is not accessible. Keeping the MP4 files requires the use of Sony's Transfer utility, and this assumes that the program will be available from Sony in future versions that will run on new operating systems and new hardware.

I have to agree with Alister. Keep the original material. Sony has promised backwards compatibility with all future XDCAM products. You're going to see the XDCAM capability extended into higher bit rates in the future, but not abandoned altogether.

For robust archival, the XDCAM optical discs should last for 50 years.

-gb-

[Alexander Ibrahim]

Originally Posted by Alister Chapman

But as I said no matter how good the codec by re-compressing you WILL introduce additional artifacts. Say you use the sheer codec, at the end of the edit you render back to MPEG2 for HD-DVD or Blu-Ray. Your material will have gone through at least 2 stages of encoding to and from different codecs. If you keep the MP4's you can go straight from the MP4 wrapped MPEG2 at 35Mb to you chosen bit rate of MPEG2 for output. 1 Pass, 1 encode step, only one codec so no additional errors introduced through cross codec conversion.

Alistair you didn't read up on the Sheer video codec did you?

It's mathematically lossless. As a result your arguments against using it are without merit.

That isn't to say I recommend it.

Essentially you are getting uncompressed video, but the codec adds another layer of technology to preserve in archival applications. In addition you "only" get a compression ratio slightly higher than 2:1, making the value proposition dubious.

Disk space will still be an issue with the sheer codec unfortunately, enough so that for intermediate archival purposes I still recommend uncompressed video.

For long term archival film remains the king... Do a filmout and have that properly stored.

For short term archival storing the acquisition master is a good solution. In this case copying the BPAV folder as has been suggested. The primary concern here is that eventually Sony will drop support for the XDCAM transfer software, either on your platform or in general. You must be prepared at that time to make other arrangements.