STORAGE / WORKFLOW Help

Mike Chandler · August 26th, 2009, 07:06 AM

Similar to Olof's. I use two sxs 16's, two sxs 8's, and the phu-60, then two g-tech 500gb raid mini's to put identical copies of the BPAV folders on, dump, and then erase cards/phu.

It would be better to not have to dump during a shoot, as it's an accident waiting to happen, especially after a 12 hour day shooting, but I'm not yet trusting enuf of the mxr solution.

After shooting, dump the two field drives onto two identical 2 TB FW raid 0's (0+1) for editing, plus put original BPAV's on a raid 1 archive drive, plus backup to Blu-ray. Erase the field drives.

When the two edit drives get filled, I'll switch to a sata raid 5 unit.

Doug Jensen · August 26th, 2009, 09:08 AM

Quote:

Originally Posted by Steve Gibbons

What are people typically doing "in the field" with their EX footage?

Keep an eye out for this coming out this fall. Most people should be able to get by with having only two SxS cards in the field.

PXU-MS240 Mobile Storage Unit From Sony

You'll be able to off-load cards 10x faster than you can shoot them.

Tyler Smith · November 5th, 2009, 05:38 AM

I've been doing quite a lot of research recently on archiving.

For the cost and convenience hard drives are definitely the go, but there is a big issue of drives seizing up after sitting on a shelf for a couple of years. They seem to need to be spun regularly to keep the lubricants from solidifying.

Maybe the Drobo would help with this, but it will fill up eventually and you might have the same problem. Incidentally, I haven't figured out how the data is spread across the drives, and what the full drives actually contain.

I prefer solid state media for archiving, and I tried going down the Gold Archival DVD road (through ProDisc), but not only is it bloody expensive, it's also painfully slow and labour intensive (relatively speaking). Not a good trait for maintaining an archive system - pity because it seemed like the most stable option. Unfortunately, Blu Ray doesn't seem to much of an archival medium, as the pits are really small and therefore more error prone for storage.

Maybe Solid State Drives could be an option for "the future".

By the way, good idea about giving the clients a hard drive, that makes a lot of sense.

Brooks Graham · November 5th, 2009, 02:51 PM

I've recently done quite a bit of research on the question of flash for "archiving" and the clear message I found was that flash memory is the worst possible choice for long-term or long-ish-term storage. Well, maybe it's better than rice paper.

Most of the conversations about flash focus on the longevity of the devices - i.e. 10000 erase-write cycles for MLC and 100000 cycles for SLC - but people don't often go into the retention of data on flash devices. The basic fundamental principle of how flash works is that each cell contains a floating gate that either holds a tiny charge or it doesn't. As that charge leaks away over time, the cell's contents become questionable.

The short of it is that retention is interactive with usage, so the more a device is written to, the shorter the retention period will be. This can sometimes end up being months and not years. And some consumer devices use really bad flash controller chips and allow "silent errors" to creep in undetected with the reasoning being that they're just consumer devices and just being used for silly photos and music and nobody will notice or care if a bit gets flipped now and again. To extend the retention of flash devices, you need to periodically (every few months or so) refresh/rewrite the data. This, of course, shortens the retention period of the device even further. Not a good way to go.

I know they're nice and small and easy to store and cheap and many here have a strong attachment to the idea of using consumer grade flash devices for long-term storage, but knowing what I now know, I won't trust any consumer MLC device to retain data beyond several weeks.

Marcus Durham · November 5th, 2009, 03:01 PM

Hard drive and blu-ray/DVD for the projects inc footage and separate blu-rays just for the footage as a separate backup. Works for me. The raw footage also goes onto another HD which is stored on a separate site.

Make sure you spin the drives up from time to time though.

The mistake some people make is to assume archiving is "forever". Storage technology is moving so fast that in 5 years time they'll be a better, larger more reliable and cheaper solution. So just work on the basis that you are only storing the data on the medium term because you'll be moving it anyway in the future.

Media deterioration is an issue pretty much whatever digital format you use. I have premium Gold CD-R's from 10 years ago that won't play, but the cheap unbranded turquoise ones are just fine. Go figure! I would take all claims about media life with a pinch of salt. After all, if you look at most of the guarantees you'll find all you are entitled to is replacement media. Not much comfort if you've lost your only copy of a project.

Brett Sherman · November 8th, 2009, 09:11 AM

Quote:

Originally Posted by Jason Leigh

To keep all the raw files of all jobs is insane in regards to space.

I don't agree at all with this assessment. This is exactly what I do. Drives are so cheap now, that there is no reason not to keep all your full projects. I don't use RAIDs - don't trust them. They do nothing to protect against file system failures. Instead I use a mirrored drive. Basically I use a trayless SATA drive tower and have sets of drives, for example "Media 05" and "Media 05 Backup". I use Sync Backup to synchronize the drives nightly. When the drives are full I stick both on the shelf and get 2 new drives "Media 06" and "Media 06 Backup" I use Western Digital Green drives because they are fast enough and run a lot cooler (to my mind this makes them less likely to fail, but I don't have empirical evidence of that).

Additionally, when I'm done with a project a render out a ProRes self-contained Quicktime master, a version without music and titles, and individual audio tracks stored on an Archive drive with it's own mirrored copy.

There are a million ways to archive and backup. I've come up with a system that works great for me.

Perrone Ford · November 8th, 2009, 02:09 PM

"I don't use RAIDs - don't trust them."

You do realize that mirrored drives = RAID 1?

Brett Sherman · November 9th, 2009, 08:50 AM

I use the term "mirrored" loosely. It's not a real-time mirror, but rather a nightly synchronization. This eliminates the risk of a file system failure affecting both drives.

Perrone Ford · November 9th, 2009, 08:58 AM

Ah ok. Gotcha. The terminology threw me.

Annen James · February 18th, 2010, 05:30 PM

As far as continuing the workflow into FCP, would you next:?

1) Open FCP
2) Import media (recently converted .mov's)
3) Set timeline settings to ProRes422
4) Drag clips to timeline and edit away

??

Quote:

Originally Posted by Mitchell Lewis

Okay I thought it might be fun to share our workflow. Maybe it will help someone. We produce television commercials and corporate videos so 99% of our footage/projects must never be lost as someone has paid us to produce them.

Here's what I'm calling THE LAZY BUT CAUTIOUS WORKFLOW. It's been working really well for us since January of this year. We haven't lost a single piece of footage. (knocks on wood)

INGEST
1) Connect the SxS card to my MacBookPro
2) Connect a Western Digital 500GB buss powered FW800 external hard drive to MacBookPro
3) Launch XD CAM Clip Browser
4) Using XD CAM Clip Browser create a new folder on the WD 500GB external hard drive
5) Using XD CAM Clip Browser drag all clips from the SxS window into the WD 500GB hard drive window
6) If there is a second SxS card, insert it, select all the clips and drag them into the same folder. This is important because any clips that might have spanned the two cards will be automatically joined together into one clip.
NOTE: This is my BPAV archive drive. Once it fills up, I put it on the shelf and purchase a new one.

EDIT
1) Move the WD 500GB hard drive to our MacPro tower
2) Launch XDCAM Transfer
3) Click the ADD button and navigate to the new BPAV folder on the WD 500GB hard drive
4) In XD CAM Transfer name all the clips (sometimes just selecting all of them and giving them all the same name)
5) In XD CAM Transfer sort through the clips and label the good clips with OK
6) Change the Status View so that only clips labeled OK are shown
7) Go to XDCAM Transfer>Preferences>Import Tab and choose a location to save your MOV files to. We save them to our fiber channel RAID.
8) Select all the clips and click the IMPORT button

BACKUP
1) We have a FW800 drive that we use to back up our RAID. We use Apple Time Machine because it's easy, and we don't forget do back up on a regular basis.
2) Our RAID is striped RAID-5 so if one of the drives suddenly goes bad, the RAID switches to RAID-0 mode and you can continue editing until you replace the bad drive.

ARCHIVE
1) When the RAID gets full, we use EMC Retrospect Backup to archive our project files (MOV videos, graphics, audio, everything except the BPAV folders) off to Western Digital 500GB portable FW800 hard drives. Retrospect automatically flows the data from one drive to another as the fill up. But the best part of using this software for archiving is when it comes time to restore. You simply launch the program, find the data or project you want to restore, select RESTORE and the program tells you want drive(s) to install. Once you install the drive it automatically starts the restore process.
2) As discussed in the INGEST description, we save all our BPAV folders on to separate portable hard drives. Worst case if we were to loose all our project files, we would at least still have the original footage. Also, we'll sometimes re-import footage from the BPAV files for use in other projects.

Let me know if anyone finds this useful. (or is pissed off that I took the time to post it) :)

Mitchell Lewis · February 18th, 2010, 07:02 PM

Quote:

Originally Posted by Annen James

As far as continuing the workflow into FCP, would you next:?

1) Open FCP
2) Import media (recently converted .mov's)
3) Set timeline settings to ProRes422
4) Drag clips to timeline and edit away

??

Close.... Here's what I would do:

1) Launch FCP
2) Close any open projects (not even an Untitled project should be open)
3) Go to Final Cut Pro>Easy Setup
4) Choose the Easy Setup that matches the format you recorded the majority of your footage. I normally shoot in XDCAM EX 1080p 30 VBR (VBR = HQ and CBR = SQ)
5) Import media (your recently converted .mov's)
6) IMPORTANT: Go to Final Cut Pro>User Preferences - Render Control tab and change the Render Codec to: Apple Pro Res 422 [HDV,XDCAM HD/EX/HD422 only]

So in summary, what you've done here is set up a project where all new sequences will be created with the XDCAM EX codec, but anytime you RENDER, the render file will be ProRes codec (which yields better results than rendering in XDCAM EX). Buy doing this, you can drop your clips into the timeline and they will play without rendering (assuming your hardware is fast enough....which it normally is).

Hope that helps! :)

August 26th, 2009, 07:06 AM	#31
Mike Chandler Major Player Join Date: May 2008 Location: Moab, UT Posts: 264	Similar to Olof's. I use two sxs 16's, two sxs 8's, and the phu-60, then two g-tech 500gb raid mini's to put identical copies of the BPAV folders on, dump, and then erase cards/phu. It would be better to not have to dump during a shoot, as it's an accident waiting to happen, especially after a 12 hour day shooting, but I'm not yet trusting enuf of the mxr solution. After shooting, dump the two field drives onto two identical 2 TB FW raid 0's (0+1) for editing, plus put original BPAV's on a raid 1 archive drive, plus backup to Blu-ray. Erase the field drives. When the two edit drives get filled, I'll switch to a sata raid 5 unit. __________________ www.packcreekproductions.com EX3, Mac Pro, FCP

November 5th, 2009, 05:38 AM	#33
Tyler Smith New Boot Join Date: Jul 2004 Location: Sydney, NSW Australia Posts: 24	Hard Drives for Archiving? I've been doing quite a lot of research recently on archiving. For the cost and convenience hard drives are definitely the go, but there is a big issue of drives seizing up after sitting on a shelf for a couple of years. They seem to need to be spun regularly to keep the lubricants from solidifying. Maybe the Drobo would help with this, but it will fill up eventually and you might have the same problem. Incidentally, I haven't figured out how the data is spread across the drives, and what the full drives actually contain. I prefer solid state media for archiving, and I tried going down the Gold Archival DVD road (through ProDisc), but not only is it bloody expensive, it's also painfully slow and labour intensive (relatively speaking). Not a good trait for maintaining an archive system - pity because it seemed like the most stable option. Unfortunately, Blu Ray doesn't seem to much of an archival medium, as the pits are really small and therefore more error prone for storage. Maybe Solid State Drives could be an option for "the future". By the way, good idea about giving the clients a hard drive, that makes a lot of sense.

November 5th, 2009, 02:51 PM	#34
Brooks Graham Regular Crew Join Date: Nov 2008 Location: San Francisco, CA Posts: 121	I've recently done quite a bit of research on the question of flash for "archiving" and the clear message I found was that flash memory is the worst possible choice for long-term or long-ish-term storage. Well, maybe it's better than rice paper. Most of the conversations about flash focus on the longevity of the devices - i.e. 10000 erase-write cycles for MLC and 100000 cycles for SLC - but people don't often go into the retention of data on flash devices. The basic fundamental principle of how flash works is that each cell contains a floating gate that either holds a tiny charge or it doesn't. As that charge leaks away over time, the cell's contents become questionable. The short of it is that retention is interactive with usage, so the more a device is written to, the shorter the retention period will be. This can sometimes end up being months and not years. And some consumer devices use really bad flash controller chips and allow "silent errors" to creep in undetected with the reasoning being that they're just consumer devices and just being used for silly photos and music and nobody will notice or care if a bit gets flipped now and again. To extend the retention of flash devices, you need to periodically (every few months or so) refresh/rewrite the data. This, of course, shortens the retention period of the device even further. Not a good way to go. I know they're nice and small and easy to store and cheap and many here have a strong attachment to the idea of using consumer grade flash devices for long-term storage, but knowing what I now know, I won't trust any consumer MLC device to retain data beyond several weeks. __________________ EX3, Z1U, FX1, D1X / NeXTstation Turbo Color / FCS3

November 5th, 2009, 03:01 PM	#35
Marcus Durham Major Player Join Date: Jul 2009 Location: Hampshire, UK Posts: 693	Hard drive and blu-ray/DVD for the projects inc footage and separate blu-rays just for the footage as a separate backup. Works for me. The raw footage also goes onto another HD which is stored on a separate site. Make sure you spin the drives up from time to time though. The mistake some people make is to assume archiving is "forever". Storage technology is moving so fast that in 5 years time they'll be a better, larger more reliable and cheaper solution. So just work on the basis that you are only storing the data on the medium term because you'll be moving it anyway in the future. Media deterioration is an issue pretty much whatever digital format you use. I have premium Gold CD-R's from 10 years ago that won't play, but the cheap unbranded turquoise ones are just fine. Go figure! I would take all claims about media life with a pinch of salt. After all, if you look at most of the guarantees you'll find all you are entitled to is replacement media. Not much comfort if you've lost your only copy of a project. __________________ Marcus Durham Media2u, Corporate Video Production For Your Business - http://www.media2u.co.uk

November 8th, 2009, 02:09 PM	#37
Perrone Ford Inner Circle Join Date: Nov 2006 Location: Tallahassee, FL Posts: 4,100	"I don't use RAIDs - don't trust them." You do realize that mirrored drives = RAID 1? __________________ DVX100, PMW-EX1, Canon 550D, FigRig, Dell Octocore, Avid MC4/5, MB Looks, RedCineX, Matrox MX02 mini, GTech RAID, Edirol R-4, Senn. G2 Evo, Countryman, Moles and Lowels.

November 9th, 2009, 08:50 AM	#38
Brett Sherman Major Player Join Date: Mar 2006 Location: Mount Rainier, MD Posts: 428	I use the term "mirrored" loosely. It's not a real-time mirror, but rather a nightly synchronization. This eliminates the risk of a file system failure affecting both drives.

November 9th, 2009, 08:58 AM	#39
Perrone Ford Inner Circle Join Date: Nov 2006 Location: Tallahassee, FL Posts: 4,100	Ah ok. Gotcha. The terminology threw me. __________________ DVX100, PMW-EX1, Canon 550D, FigRig, Dell Octocore, Avid MC4/5, MB Looks, RedCineX, Matrox MX02 mini, GTech RAID, Edirol R-4, Senn. G2 Evo, Countryman, Moles and Lowels.