STORAGE / WORKFLOW Help

[Mitchell Lewis]

Okay I thought it might be fun to share our workflow. Maybe it will help someone. We produce television commercials and corporate videos so 99% of our footage/projects must never be lost as someone has paid us to produce them.

Here's what I'm calling THE LAZY BUT CAUTIOUS WORKFLOW. It's been working really well for us since January of this year. We haven't lost a single piece of footage. (knocks on wood)

INGEST
1) Connect the SxS card to my MacBookPro
2) Connect a Western Digital 500GB buss powered FW800 external hard drive to MacBookPro
3) Launch XD CAM Clip Browser
4) Using XD CAM Clip Browser create a new folder on the WD 500GB external hard drive
5) Using XD CAM Clip Browser drag all clips from the SxS window into the WD 500GB hard drive window
6) If there is a second SxS card, insert it, select all the clips and drag them into the same folder. This is important because any clips that might have spanned the two cards will be automatically joined together into one clip.
NOTE: This is my BPAV archive drive. Once it fills up, I put it on the shelf and purchase a new one.

EDIT
1) Move the WD 500GB hard drive to our MacPro tower
2) Launch XDCAM Transfer
3) Click the ADD button and navigate to the new BPAV folder on the WD 500GB hard drive
4) In XD CAM Transfer name all the clips (sometimes just selecting all of them and giving them all the same name)
5) In XD CAM Transfer sort through the clips and label the good clips with OK
6) Change the Status View so that only clips labeled OK are shown
7) Go to XDCAM Transfer>Preferences>Import Tab and choose a location to save your MOV files to. We save them to our fiber channel RAID.
8) Select all the clips and click the IMPORT button

BACKUP
1) We have a FW800 drive that we use to back up our RAID. We use Apple Time Machine because it's easy, and we don't forget do back up on a regular basis.
2) Our RAID is striped RAID-5 so if one of the drives suddenly goes bad, the RAID switches to RAID-0 mode and you can continue editing until you replace the bad drive.

ARCHIVE
1) When the RAID gets full, we use EMC Retrospect Backup to archive our project files (MOV videos, graphics, audio, everything except the BPAV folders) off to Western Digital 500GB portable FW800 hard drives. Retrospect automatically flows the data from one drive to another as the fill up. But the best part of using this software for archiving is when it comes time to restore. You simply launch the program, find the data or project you want to restore, select RESTORE and the program tells you want drive(s) to install. Once you install the drive it automatically starts the restore process.
2) As discussed in the INGEST description, we save all our BPAV folders on to separate portable hard drives. Worst case if we were to loose all our project files, we would at least still have the original footage. Also, we'll sometimes re-import footage from the BPAV files for use in other projects.

Let me know if anyone finds this useful. (or is pissed off that I took the time to post it) :)

[Doug Jensen]

Looks like a great workflow to me, and it very close to what I do. Thanks for taking the time. Newbies should print it and post it on the wall.

[Jason Leigh]

Wow, thanks guys....what great feedback. I will have to sit down and go through all that but to it will definitely help.

David, We have a production company in Fitzroy and we do a lot of corporate work. I'm dealing with terabytes of footage over a dozen projects and that's when it is becoming difficult to get the workflow right. Will sit down over the wekend and digest all the comments.

[Brett OBourke]

Thanks for this. Is helpful.

Am surprised to hear you're using Western Digital drives. We've had several fail and have collegues who have had WD drives fail and - while searching the interwebs for info on reliable drives that are cheaper than the G-Techs I'm presently using - I came across nothing but loathing for the WD drives. Like really vehement hatred the world over for WD. I was surprised and surprised to hear that you've had good luck with them.

B

[Mitchell Lewis]

Thanks!! I'm glad you find it useful. (I'm flattered Doug....means a lot coming from you)

I don't have any good data on why we chose Western Digital. We used to use Lacie, but they had some bad problems with their power supplies. After about 4-5 of our drives going bad, we switched to WD. Honestly, the main reason is that Best Buy sells them and there's a Best Buy 2 blocks from our office. I'm crossing my fingers that they will hold up.

We will be upgrading our RAID soon. We're going to switch from fiber channel to eSATA. We're going to go with a 8TB G-Tech G-SPEED ES drive for our RAID and then a Drobo Pro for backup. I really like the expandability of the Drobo Pro. Hopefully that will make it more future proof than the drives we've purchased in the past, as you can add/remove drives for more storage capacity. This is something I wish you could do with a high-speed RAID. Why Drobo didn't release the Drobo Pro with an eSATA connection is beyond me. That's another reason we're waiting a bit before we purchase a new high-speed storage solution.

[Sverker Hahn]

Originally Posted by Brett OBourke

Am surprised to hear you're using Western Digital drives.

I have bought 6 Iomega drives and all have failed (!). Now I have more than 8 WD drives, including 3 passport, no one have failed (yet).

[Alister Chapman]

I have a dozen WD drives and no problems so far. Can't say the same for Lacie.

[Olof Ekbergh]

I have had mixed luck with Lacie's (older ones are better it seems), some have lasted 8 years some failed within 2 years, my first Lacie was a 20MB hooked to a Mac Plus, 1989 I think. I have more than 2 dz of those and about 25% have failed. The worst drives I have ever used are Maxtors.

Lately I have switched mostly to Graid (I have 8) and WD (I have 17) . No problems with either of those brands some are 4+ years old now.

These are just for client storage. Our working drives are all Raid 5 or 0.
For clients who pay for this service, we have always kept all digitized files and projects on FW drives so we can quickly restore old projects and re edit. Clients love this. I also encourage them to buy 2, one for us to store and one for them to keep.

The new file based is an absolutely wonderful addition in our studio, makes life so much easier, we also back all files up on Bluray/DVD and now have a database of all clips.

[Steve Gibbons]

Originally Posted by Alister Chapman

I have a dozen WD drives and no problems so far. Can't say the same for Lacie.

Lacie uses Western Digital drives (and perhaps other brands) inside their enclosures - they don't actually manufacture hard disks, just the enclosures. The problem with the Lacie's are their power supplies - they are notorious for failing. But if you replace the power supply, the drive and the data on them are fine. This has been our experience.

[Mitchell Lewis]

Originally Posted by Steve Gibbons

Lacie uses Western Digital drives (and perhaps other brands) inside their enclosures - they don't actually manufacture hard disks, just the enclosures. The problem with the Lacie's are their power supplies - they are notorious for failing. But if you replace the power supply, the drive and the data on them are fine. This has been our experience.

That has been my experience with them as well. Never actually lost any data - thought we did with the first failure until we figured out it was the power supply. All the other failures have been power supply failures, but when they failed to replace the last one due to it being out of warranty (even though it was a documented known problem), we switched to Western Digital.

[Jason Leigh]

Hi Guys,

Well I've been over everything and all the advice has been fantastic....great post Mitchell. However, Its not really solving the issue of space. I thought maybe that the BPAV folder would end up smaller than the FCP Mov's but its exactly the same. I guess we're just going to have to look at a really large RAID server. On average I would say we are capturing 2 hours of footage a week so this tends to add up when we're backing up our final projects to include all the renders, project files and raw files. Copying all this to external drives just doesn't cut it....I have 4 WD 2TB drives sitting full on my floor already, as well as 8 TB of RAID storage which is at its max. I think the only long term solution will be a SATA storage server with 2TB RAID drives. I think using FCP's Media Manager (Thanks Doug) is also a good option to save some space on certain projects so will give that a go too. Thanks everyone for a great discussion!

Cheers

[Olof Ekbergh]

I actually find HD/Bluray/DVD storage to be much easier and cheaper than tape.

I know some people shoot on HDSD cards and just archive those, now that is easy and about the same cost as tape. Also storage space is a lot better than hundreds of feet of temperature controlled tape storage. We don't do this, but I would consider it if I was shooting an incredible # of hrs.

We probably average 3-5 hrs of shooting here a week and I find that dealing with file based media is way easier than tape.

[Mitchell Lewis]

Originally Posted by Jason Leigh

Hi Guys,

Well I've been over everything and all the advice has been fantastic....great post Mitchell. However, Its not really solving the issue of space. I thought maybe that the BPAV folder would end up smaller than the FCP Mov's but its exactly the same. I guess we're just going to have to look at a really large RAID server. On average I would say we are capturing 2 hours of footage a week so this tends to add up when we're backing up our final projects to include all the renders, project files and raw files. Copying all this to external drives just doesn't cut it....I have 4 WD 2TB drives sitting full on my floor already, as well as 8 TB of RAID storage which is at its max. I think the only long term solution will be a SATA storage server with 2TB RAID drives. I think using FCP's Media Manager (Thanks Doug) is also a good option to save some space on certain projects so will give that a go too. Thanks everyone for a great discussion!

Cheers

Thanks!

I'm reading between the lines of your post. It sounds like, like us, you're tired of buying an expensive high-speed RAID only to have to replace it after a year or two because it's full. The solution we're looking for is a RAID that would grow with you. Drobo almost has the solution with their Drobo Pro. When the drive fills up, just add more drives. When those fill up, replace some with bigger drives. But it has too big drawbacks....1) It's not fast enough. Why they don't release an eSATA version is beyond me. 2) It's limited to 16TB max. Actually I think this is a operating system limitation. But Apple's new Snow Leopard OS fixes that.

In the mean time we're looking at purchasing 2 x 8TB G-Tech G-SPEED es eSATA drives and a Drobo Pro for backup. It's a somewhat expensive solution, but still cheaper than a huge XSAN setup. We're still looking for other options over the next couple of months.

It's true that storage is really cheap these days, but high-speed storage is still expensive.

[Steve Gibbons]

What are people typically doing "in the field" with their EX footage?

Are you making backups to your notebook and a portable hard drive? Two backup hard drives or one?

And do you use the Sony Clip Browser to shot log while out or during travel/downtime as well?

Finally, do you bring lots of SxS cards or rotate through just a few by offloading to a hard drive?

[Olof Ekbergh]

In the field:

I have 2 16GB and an 8GB SxS, a PHU-60 (Sony HD that mounts on EX3 and has 60GB almost 4 hrs rec time), and 4 16GB HDSD in M&R adapter.

I usually only use the SxS's. And offload at night to 2 Portable HD's hooked to my MBP, I use trans app to rewrap to .mov on one HD, and copy BPAV to the other.

If I need more than the 2 1/2 hrs worth of SxS, I use the PHU-60 first (4hrs). I am still reluctant to completely trust my SD cards. But I have used them for B-roll that is not crucial.

I plan to get 2 more 32GB SxS, they are best you can over crank any time and they are the fastest to transfer from. I have never had a problem with any of the media, even the SD cards.

If I had a shoot with minimal equipment, I would just have a bunch of SxS.