is RAID 5 necessary do you think?

[Malcolm Hamilton]

Hi there,

re protecting myself from hard drive failure and losing an edit, my new improved workflow could be to:
1. backup my Sony EX-1 BPAV files to Blu-ray immediately after a shoot (as some of you have recommended. Too bad I recently bought a 32GB SXS card, which means the big, and expensive 50GB Blu-ray discs, but so be it... the price will surely come down)

2. export to mxf (I edit in Avid; just substitue whatever your NLE uses) and then keep this mxf backup drive for security (I'll put it on a shelf) until the project is finished, and then

3. import these mxf files into my NLE (on the hard drives in my e-SATA enclosure) and edit away.

Given this, do I need to RAID 5 the drives in my external enclosure, would you say? I guess I'm hoping to maybe just have four good "enterprise" grade drives in it (it's a 4-bay SATA enclosure) at RAID 0 (for speed only). If I do have a drive fail on me, it would seem to me that I can always go back to the mxf backup, and if this hard drive has also failed (passing meteor shower?), I can go back to the BPAV files on optical disc.

There are probably all sorts of issues I'm not aware of.
Any other comments, advice, suggestions?

thanks,

Malcolm

[Brooks Graham]

Hey Malcolm-

Your proposed workflow is dangerously close to what I do, so therefore I like it! ;-)

For the purposes of full disclosure, I'll admit that in a previous life, my employment (Veritas Software) and main area of focus revolved around the topics of storage management, storage-based workflow optimization, disaster recovery and data archival for large organizations (content producers as well as for orgs like NASA).

Originally Posted by Malcolm Hamilton

Hi there,

re protecting myself from hard drive failure and losing an edit, my new improved workflow could be to:
1. backup my Sony EX-1 BPAV files to Blu-ray immediately after a shoot (as some of you have recommended. Too bad I recently bought a 32GB SXS card, which means the big, and expensive 50GB Blu-ray discs, but so be it... the price will surely come down)

My storage workflow is broken up into two distinct areas: acquisition and online

For acquisition, I back up BPAV to spinning rust (hard drives) as soon as humanly possible, and then "archive" to BD-R as soon as convenient (but soon). I put the word "archive" in quotes because that word is so often abused. For this purpose, the BD-R copy of the BPAV structure allows me to go back to square one for at least five years. The spinning rust backup will remain for some moderate amount of time, but that space will eventually get re-used later. For me, it usually lasts until the project is over. (don't get me started on the horrible idea of archiving to flash media)

Initially, I used single-layer Taiyo Yuden DVD+R media (better cost per GB than BD-R), but since then, quality BD-R media has come down and although still more expensive per GB, the convenience factor overrides the increased cost. When I switched to BD-R, I thought I'd be clever and use one 16GB and one 8GB SxS card since those two sizes together fit nicely on one single-layer BD-R. In practice, however, it rarely worked out that well. I now use Toast's spanning feature which I've found to be safe.

2. export to mxf (I edit in Avid; just substitue whatever your NLE uses) and then keep this mxf backup drive for security (I'll put it on a shelf) until the project is finished, and then

3. import these mxf files into my NLE (on the hard drives in my e-SATA enclosure) and edit away.

Now going from acquisition to online. I also use a direct-attached RAID-0 SATA array for online storage. Using the spinning-rust backup as the source, I run XDCAM Transfer to re-wrap into QT .MOV files onto the online array. Since I've both backed up and archived the BPAV, I'm okay with the risk of losing the entire array should one drive fail. I use FCP and keep the project file as well as other project-related media on system drives that are regularly backup with Time Machine (for the duration of the project). The array is *only* used for the large media files in my projects and *only* if they exist somewhere else. The risk I run is the time for rebuilding the array after replacing a failed drive (short) and copying the data back (potentially long). For this to work, however, I must be disciplined in never placing anything on the array I can't afford to lose. (and cleaning it off when the project's over or stalled)

Once the project is completed, or as sometimes happens, stalls, I backup the entire project to a DroboPro 8-drive array and then eventually to BD-R as a self-contained project. (the Drobo is awesome for this purpose. not quite usable for online, but fantastic for near-line)

Given this, do I need to RAID 5 the drives in my external enclosure, would you say? I guess I'm hoping to maybe just have four good "enterprise" grade drives in it (it's a 4-bay SATA enclosure) at RAID 0 (for speed only). If I do have a drive fail on me, it would seem to me that I can always go back to the mxf backup, and if this hard drive has also failed (passing meteor shower?), I can go back to the BPAV files on optical disc.

There are a few factors here:

- Does your array do hardware RAID? or are you thinking of using host-CPU-based RAID?

If you must use RAID-5, only use hardware-based RAID-5. If/when I can cost-justify a new online array, it will probably be something like from CalDigit and be set to RAID 0+1 or 1+0.

- Performance. Can you live with the reduction in performance with RAID-5?

Even with hardware-based RAID-5, there's a significant write penalty.

If you have confidence in your backup/archive of the BPAV and can risk a few hours of re-copying if a drive fails, then I'd say your fine with RAID-0. If time is more important than cost, then look into CalDigit. I've been very impressed with their gear. (no relationship whatsoever with them)

There are probably all sorts of issues I'm not aware of.
Any other comments, advice, suggestions?

thanks,

Malcolm

[Craig Seeman]

Malcolm there is a "Split" function in ClipBrowser which can target 25GB Blu-ray. Basically one part would fit on Blu-ray and the other on Dual Layer DVD or split again for two DVDs.

[Perrone Ford]

Maybe I'm missing something, but why would anyone export to MXF for going into Avid? Or is this a pre-AMA version of Avid you're talking about?

[Malcolm Hamilton]

Brooks - - a wealth of information. Thanks so much. This will take me a while to go through, but I certainly will.

Just to clarify:

1. for actual editing, I'm hoping to use a direct-attached RAID-0 SATA array (in a 4-drive enclosure; only 4 drives because I never edit uncompressed so I don't need a huge number of Terabytes; also, I don't want too many projects to get their media files mixed up together). It'll be a Sans Digital enclosure connected to my Mac Pro with what I'm told is a very good Areca ARC 1880X card.
The Sans Digital unit details say: "Supports JBOD, spanning, RAID 0, 1, 1+0, 5, 6 with spare via RAID controller"
The Areca card specs say: "Areca high-performance ARC-1880 series PCIe 2.0 to 6Gb/s SAS RAID adapter can provide up to (128) 6Gb/s and 3Gb/s SAS/SATA/SSD peripheral devices using 6Gb/s SAS expanders. The adapters are based on the same RAID kernel of field-proven external RAID adapter and same device drivers with widely used SAS/SATA RAID adapters"
(I'm spelling this out because most of it is Greek to me. Does it all sound good? I think if I go RAID-0, it's considered hardware RAID, isn't it?)

I thought about CalDigit but they're so darn expensive, and in many of their models it seems hard to take out and put in new drives. Also, a very knowledgeable guy on another forum said that the drives in the Caldigit HDElement are Hitachi consumer-models. CalDigit looks great, and may be, but it is expensive.
2. for backing up files, and keeping finished projects, etc., I'll use my existing (non-RAID) 5-bay FirmTek e-SATA enclosure with 5 drives. I should check out this DroboPro unit. I've heard people speak highly of Drobo before.

(I'm hoping to have such a good backup system in place that I won't feel the need to go RAID-5 or 6)

2. for backup and storage, I was planning to use my existing FirmTek 5-bay eSATA unit, non-RAIDED. Some day I'll check out this Drobo unit you mention. Does it somehow make it easier to back up and stay organized?
I will also have some storage drives in my Mac Pro.

Craig - - this is brilliant! This will save me a lot of money. Thanks so much

Perrone - - very good question. And yes, I'm using the latest Media Composer, which can import this stuff via AMA. What can I say?!! I'm used to doing things a certain way (when I go through the somewhat time-consuming step of importing into Avid, I'm somehow reassured by the time it takes, and the clicking and whirring of my hard drives); the one time I used AMA it all appeared so quickly that I wasn't sure if the media would still be in Avid the next time I turned my computer on. Truly, I'm a bit of a Luddite, who uses all this modern technology with my fingers crossed and my lucky rabbit's foot in my pocket. I guess it's just that I don't understand AMA. Are you using it?

Thanks everyone... Malcolm

[Perrone Ford]

Originally Posted by Malcolm Hamilton

Perrone - - very good question. And yes, I'm using the latest Media Composer, which can import this stuff via AMA. What can I say?!! I'm used to doing things a certain way (when I go through the somewhat time-consuming step of importing into Avid, I'm somehow reassured by the time it takes, and the clicking and whirring of my hard drives); the one time I used AMA it all appeared so quickly that I wasn't sure if the media would still be in Avid the next time I turned my computer on. Truly, I'm a bit of a Luddite, who uses all this modern technology with my fingers crossed and my lucky rabbit's foot in my pocket. I guess it's just that I don't understand AMA. Are you using it?

HECK ya I'm using it! Are you kidding? For XDCAM footage it can't be beat by any other workflow. Once you have the native files stored on your drives, using AMA is the best workflow there is. By going to MXF files from the XDCam tool, you are tossing away much of your intrinsic metadata. AMA preserves all of that. If you're going to do a conversion to MXF, do it inside Avid with an import, so Avid can retain all that metadata.

But I can be FINISHED with an edit using AMA before I could START an edit the traditional way. AMA was a primary reason for my purchase of Avid in the first place. There are some pitfalls to using it, like not being able to export a reference .mov to Squeeze, but that's a minor annoyance on most quick edits. I still do a traditional import for big projects (like movies), but for quick commercials and corporate work, it's AMA all the way.

[Malcolm Hamilton]

Originally Posted by Perrone Ford

I still do a traditional import for big projects (like movies), but for quick commercials and corporate work, it's AMA all the way.

O.K., I will take my head out of the sand and look at AMA seriously. But your last sentence gave me pause. I don't do quick commercials. Mostly corporate work, and the odd doc -- but for me, these are big projects.
Why don't you use AMA for your big projects?
Thanks for this,
Malcolm

[Mohammad Farooqi]

Just wanted to post my procedure.
While acquiring the footage I am simultaneously recording on the sxs cards and Flash XDR. Just in-case one decides to act up. MY XDR can hold about 4-5 hours of 35 Mbps footage, probably more,this is off the top of my head, the sxs cards hold far less. I have not invested in much sxs cards since I have a XDR.

So while I am shooting, I am also utilizing a laptop with an express card slot to copy over the files to two different hard drives. Different means also the manufacturer. So I am constantly swapping sxs cards and backing them up. At the end of the day I also copy the flash XDR files to the same hard drives and just in-case, the laptop hard drive as well.

For storage. I archive the footage, final render and project files to a Drobo which is basically similar to a Raid 5 config. but not as annoying to maintain and not as expensive. I also have another copy of these files on a regular external hard drive or a blu ray disc depending how large the file size is. And that is about it. Let me know your comments and suggestions.
Thanks

[Perrone Ford]

Originally Posted by Malcolm Hamilton

O.K., I will take my head out of the sand and look at AMA seriously. But your last sentence gave me pause. I don't do quick commercials. Mostly corporate work, and the odd doc -- but for me, these are big projects.
Why don't you use AMA for your big projects?
Thanks for this,
Malcolm

I have more options available to me, especially down the line, when working with DNxHD footage. Especially in exporting reference .mov files. I also find the timeline more responsive when working with multi-camera, multi-effects, etc.

Take a project you've done via importing, and try it with AMA and see how it responds for you. You might just save yourself a lot of time. Or maybe not. But at least with Avid and AMA, you have the CHOICE.

[Andy Taplin]

My ideal workflow:

Have enough SxS cards for a full days shoot - no messing about transferring data when shooting.

Copy SxS cards to a NEXTO NVS2500 and a USB drive whilst packing up after shoot. Give client the USB.

Copy from the Nexto to an internal 3 disc editing RAID 0 back at base.

Back up the RAID 0 to a Drobo RAID 5 asap for safety.

If the RAID 0 goes down I can carry on working with no downtime using the DROBO.

[Tim Polster]

Malcolm, in reading your post I do not know why you would necessarily need a RAID setup for your editing drives. Since you will already have multiple backups the edit drive(s) only need to hold the material for editing and be fast enough.

In my experience RAID is only needed for uncompressed editing. If you are using compressed footage (Iike XDCAM) any modern single drive has plenty of throughput for any editing to be done. If you have a multicam shoot I would put a camera per drive just to make everything work less hard though.

To illustrate this point, in my editor (Edius) I can playback 100mbps 1080p Nanoflash footage from the CF card in a USB reader with color correction and sharpness filters applied.

XDCAM is at 35mbps which requires 1/3 the throughput from the hard drive than 100mbps.

RAID may solve some issues like redundancy but most of the time is not needed from a speed point of view.

[Malcolm Hamilton]

Hi Tim,
Your timing is incredible. I've been putting off the actual purchase of a unit (I'm currently on a shoot and was waiting until I got back), and if I don't need to go RAID, I won't. It will save me money and probably some grief.
The fact is, I don't edit uncompressed, and I only notice a few hesitations (this is in Avid Media Composer) when I've got multiple video tracks, with effects. And that's not really been a huge problem. And even this problem might be solved, now that I'm switching from a MacBook Pro to a Mac Pro.
So I think I might just stick with what I've got.
Thanks for the advice. Actually, I'm thankful to everyone for their advice... (and when I do eventually go RAID, which probably we'll all do some day), I'll know what to do.
Regards,
Malcolm

[Vincent Rozenberg]

To whom it may concern here is our workflow, which we use with 6 to 8 directors at this time in our company:
-Every director has 8 cards, and is responsible for them.
-We are mac based, every mac here at the office can be the ingest station and is connected to a server.
-Attached to the server there's a Drobo Pro with lots of terrabytes. Every director has it's own folder and puts their cards on it after a shoot.
-The Drobo is redundant but to be sure it backs up every night to a separate RAID system.
-The edit suites grabs the files for edit, so they have a local copy, just on the internal drives, no need for redundancy because there's always a copy on the Drobopro and the RAID backup.
-And just to be sure, we upload constantly to a remote backup server out of the office, in case of an emergency (fire, burglary).

[Tim Polster]

Sure. Hope it helped.