Percentage of raid space?

[Betsy Moore]

Hey guys! I'm just finishing the production end of a long-in-the making feature length project I shot with HD Pro Res files. Money got really tight so all I could do during shooting was save the footage to rather inexpensive hard drives. Now I'm ready to just barely afford to push the button on a new Mac Pro (to replace my limping 2008 tower) and transfer all the footage onto a robust raid for editing. There's about 40TB of footage (yeah, I know) and I'm looking at buying 2 OWC ThunderBay IV 20.0TB High-Performance 4-Bay drives for the process as these seem to be the least expensive high speed options.

Problem is of course, the more footage you put on the hard drives the slower they go. My project involves a lot of masks, glow filters, color grading, speeding up/slowing down, and I'd like to be able to render my decisions in as close to real time as possible.

So my question is (finally!) what percentage of a 20TB raid array should I fill up before it slows down the rendering process unduly? Is there a happy medium? Am I looking at having to buy 3 20tb raids to keep the project nice and fast?

I know there are a few 32TB drives out there but they're more expensive per TB than the OWC.

If any of you have any advice I appreciate it:)

[William Hohauser]

You really don't need 4 bay RAID arrays for HD ProRes unless you captured in PRoRes444. Even then you could get away with 2 bay RAIDs if you are not editing Multi-cam. This way you can purchase more economical 2 drive RAIDs. If you are thinking of RAID5 for safety then you should think of purchasing more than three 4-bay arrays.

The renders will end up on the drive the FCPX library is put so figure that in as you set up the system for the edit. You may find that rendering isn't needed to view many effects, the new MacPro is that good.

1) leave the footage on their own drives when importing into the FCPX library (there is the option to leave the footage where it is in FCPX) and have the library on what ever drive has at least 200Gbs free of space.

2) devote a drive to the FCPX library and import and transfer the footage as needed thru FCPX. This will bring the footage you are using into the library. A finished feature is about 125Gbs in ProResHQ so a terabyte RAID drive is probably enough if you have pre-chosen the good takes outside of FCPX.

And back everything up on cheap USB3 drives.

[David W. Jones]

Be aware that you can only connect 6 Thunderbolt peripherals, and each of those OWC ThunderBay IV RAIDS counts as 4.

[William Hohauser]

That's very good info to pass along. I would make a suggestion that using a LaCie Thunderbolt to eSATA adaptor might circumvent that limitation as it gives two eSATA ports per Thunderbolt port. And eSATA RAIDS are cheaper than Thunderbolt RAIDs. At least they were late last year when I was buying.

[David W. Jones]

Or purchase a Thunderbolt raid with a built in multiplexer.

[Betsy Moore]

Originally Posted by David W. Jones

Be aware that you can only connect 6 Thunderbolt peripherals, and each of those OWC ThunderBay IV RAIDS counts as 4.

Whoa... Good to know David!

[Betsy Moore]

Originally Posted by William Hohauser

That's very good info to pass along. I would make a suggestion that using a LaCie Thunderbolt to eSATA adaptor might circumvent that limitation as it gives two eSATA ports per Thunderbolt port. And eSATA RAIDS are cheaper than Thunderbolt RAIDs. At least they were late last year when I was buying.

That is tempting, William. But do you think eSATA would be fast enough to handle multiple layer masks, glow filters, speed alteration, in real time? Even if it is only ProRes HQ 422 in 1080p?

[Betsy Moore]

Originally Posted by William Hohauser

You really don't need 4 bay RAID arrays for HD ProRes unless you captured in PRoRes444. Even then you could get away with 2 bay RAIDs if you are not editing Multi-cam. This way you can purchase more economical 2 drive RAIDs. If you are thinking of RAID5 for safety then you should think of purchasing more than three 4-bay arrays..

Wow, William:) Do you think I could get away with a 4 core then? I was going to go 6 core and maybe trade out when I do my next project presumably in 4k but do you think even a 4k MacPro with a great GPU could handle lots of real time effects in a HD ProRes project?

[William Hohauser]

Yes, most of what you are describing (if I am understanding it correctly) is more related to the video boards in the computer than the hard drives or even the CPU. Personally I would go for as many cores as your budget allows. That will help rendering later on or even while you are doing the project. At least six cores.

One thing to be aware of is that most, if not all, of the filters provided by Apple in FCPX or made by third parties with Motion are optimized for the dual video boards in the MacPro. This means that you can do a lot of real-time effects while you are editing. Some third party effects/filters may not be optimized and will cause slow-downs if not rendered. I have a couple of third party filters that with the old MacPro I would apply, set up and then turn off while editing. There was no way to do real-time editing while they were activated. With the new MacPro, the filters cause very brief slow-downs or make transitions skip while playing real-time but I don't turn them off anymore.

That said, if you are really going to pile the filters on, you might want to render frequently and more cores will help you render faster. Do some research and find out how some people manage heavy filter use in their FCPX projects. There are tricks especially if you plan to put the same filter with the same settings across a continuous range of clips.

[Betsy Moore]

I'm embarrassed to admit this but I've re-read this post several times in the last few weeks and scoured the internet and read old forum arguments on different sites and I'm still confused about a basic question, which is how would you link 50 TBs of footage to the new Mac Pro in a way that all the data is accessible at all times and in some sort of configuration that's fast and stable? If it was you and you didn't have an unlimited budget but the project had to get done, how many hard drives of what size would you put into how many ports?

I feel like the above answers probably should have been enough for me to figure it out, which is why I hessitated posting this for so long. :)

[William Hohauser]

50tb is a lot of footage to have accessible at all times but not impossible. Three 20TB Thunderbolt RAIDs from Other World Computing will be slightly over $5000. OWC ThunderBay IV - Thunderbolt External Hard Drive & SSD Enclosure

Is that in your budget?

[Betsy Moore]

I think I can swing that, especially if I buy them one at a time:) Should I daisy chain them or plug them directly into the Thunderbolt ports? I currently have 3 non-4k monitors: two 30"s and one 24"

[William Hohauser]

The manufacturer says they are chain-able, so why not?

[Betsy Moore]

If it doesn't slow them down why not in deed? :) thanks again!! Awaiting delivery of my brand new six core dual 700 gpu beauty!

[Bill Davis]

Betsy,

Excuse me for coming to the party late, but I'm not sure you're thinking like a FCP-X editor.

I don't care HOW efficient your field shooters were, the VAST majority of the 40TB of footage is going to be useless dreck.

The whole point of having a functioning database bolted into your editorial system is to allow yourself to push the useless stuff away from your consciousness and just concentrate on what's potentially useful.

I'd be ASTONISHED if you had more than 1/3 of your raw footage that's going to be truly useful after you go through things and simply REJECT all the pre-scenes, post roll, completely blown takes, and on-set discussions where the camera was still rolling but nothing useful was happening. I'd actually bet it's more like 20% useful. So it just makes NO sense to me to clog up your workflow as you're suggesting.

You already have 40 TB of backup drives. Link them to X, do a REJECT pass and keyword the remainder by scene of weaver you like. When you get down to the stuff you'll ACTUALLY need, just move THAT onto your raid using the Consolidate command. I bet it's something like 5-8 TB Max.

Use the tools you have to make your editing life easier. That's what X is so superb at.

Just some suggestions from a guy who's learned this stuff the hard way.

Good luck.