best RAIDed hard-drive set-up for EX-1?

Malcolm Hamilton · May 14th, 2008, 06:17 AM

Hi Jonathan,
I asked a question along these lines on an Avid editing forum, and I kind of inferred from the response that it might not be a good idea ("that's crazy. Use one of those HD codecs to bring your file sizes and data rates down while you're editing, then bump up your picture-locked cut to the best resolution your drives/system can handle")
Let me know what you think.
Malcolm

Sean Donnelly · May 14th, 2008, 06:35 AM

The price is definitely higher for a system like caldigit, dulce, ciprico, etc. but there is a huge difference. The only question is if you really need it. If you don't ever plan to capture uncompressed 422 or 444, or work with 2k files, then a software raid or an eSata raid card is all you really need, and in fact a single eSATA drive can keep up with XDCAM easily in my experience. If you want to capture uncompressed 422 via HD-SDI, you will need to sustain 200MB/s, and you'll need something better than eSATA to sustain that, along with better drives. Many large RAID arrays will use 15k rpm SAS drives which in addition to being much faster have a secondary controller board, in case one fails. One of the many reasons they are 4-5 times the price of a 7200 rpm SATA drive.

-Sean

Malcolm Hamilton · May 14th, 2008, 06:56 AM

Quote:

Originally Posted by Sean Donnelly

If you don't ever plan to capture uncompressed 422 or 444, or work with 2k files, then a software raid or an eSata raid card is all you really need
-Sean

Thanks, Sean, this is exactly what I needed to hear (and perhaps what people have been trying to tell me). Now the only thing I need to know is... why do people edit uncompressed? And is there any likelihood that I'll ever need to?

I'm just getting going in this business - - just doing half-hour long projects, mostly for DVD... so it would seem to me it's very unlikely I'd kick myself, five months from now, for not being able to edit uncompressed. It's very unlikely that National Geographic is going to come calling. In other words, and please stop me if I'm jumping to conclusions, even though I can now shoot very high quality footage, I will probably never need to, or want to, edit it uncompressed.
If that's the case, it resolves my conundrum about eSATA editing. I'll go with Firmtek or G-Tech or Sonnet. Then I just have to figure out how best to RAID it.
Cheers, Malcolm

Sean Donnelly · May 14th, 2008, 09:29 AM

Quote:

Originally Posted by Malcolm Hamilton

...even though I can now shoot very high quality footage, I will probably never need to, or want to, edit it uncompressed.
If that's the case, it resolves my conundrum about eSATA editing. I'll go with Firmtek or G-Tech or Sonnet. Then I just have to figure out how best to RAID it.
Cheers, Malcolm

Uncompressed workflows are more important when you're doing an online with film scans or material that was actually shot in an uncompressed format (or close to it). Compression is something to be used wisely, and makes shooting high def practical in most cases. You're not going to decompress the signal by shooting in XDCAM and cutting uncompressed, but if you are doing effects, etc. it's worth transcoding to something like prores 422 (roughly 50 MB/s). My recommendation would be something in a 4 disk RAID 5 over eSATA for your purposes right now.

-Sean

Jonathan Bland · May 14th, 2008, 09:30 AM

"even though I can now shoot very high quality footage, I will probably never need to, or want to, edit it uncompressed.
If that's the case, it resolves my conundrum about eSATA editing. I'll go with Firmtek or G-Tech or Sonnet. Then I just have to figure out how best to RAID it."

Same same Malcolm.

My feeling is that almost nobody edits uncompressed except the "big guys".

So now.....

What is the process?

1) Ingest SXS cards?

2) Back them up to another drive in the enclosure?

3) Render native files to something like Apple Pro Res?

4) Throw the files on the Final Cut time line and start cutting?

Almost there Malcolm :) Which enclosure and drives are you leaning towards?

Jonathan Bland · May 14th, 2008, 09:37 AM

Right-on Sean :)

"you're not going to decompress the signal by shooting in XDCAM and cutting uncompressed, but if you are doing effects, etc. it's worth transcoding to something like prores 422 (roughly 50 MB/s)."

What???

A simple question here:

Can I ingest and start cutting the EX1 footage AS IT IS from the cards on my new Macbook and SATA drives with out compressing it with ProRes first.

Any thoughts on the best 5TB enclosure and drives to go with?

Malcolm Hamilton · May 14th, 2008, 11:00 AM

Hi again...
I thought I'd share a bit of new info with Sean (and it very much pertains to you Jonathan)... a technical help guy at one of the eSATA hd places wrote back to me this morning, on the subject of RAID. He says:

"I find that using 4-bays as a striped RAID set (RAID 0) and the 5th bay for off-line backup disks works well.
You should know that RAID 5 may be recoverable in the case of a hard disk failure but it cannot protect you from directory errors or a user error. An off-line backup is required for any important data NO matter what RAID type you use. In addition, rebuilding a RAID 5 with large hard drives can take a day or more. Sometimes recovering with a new RAID 0 is actually
faster with a backup than waiting for a RAID 5 to rebuild"

What do you think? I guess with this RAID 0 set-up, if you back up to theat fifth drive once a day, the most you can lose is one day's work, right? Or however long it takes, if you lose all your media, to... what, re-copy it from the back-up drive that I guess all EX-1 users have to make when they come back from a shoot (I'm trying to figure out a workflow too, J.), and re-connect it to our timeline.

Is this correct?
thanks,
Malcolm
Malcolm

Sean Donnelly · May 14th, 2008, 09:19 PM

Now your 5th bay has to be 4 times the size of the others to be able to back up a full array. RAID 5 isn't a foolproof backup system, but it's great to work from. Can I ask which place this was?

-Sean

Jonathan Bland · May 14th, 2008, 10:35 PM

Sounds like you are on a good roll Malcolm. Hang in there....
Your "thinking out loud approach is most helpful and thanks for including me on this :)

I'm not sure.... but now I'm thinking.... for what I'll be doing (international doc.... gotta keep it small and compact) I be able to get away with a smaller enclosure than the 5TB.

1) Come back from a shoot and offload the cards to a drive....
2) Plug in a second drive and back up the 1st drive....
3) Take out the second drive and keep it in an thick plastic pouch with silica packet (for humidity control).
5) Eat a mango.... and smile
4) Start cutting.

I'll try to work through this now for the first time:

This is for a 2 bay enclosure....

Pop 2 fresh drives into both bays. Start offloading the cards to both drives. Eject drive 2 and store it.
Start cutting on drive one.

Keep shooting, offloading, backing up, cutting until you fill the first drive and it's backup then....

Start over with 2 new drives.

I now see that one runs into a snag when you need to be cutting from more than 2 drives (ie.2 TB) at the same time. I see a bigger enclosure would be necessary. I wonder if I could get away with a 3 or 4 bay?

All of this is coming from the idea that I would not need a stripped raid because I would be using Apple ProRes to compress the native EX1 files into something that is possible to cut with.... Something I think I understand most of the world does to deal with these massive file sizes. (Sorry Malcolm I know your not using Mac.)

Nobody has answered my question of whether or not it's even possible to edit native EX1 footage on a new Macbook Pro or if one would even want to attempt this? Maybe it is possible with the striped raid array. (Still awaiting delivery on my new one so I don't know yet).

It's tough to be a one man band and it sure would be cool if a edit pro could chime in here and nail all this down in a concise well-worded/ constructed way instead of all the hit and miss.

Any thoughts?

Daniel Weber · May 14th, 2008, 10:56 PM

Quote:

Originally Posted by Jonathan Bland

Nobody has answered my question of whether or not it's even possible to edit native EX1 footage on a new Macbook Pro or if one would even want to attempt this? Maybe it is possible with the striped raid array. (Still awaiting delivery on my new one so I don't know yet).

Jonathan,

I have been editing footage from my EX1 on my Macbook Pro. I bought both of them in December. I use Final Cut Pro. I don't know if the footage is considered "native" or not since Final Cut needs to put a quicktime wrapper on the MXF files to edit them. I think that you saw my Yosemite piece. That was edited on my Macbook Pro in about 45 minutes while sitting in a Starbucks. You can edit on a Macbook Pro. It works great if you have portable FW 800 drives.

Daniel Weber

Jonathan Bland · May 14th, 2008, 11:03 PM

Right on Daniel.

So why would I want SATA drives? In case a drive fails during editing? To get faster speeds/ throughput if I was doing multiple timelines?

By native footage I mean full res 1920 x 1280 35Mbs

Do you compress your footage in Prores so you can edit it???

Thanks for your help on this :)

Daniel Weber · May 14th, 2008, 11:28 PM

then you can edit native!!!

It works well.

You might want to look at a small bus powered firewire unit that uses FW 800 and has 2 drives raided. I think the G-Tech makes one of these. I think you can also "roll your own" if you look online for enclosures. Make sure that you get ones that use the notebook size SATA drives.

Daniel Weber

Daniel Weber · May 14th, 2008, 11:30 PM

Quote:

Originally Posted by Jonathan Bland

Do you compress your footage in Prores so you can edit it???

Nope. I edited it native. The ProRes codec takes up more bandwidth on the drives than native XDCAM HD files.

Daniel Weber

Jonathan Bland · May 14th, 2008, 11:38 PM

"Nope. I edited it native. The ProRes codec takes up more bandwidth on the drives than native XDCAM HD files."

More bandwidth?

You mean more space?

What are we talking about?

Prores requires greater bus/ throughput speeds?

Man this could take forever if we don't use the right terms. Myself included ;)

Barry J. Anwender · May 14th, 2008, 11:52 PM

Quote:

Originally Posted by Jonathan Bland

Nobody has answered my question of whether or not it's even possible to edit native EX1 footage on a new Macbook Pro or if one would even want to attempt this? Maybe it is possible with the striped raid array. (Still awaiting delivery on my new one so I don't know yet).

It's tough to be a one man band and it sure would be cool if a edit pro could chime in here and nail all this down in a concise well-worded/ constructed way instead of all the hit and miss.

Any thoughts?

Jonathan, I'm not sure that I clearly understand your configuration or workflow, but I'll try to explain from my experience using a MacPro with FCP 6.0.3 and my RAID which I setup differently depending on the type video format that I edit with.

First FCP principle. Do not use your boot drive to store and edit video clips/footage. Not on a MacPro and definitely not on a MacBook Pro even if it has a 7200rpm drive. As I mentioned above, SATA drives slow down as they fill up. A SATA drive transfer rate is cut in half when near full. The boot drive needs to run the OS (i.e. Leopard) and the FCP application. Rendering sequences also plays in this, if you setup FCP to store its cache files on the boot drive, which is OK to do, but does add more load to the boot drive.

Second, based upon my experience with FCP ProRes(HQ) video sequences. Two drives stripped (i.e. RAID-0) together are fast enough to edit at least three streams without dropped frames. I usually edit in FCP with timeline viewing set to full quality i.e. RT editing disabled. So for your setup, use a two port e-SATA card with your MacBook Pro and strip two drives together. I'm confident it will work fine for you with ProRes(HQ) sequences.

Third principle -- if all else fails read the manual. Apple has designed FCP with RT editing capabilities, precisely so that you can edit using a laptop. Dig into the FCP manual in the sections on setting up your system for RT editing. Alternately, find a Lynda.com or ApplePro training series that covers FCP setup and RT optimization.

Fourth, I personally learn much better by using video based instruction and so I rely on ApplePro Training Series, Lynda.com, Digital Film Tree, et.el. to get up to speed and to address specific editing needs that I run into. I have one drive dedicated to video based training. I can quickly and easily use these video tutorials to find answers or to advance up the learning curve for FCP, Color, Motion, Soundtrack, DVDSP etc.. Each project seems to require something unique that demands more learning on my part. As you said, it's tough being a one band man.

This is what works for me, with eight years of using FCP studio. I'm still learning or a very slow learner. Some days, I also suffer with "sometimers" :-)) Sorry, if I have missed the point of your question or your particular needs. Just trying to help out and that is the treasure of these forums, to draw upon the collective experience, so that we can learn from each other.

sincerely, Barry

May 14th, 2008, 09:37 AM	#21
Jonathan Bland Major Player Join Date: Apr 2008 Location: Way Far Away Posts: 230	Right-on Sean :) "you're not going to decompress the signal by shooting in XDCAM and cutting uncompressed, but if you are doing effects, etc. it's worth transcoding to something like prores 422 (roughly 50 MB/s)." What??? A simple question here: Can I ingest and start cutting the EX1 footage AS IT IS from the cards on my new Macbook and SATA drives with out compressing it with ProRes first. Any thoughts on the best 5TB enclosure and drives to go with? Last edited by Jonathan Bland; May 14th, 2008 at 10:40 PM.

May 14th, 2008, 06:17 AM	#16
Malcolm Hamilton Major Player Join Date: Jan 2007 Location: Ottawa, Canada Posts: 844	Hi Jonathan, I asked a question along these lines on an Avid editing forum, and I kind of inferred from the response that it might not be a good idea ("that's crazy. Use one of those HD codecs to bring your file sizes and data rates down while you're editing, then bump up your picture-locked cut to the best resolution your drives/system can handle") Let me know what you think. Malcolm

May 14th, 2008, 06:35 AM	#17
Sean Donnelly Regular Crew Join Date: Jan 2008 Location: New York, NY Posts: 188	The price is definitely higher for a system like caldigit, dulce, ciprico, etc. but there is a huge difference. The only question is if you really need it. If you don't ever plan to capture uncompressed 422 or 444, or work with 2k files, then a software raid or an eSata raid card is all you really need, and in fact a single eSATA drive can keep up with XDCAM easily in my experience. If you want to capture uncompressed 422 via HD-SDI, you will need to sustain 200MB/s, and you'll need something better than eSATA to sustain that, along with better drives. Many large RAID arrays will use 15k rpm SAS drives which in addition to being much faster have a secondary controller board, in case one fails. One of the many reasons they are 4-5 times the price of a 7200 rpm SATA drive. -Sean

May 14th, 2008, 09:30 AM	#20
Jonathan Bland Major Player Join Date: Apr 2008 Location: Way Far Away Posts: 230	"even though I can now shoot very high quality footage, I will probably never need to, or want to, edit it uncompressed. If that's the case, it resolves my conundrum about eSATA editing. I'll go with Firmtek or G-Tech or Sonnet. Then I just have to figure out how best to RAID it." Same same Malcolm. My feeling is that almost nobody edits uncompressed except the "big guys". So now..... What is the process? 1) Ingest SXS cards? 2) Back them up to another drive in the enclosure? 3) Render native files to something like Apple Pro Res? 4) Throw the files on the Final Cut time line and start cutting? Almost there Malcolm :) Which enclosure and drives are you leaning towards?

May 14th, 2008, 11:00 AM	#22
Malcolm Hamilton Major Player Join Date: Jan 2007 Location: Ottawa, Canada Posts: 844	Hi again... I thought I'd share a bit of new info with Sean (and it very much pertains to you Jonathan)... a technical help guy at one of the eSATA hd places wrote back to me this morning, on the subject of RAID. He says: "I find that using 4-bays as a striped RAID set (RAID 0) and the 5th bay for off-line backup disks works well. You should know that RAID 5 may be recoverable in the case of a hard disk failure but it cannot protect you from directory errors or a user error. An off-line backup is required for any important data NO matter what RAID type you use. In addition, rebuilding a RAID 5 with large hard drives can take a day or more. Sometimes recovering with a new RAID 0 is actually faster with a backup than waiting for a RAID 5 to rebuild" What do you think? I guess with this RAID 0 set-up, if you back up to theat fifth drive once a day, the most you can lose is one day's work, right? Or however long it takes, if you lose all your media, to... what, re-copy it from the back-up drive that I guess all EX-1 users have to make when they come back from a shoot (I'm trying to figure out a workflow too, J.), and re-connect it to our timeline. Is this correct? thanks, Malcolm Malcolm

May 14th, 2008, 09:19 PM	#23
Sean Donnelly Regular Crew Join Date: Jan 2008 Location: New York, NY Posts: 188	Now your 5th bay has to be 4 times the size of the others to be able to back up a full array. RAID 5 isn't a foolproof backup system, but it's great to work from. Can I ask which place this was? -Sean

May 14th, 2008, 10:35 PM	#24
Jonathan Bland Major Player Join Date: Apr 2008 Location: Way Far Away Posts: 230	Sounds like you are on a good roll Malcolm. Hang in there.... Your "thinking out loud approach is most helpful and thanks for including me on this :) I'm not sure.... but now I'm thinking.... for what I'll be doing (international doc.... gotta keep it small and compact) I be able to get away with a smaller enclosure than the 5TB. 1) Come back from a shoot and offload the cards to a drive.... 2) Plug in a second drive and back up the 1st drive.... 3) Take out the second drive and keep it in an thick plastic pouch with silica packet (for humidity control). 5) Eat a mango.... and smile 4) Start cutting. I'll try to work through this now for the first time: This is for a 2 bay enclosure.... Pop 2 fresh drives into both bays. Start offloading the cards to both drives. Eject drive 2 and store it. Start cutting on drive one. Keep shooting, offloading, backing up, cutting until you fill the first drive and it's backup then.... Start over with 2 new drives. I now see that one runs into a snag when you need to be cutting from more than 2 drives (ie.2 TB) at the same time. I see a bigger enclosure would be necessary. I wonder if I could get away with a 3 or 4 bay? All of this is coming from the idea that I would not need a stripped raid because I would be using Apple ProRes to compress the native EX1 files into something that is possible to cut with.... Something I think I understand most of the world does to deal with these massive file sizes. (Sorry Malcolm I know your not using Mac.) Nobody has answered my question of whether or not it's even possible to edit native EX1 footage on a new Macbook Pro or if one would even want to attempt this? Maybe it is possible with the striped raid array. (Still awaiting delivery on my new one so I don't know yet). It's tough to be a one man band and it sure would be cool if a edit pro could chime in here and nail all this down in a concise well-worded/ constructed way instead of all the hit and miss. Any thoughts?

May 14th, 2008, 11:03 PM	#26
Jonathan Bland Major Player Join Date: Apr 2008 Location: Way Far Away Posts: 230	Right on Daniel. So why would I want SATA drives? In case a drive fails during editing? To get faster speeds/ throughput if I was doing multiple timelines? By native footage I mean full res 1920 x 1280 35Mbs Do you compress your footage in Prores so you can edit it??? Thanks for your help on this :)

May 14th, 2008, 11:28 PM	#27
Daniel Weber Major Player Join Date: Mar 2005 Location: Maryland, USA Posts: 696	then you can edit native!!! It works well. You might want to look at a small bus powered firewire unit that uses FW 800 and has 2 drives raided. I think the G-Tech makes one of these. I think you can also "roll your own" if you look online for enclosures. Make sure that you get ones that use the notebook size SATA drives. Daniel Weber

May 14th, 2008, 11:38 PM	#29
Jonathan Bland Major Player Join Date: Apr 2008 Location: Way Far Away Posts: 230	"Nope. I edited it native. The ProRes codec takes up more bandwidth on the drives than native XDCAM HD files." More bandwidth? You mean more space? What are we talking about? Prores requires greater bus/ throughput speeds? Man this could take forever if we don't use the right terms. Myself included ;)