Hard drives for Mac Pro and external storage

[Lizi Hesling]

Hi guys,
Please do redirect me if any of these questions have been answered elsewhere!
Basically, I've come up against a major storage bottleneck.
First, perhaps I should say a little about my set-up. I've got a Sony Z1 (and a lot of HDV footage on tape), and a very new Sony EX1. For editing, I have a Mac Pro and a 15 inch Macbook Pro. For external storage, I have very little - one very old 500 GB La Cie Brick and one new 500 GB La Cie Rugged (which I use only for current projects, moving between computers). For in-the-field storage for the EX1, I have a Nexto 500 GB, plus another back-up hard drive that I've had problems with - but I don't want to go into that here....
I do mainly short documentary-like films for the web. One project usually won't have more than 20 hours of footage.
Anyway....
At the moment, what I have in my Mac Pro (apart from the hard drive holding my OS files) is a 1TB Hitachi Deskstar, which it came with, and a 1.5T Seagate. The Seagate is an AS - and noticeably slower than the Hitachi.
What I want to do now is take out these two hard drives and put in three new ones that I can link up with RAID 0 (because I've been advised that for RAID, you should really use all the same type of hard drive - right?)
Could anyone give me any advice about this - particularly about what hard drives I should buy? I am currently looking at the 1T enterprise-level Seagate Barracudas.
So, for the time being, I should be able to keep most of my projects inside the computer. What I will do when I've used up all this 3T of space is probably throw old projects onto external hard drives, for storage (plus back up on a separate hard drive and also cut onto DVD.)
For back-up, I want to get a Sans Digital box: MobileSTOR MS1CT - 3.5" SATA to USB/eSATA/Firewire 800 (1394)/400 (1394a) Quad Interface
When I fill up one hard drive, I will just take it out and put in a new one.
What do people think of this plan - does it sound sensible?
On a different subject - I have one large-scale long-form documentary project for which I want to buy dedicated external storage. There's more than 200 hours of footage - over 2500 GB. What should I do? I want the most reliable, future-friendly solution possible - and something that doesn't cost the earth!
Please help!
Thanks in advance.

[Shaun Roemich]

One problem I see with this scenario: starting with 4 new drives, your OS System files would exist on the RAID which is detrimental to system performance. Media should ALWAYS be separate from OS System.

[Lizi Hesling]

Sorry - yes, I'm being stupid. My OS files are on 350 GB drive that I won't take out. So I'm working with 3 bays and plan to expand to 3 TB.
Wrote my post rather late last night...
Has anyone else found drives over 1TB to be slow?

[Lizi Hesling]

So I just edited my original post.
Thanks!

[Greg Quinn]

Lizi, I feel your pain; it seems like I'm forever buying hard drives. The one comment I'd make is to be sure you factor in your budget large backup hard drives as backups. Also, my personal experience in using LaCie drives has been bad (I've used quite a number of them); I've found them to be very unreliable.

[Noah Kadner]

That sounds like a lot more hassle than needed. Why not spring for some G-Tech G-RAIDs or CalDigit drives with FW800 or eSATA. Then you can share between your two computers and cut through the clutter.

Noah

[Don Miller]

Large drives are often faster, as the data is more dense. Last week Western Digital announced two version of the 7200 rpm 2 TB drive. The Caviar Black and the RE4. These would be a good choice for HD video projects.

[Robert Lane]

Lizi,

The implications of your questions are unfortunatley multi-faceted and have less to do with what drive you select and much more about the architecture of your setup and, the difference between working drives, RAID's and an archive solution.

Suffice it to say that these answers have been spread across multiple threads and probably not easily found since the answers have been recycled so many times in different threads.

Later tonight I'll condense that information into a single response that will outline a potential strategy and performance options for you.

In the future this same and related information will be in the workflow section of a new website I'm producing that will encompass all this and tons more info for the indie producer/pro computer user.

[Robert Lane]

Lizi,

You seem to have a good understanding of storage; the Sans Digital option you're looking at is fine however, there are a plethora of storage options using various scenarios from a multi-drive hot-swap bays to multi-drive RAID's configured within a non-removable enclosure (see Find the latest Performance Upgrades, Firewire and USB Hard Drives, SATA, Memory, Laptop Battery, and more at OWC) for the most options in this arena.

If you want to RAID together 2 or more drives in your Mac Pro that's fine; it's a cheap-and-quick solution but it will become limiting if you start having more than 5 streams of HD in your timeline. Although an internal RAID is a simple setup (which does NOT require the RAID card offered by Apple) it is limited to approximately 160Mb/s of I/O throughput. If you find that you're hitting the wall of real-time playback with an internal setup then you should consider an external RAID, the most cost effective is JBOD eSATA (there are 3 different eSATA connectivity types; JBOD, Port Multiplier or PM and, InfiniBand-MiniSAS).

Understand that a RAID of any type is not considered an archival storage option because you're splitting the data between multiple drives rather than contiguously on one. So for your mass storage project that requires 2.5 TB of data you'll need to use really good media management and figure out how to split up the information between single drives - and then make clones of those drives for diversity and safety. If you do opt for using a RAID as an archive/backup then you'll need to treat it as such - turning the unit on only when needed to either add to the storage or, copy off what it holds. All other times it should be powered-off and put in a safe location. "A drive that is powered-on and connected to a system on a daily basis is a *working* drive, not an archive."

There's no need to purchase "enterprise" class HDD's; you'll not see any performance increase at all from the non-enterprise versions and, many of the non-E drives from both Seagate and Hitachi now come with 5-year warranties anyway. HDD performance changes from model to model and currently the lineup from Seagate can hit just over 100 MB/s which was unheard of just a year ago. BareFeats and Storagereview.com do a good job of covering the model lineups and performance differences.

The two most important things when it comes to keeping your storage of media and archive files is

1. To make sure you have physically diverse redundancy; simply put, you want (2) copies of the same thing kept in different locations. That way if an act of god or other catastrophe eliminates your primary copy you've got the backup put-up elsewhere. And;

2. To make sure your files are properly maintained. Use DiskWarrior and other data-file tools to make sure you don't have any resource-fork or directory issues prior to putting away your archives. An archive with corrupted or broken directories is as useful as an empty drive when it comes time for retrieval of that old data.

With regard to future-friendly… with proper maintenance and storage procedures you can expect you'll be able to access the data for the next 5-7 years, safely. Meaning, the technology you use today to create your archives should still be in use by then as well.

However, computer technology is constantly changing and assuming you'd want access to those old files say, 10 or 20 years from now you might be forced into migrating your archives into whatever the newer technology ends up being. For example: How many floppy drives do you see in use today? None. How many PATA drives are being used? Many in older systems. How long will it be before SATA, USB and Firewire are replaced with something newer and faster? There's no way to know - and many will offer their best-guess speculation on that point, but it's all just speculation - especially where Apple is concerned. Count on 5-7 years and be prepared to move into future tech as it arrives.

To wit, HDD's are the most cost effective and reliable archival solution available today. Some will say that tape-drives are still being used - which they are in older archives - but not for newer systems. Some may add that Blu-Ray or other large-scale optical storage media could be used for archiving purposes but when you look at the cost of *any* optical solution their associated costs per-Gigabyte are exponentially higher - by a factor of 20x or more - than any HDD available today, even SCSI.

Lastly, your best bet for any RAID solution would be an external enclosure using hot-swappable drives. An eSATA RAID using the PCI-e connectors will have far greater I/O speeds than any internal RAID you can setup in a MacPro and, it also gives you ease of access to the drives when - or if - you need to replace them either because of storage limits or failure-replacements.

I could go on but I think this is more than enough for you to continue your research.

As I mentioned before, this and other related info will be put into a contiguous site specifically for people needing access to this information in an easy to find format; I'll make that announcement when the site is live.

[Lizi Hesling]

Wow - thank you Robert and everyone else who has replied to my post. Really helpful.

The only reason I thought I might put RAID on the hard drives inside my computer is for convenience and order - all my (current) footage can be kept in one Capture Scratch file, all my projects in a Project file, etc etc.

But maybe this is because I don't know how to order things properly.

Having said that, it's really not too much of a hassle to keep the drives separate - and for speed, this seems to be what you suggest, right Robert?

For archiving, I don't intend to use RAID - I just want to put individual drives into that Sans Digital box, fill em up and put em away (I suppose I should get something like the Wiebe hard drive storage boxes to put them in - although I have no idea where I'd buy Wiebe here in China).

As for what hard drives to buy - if you guys seem to think the 2TB drives are good to go, then I will look into finding the ones you recommend. Is WD really good? I thought Seagate was preferred. But I have to say that I've found a noticeable difference in speed between my 1TB Hitachi Deskstar and my 1.5T Seagate Barracuda - why would this be??

I think I need to clarify something about my internal storage and the external RAID tower I want to buy. I want to expand my internal storage because I have empty bays and I might as well - With 3TB (or 6TB, if I go for the 2TB hard drives) of space on my computer, I can have all my current files in the same place without having to worry about external hard drives. Old projects/footage will be archived. The documentary project I mentioned (which has over 200 hours of footage) is separate from my day-to-day work. I want to keep the whole project (including all edited versions, render files, etc etc) in one place, separate from my editing computer, and mobile so that I can edit on different computers. That's why I want to put it all on a RAID tower. So - Noah, I see you recommend G-drive and Caldigit - I've looked at both of those brands - is this the industry-standard in the US? Not cheap... but are they worth it?

I have one more question - it's kinda dumb, so I hope you don't mind. I see that most of those bigger multi-bay external hard drives only have eSATA connections. How do I connect that to my computers, which only have USB and firewire?

Thank you all again for being so informative and helpful!
Lizi

PS I haven't used La Cie long enough to know if it's reliable or not, but it's the best thing I've been able to get my hands on so far!

[Shaun Roemich]

Originally Posted by Lizi Hesling

PS I haven't used La Cie long enough to know if it's reliable or not, but it's the best thing I've been able to get my hands on so far!

I use LaCie as my media drives exclusively and I've had great luck. Even my two 10ish year old 80GB externals come out occasionally. I've burned through 3 Seagate Barracuda internal SCSI-2's, one Maxtor ATA66 30GB drive is still clinging to life... barely, and my one year old Western Digital external quit earlier this summer but my LaCies just keep running.

[Don Miller]

LaCie doesn't make hard drives. They make enclosures and put in one of the major brands. I expect the number of companies that actually make drives is less than 10.

[Don Miller]

Not recommending this setup, but this is the extreme of internal storage:

Eight drives can be placed in a mac pro if the optical drive(s) are made external. There's an adapter that holds 2 3.5 drives in each optical bay.
The Mac Pro has 6 sata connectors. So a sata or raid card would need to be added.

The cost of a good external 8 drive case with SAS cables is about $650. So there is some cost savings with eight internal drives, but not that much.

A typical 8 drive setup would be a boot drives and seven drives on a raid 5 card.

Note that the 2009 Mac Pro do not have a typical sata connectors for the four hard drive bays. There is a product to work around this issue, but that expense reduces the savings over the external case option.

Everyone has their preference, I like the empty Proavio cases for building external drive solutions.

When buying a complete external raid solution, you are purchasing support, not a special product. That can be a good thing, but check out the firms reputation for fixing problems.

[Lizi Hesling]

Proavio - nice tip... and I see there's a company that sells it here in Beijing.
Thanks Don

[Greg Joyce]

Originally Posted by Noah Kadner

That sounds like a lot more hassle than needed. Why not spring for some G-Tech G-RAIDs or CalDigit drives with FW800 or eSATA. Then you can share between your two computers and cut through the clutter.

Noah

Visit http://wwwcallboxlive.com for Final Cut Studio training DVDs from Noah!

Hey Noah,

Just wanted to let you know the link in your signature is missing a period.

Enjoyed your FCS Color DVD!

Best,

--Greg