eSATA RAID 0 question

[Scott Auerbach]

Pardon my ignorance on RAID issues, but I'm still trying to figure some of this geeky stuff out...
If I were to get an external SATA box (say, for example, the mini-G 4-drive box) and stripe 4 drives as RAID 0 for fast throughput, can that box hang off a single eSATA port on a MacBook Pro, or does it require 4 eSATA channels?

I've seen RAID expresscard controllers for the G5/desktop world that have multiple channels on one card, but I can't seem to determine if such a card is ~necessary~ for a RAID stripe. I wouldn't think so, given the eSATA II high-speed protocol... but I can't figure out if the multi-lane cards are just a way to take the RAID processing overhead off the computer, or if they're essential.

And if you can hang a 4-way stripe on a single eSATA port, does the external box need its own RAID controller hardware, or can it be just a bunch o' drives?

All this is on hold, of course, until there's an express/34 eSATA card, but that appears to be on the horizon.

[Rob McCardle]

Hi Scott - the rule of thumb with raid 0 configs (and this only relates to raw data throughput) is to have the drives on seperate channels - yes, despite what the enclosure manufacturers will tell you - you will get faster throughput when your drives are each on their own dedicated channel and then combined in a raid set.
That is why the controller cards for the towers have all those channels, each channel is on it's own bus - no head banging (I/O data bashing) in the bus pipelines.

I can't comment on eSata performance with a MBP (later this year for me when Merom comes out).
I hope that Apple will add an eSata port so that you could add a card and stripe 2 drives RAID 0 like we can now with fw 800 on the pb's.

eg. <link removed, sorry check barefeats search for eSata and a reveiw of the FirmTek 4 drive enclosure> - for this to fire optimally you would have a 4 channel card in a tower and run the leads. You would not run one lead and stripe the drives.

Also here - <link removed, sorry check barefeats and an 8 port PCIx card> - ok he's yapping on about pciX but the principle is the same.

Hope that helps.

edit: oh yeah, should add (I'm sure you know but just in case you don't) - striping is utterly great and can give the sheer speed required as long as you realise it's a total crash and burn type of deal.
Lose one drive and you're toast - so have a good backup plan in place - regularly.

[Scott Auerbach]

Originally Posted by Rob McCardle

Hi Scott - the rule of thumb with raid 0 configs (and this only relates to raw data throughput) is to have the drives on seperate channels - yes, despite what the enclosure manufacturers will tell you - you will get faster throughput when your drives are each on their own dedicated channel and then combined in a raid set.
That is why the controller cards for the towers have all those channels, each channel is on it's own bus - no head banging (I/O data bashing) in the bus pipelines.

Thanks...I figured performance would be better with a multi-lane setup...but that's not an option on the MacBookPro. The best we can hope for is the upcoming 2-lane express/34 card. That being the case, can a 4-way stripe be hung off a single eSATA channel? Even with diminished performance? I just don't want to invest in something that won't work at all. I don't expect ideal performance, but if it'll work...and significantly speed up the process of offloading media from the P2 cards...then it's worth the investment.

As for the crash-n-burn... yeah, been there done that on Avid stripes. Argh.

[Rob McCardle]

Scott - you're on the bleeding edge !

I'm going to send you over yonder -
<link removed, sorry check Firmteks site for their Seritek adapter>

I'll have a stab and say you'll be ok - but then again ...

Look at the data throughput/port AND on the one eSata bus 3 freak'n Gb/sec -eeekkk.

pls note: I'm not affiliated to FirmTek in way shape or form - but they make some cool stuff, IMO.

[Robert Lane]

Scott,

As you might know, there are 2 kinds of SATA enclosures: the direct connect type or JBOD where each drive gets it's own data cable and, the port multiplier type (PM) which gangs several drives together into one eSATA cable.

For the towers that means there are 2 kinds of cards: The type that uses a single connector for each drive such as the HighPoint RocketRAID 2320 or, uses the PM type connector like the Sonnet Tempo E4P. The two types have completely different setup schemes and, right now the PM type has many hidden bugs that haven't been worked out yet. As Rob mentioned, it's "bleeding edge" technology right now still in it's infancy.

OWC makes a great enclosure that might work for the MBP: It's a single drive case (nice black color too) that has (2) eSATA connections, one for each drive. You could RAID those together in OSX as software RAID, and in a RAID 0 configuration you should get about 137 mb/s READ speed and a little less on WRITE. This assumes that you're using SATA-II drives with lots of cache on each drive.

SIIG and a few others are currently making a 2-channel Express Card 34 SATA adatper which would give you the ability to use the OWC enclosure.

The real caveat here, is that you'd still need one more external drive, say a FW400 type to use for your backups, so that if the array fails for any reason you've got a current copy of your project and media files.

[Scott Auerbach]

Originally Posted by Robert Lane

Scott,

As you might know,

Hell no, Robert... that's why I ask Godzilla! <g>
Leave it to me to be bleeding edge and not even realize it...

In my techno-flailing, I'd kinda gotten the impression that there were, as you say, two ways to skin this cat. But all of the deeper articles I'd found dealt with multi-channel RAIDs...no reference at all to spitting it across a single cable.

So far I haven't found any express/34 eSATA II cards, though Vydeo has announced a two-lane card for release sometime soon.

Correct me if I'm misunderstanding, but it sounds like you're describing close to double the I/O of a single eSATA drive by doing a two-way software RAID 0 stripe. If that's the case, isn't it reasonable to assume that I could get close to 4x single-drive speeds on a 4-way stripe? That would sure help get the files off the P2 cards in PC mode without totally slowing down the shoot day. FW (400 at best possible performance) out of the camera, and eSATA at 250-ish speeds onto the RAID array.

I definitely understand the "lose-one-lose-all" issue with RAIDs (having experienced that twice in 10 years on Avids), and the need for some kind of backup...that's not a problem. At this point, I'd just like to get the footage offloaded while on location as fast as possible...I anticipate doing a lot of SELLING to get the budgets for a three-man crew (me as DP, camera assist/data manager/focus puller/grip, and audio tech). If the sales job on having the camera assist includes the increased production speed by not having to do my own media management, then I have to be able to keep the camera up and shooting as much as possible. Especially once there's some kind of P2 external reader...I have 3 8GB cards in hopes that I can get media off card A while B and C are shooting. If I can get 1GB off in significantly under a minute, 3 cards is all I'll ever need.

Beyond the "will this work?" question: I can't find any OSX Tiger documentation/help text on doing a RAID 0 stripe. Anyone have a Kbase link?

[Rob McCardle]

<link removed, sorry check barefeats front page, scroll a little to the review of the Seritek adapter>

Check the graphs - that is what you can expect.

Also note the raid stripe - 2 ports active, 2 drives in 1 enclosure.

The issue for transfer/copy will not be bandwidth for eSata (there's bandwidth for miles there) it will be fw400 - that will be the slowest link in the chain. Not much you can do about that ...

Sorry Chris - I had links pasted in there - forgot about policy - rush of blood to the brain ...

[Scott Auerbach]

Will do... thanks!!!

[Scott Auerbach]

Originally Posted by Scott Auerbach

Beyond the "will this work?" question: I can't find any OSX Tiger documentation/help text on doing a RAID 0 stripe. Anyone have a Kbase link?

Gee...quoting myself here. I found the RAID stripe utility in DiskTools. Shoulda thunk to look there.

[David Tamés]

Originally Posted by Scott Auerbach

I found the RAID stripe utility in DiskTools

My suggestion for any Macintosh user that's setting up a software RAID is to consider doing it with SoftRAID rather than using Apple's Disk Utility for setting up a software RAID. This is what I've been using since I discovered that the friendly Apple interface does not extend into error reporting and disk recovery. SoftRAID allows you to set up a RAID 0 (striping) or RAID 1 (mirroring) with any disk that's connected to the Mac via Fibre Channel, SCSI, ATA, SATA, FireWire, or USB 2.0. I hope in a future version they will also support RAID 10 (a.k.a. RAID 1+0), which Apple added to their Disk Utility in Tiger.

Disk Utility provides a no-frills interface but you have to use the command line interface to rebuild in the event of the failure of a mirror disk, and a RAID configured with Disk Utility is less reliable because Apple does not offer the level of error checking and reporting that SoftRAID does, letting you see I/O transactions, error counts, and volume status through a very Mac-like interface that makes it a snap to create and maintain RAID volumes. A SoftRAID vs. Apple Disk Utility Comparison is available on the SoftRAID web site.

[Robert Lane]

Scott,

One last thing to consider about any SATA-type RAID setup:

Unless you use a RAID card that has on-board cache on the controller (I only know of one and it's pricey) you'll find that as you work on video files the RAID starts to slow down as you work.

This is because as you import, render etc. the only cache available to help process the large video files lives on the HDD's themselves, which is why it's always best to use drives that have "big" cache on-board - no less than 16mb per drive.

So, as you work the cache on each drive fills up immediately while it x-fers the info to the disk and back to the system and so forth. This overloads the on-board drive cache to the point where it literally can't keep up.

Case in point:

Using the KONA System tester I put a 2-disk RAID 0 array to the test. Using the DVCPRO codec, 128mb file test the first test showed about 140 mb/s. After the 5th consecutive test the best READ speed was only 109.7 mb/s. This accurately represents how the drives would perform in a video editing environment moving large video files - which would be greater than 128MB I might add - back and forth.

Those results are a very important thing to note, because all the tests posted by AMUG, BARE-FEATS and others don't address the degradation-over-time issue, they simply show what the maximum *possible* throughput is for the various setups that have been tested.

So while SATA and eSATA enclosures are showing up on the market they're not exactly optimized for a video editing platform. They will work (once the PM bugs have been worked out) and they will perform faster than a JBOD or any single drive, but they do have a built-in "peformance degradation factor" that can't be overcome with 98% of the setups currently offered.

[Scott Auerbach]

Godzilla squash my hopes! <g>

Thanks, once again (I feel like I need to do that daily) for yet another incredibly important nugget that has gone completely overlooked in all the material I've read (here and elsewhere) on P2 workflow. I was always assuming a 4-way SATA stripe (minimum) would be the way to go, but clearly a 2-way would offer even more iffy performance than I realized.

So tell me... has anyone ventured an opinion about whether it's even physically possible to cram the electronics for a 4-drive eSATA raid controller on an express/34 card? I know they're all PCI cards at the moment, but it looks to me like it's a whole lot of circuit board with not much on it...

[Robert Lane]

Keep in mind, that while SATA and eSATA setups do lose speed as you progress in your editing, they still perform better than a single drive. It's just that a 2-drive stripe really isn't much faster than a single drive over time, which doesn't really make it a cost effective solution.

The Express Card adapters that are coming so far only allow 2 drive connectors. I haven't done any research into what else is coming down the pike.

My suggestion, is that if you intend to do most of your edits on a MacBook Pro that you use multiple external FW drives instead of the SATA striping method. It's more stable, less expensive and you'll be working with proven technology rather than putting your work on the backbone of "bleeding edge" tech that really needs to be refined for video use.

I don't know how much storage you need or what your project requirements are, but you could use (3) FW drives to accomplish fast and stable edits on the MBP: One for your capture scratch, one for stock video, music, SFX, Waveform and Thumbnail cache files, and one to save your project files and to using Media Manager to create your backups.

At some point within the next 6 months to a year the SATA RAID setups will get better and there will be controller cards with on-board cache to eliminate the time degradation factor, but for now you're better off with FW externals for the MBP.

[Scott Auerbach]

Originally Posted by Robert Lane

My suggestion, is that if you intend to do most of your edits on a MacBook Pro that you use multiple external FW drives instead of the SATA striping method.

I don't know how much storage you need or what your project requirements are, but you could use (3) FW drives to accomplish fast and stable edits on the MBP: One for your capture scratch, one for stock video, music, SFX, Waveform and Thumbnail cache files, and one to save your project files and to using Media Manager to create your backups.

Thanks, Robert. Realistically, it'll probably be 6 months before I'm doing a whole lot on FCP anyway, so the technology will be farther along, I hope... I'm an Avid editor and have regular access to Adrenaline, but bought FCP to be able to wean myself from that relationship...and to offer some degree of in-field editing.

That said...I'm a bit confused by your quote above. I see the advantage of splitting the files up like this, to avoid hardcore head-bashing running between project and media files... but wouldn't a single FW drive for media end up being too slow for HD100? I thought the 7200 rpm drives were still only delivering 60-70Mbs sustained read rates...? I thought I'd need at least a 2-drive stripe no matter what the I/O protocol was. Am I (probably) wrong?

[Drew Harty]

One last thing to consider about any SATA-type RAID setup:

Unless you use a RAID card that has on-board cache on the controller (I only know of one and it's pricey) you'll find that as you work on video files the RAID starts to slow down as you work.

This is because as you import, render etc. the only cache available to help process the large video files lives on the HDD's themselves, which is why it's always best to use drives that have "big" cache on-board - no less than 16mb per drive.

So, as you work the cache on each drive fills up immediately while it x-fers the info to the disk and back to the system and so forth. This overloads the on-board drive cache to the point where it literally can't keep up.

Case in point:

Using the KONA System tester I put a 2-disk RAID 0 array to the test. Using the DVCPRO codec, 128mb file test the first test showed about 140 mb/s. After the 5th consecutive test the best READ speed was only 109.7 mb/s. This accurately represents how the drives would perform in a video editing environment moving large video files - which would be greater than 128MB I might add - back and forth.




Hello Robert,

Doesn't the throughput of all software controlled raid 0 arrays, whether SATA, firewire, or SCSI, decrease as the drives fill up? (Unless, as you say, there is a hardware controller like Medea uses.) From test I have seen a decrease in read/write speeds of 30%-40% is typical. Is there something about the SATA or eSATA connectors that makes throughput decrease more than a firewire 800 connector? Doesn't DVC Pro HD only require 13 mbs throughput?

I have used a SATA raid O on my desktop for a couple years editing uncompressed 10 bit SD with no problems and was going to use two SATA drives with my MBP (when Firmtek releases it new express 34 card) in a raid 1 for instant backup using Soft Raid to record interviews directly from my HVX in 720 24p. Do you think this will be a problem? Are you suggesting firewire drives would perform better in this use?

thanks,
Drew Harty