Raid-0 Expectations

[Dana Salsbury]

I'm finally going RAID-0 on my Quad w/ 6gb memory. I'm installing two VelociRaptor 300 internals @ 10000rpm. Until now I've relied on firewired externals.

Question: With this setup will I be able to simultaniously render HDV to the VelociRaptors while capturing HDV with a networked Macbook pro (also to the VelociRaptors)?

I don't want any gremlins, but capture time is killing me in producing three-camera weddings.

[Robert Lane]

Dana,

Unless there's an adapter I'm not aware of you can't install the VelociRaptor in a MacPro or any other RAID-tower/server setup with removable trays; the SATA/power connectors have moved from the normal lower-side position to squarely in the middle of the drive - they wil not match up with where those connectors on the MacPro mainboard. Sorry.

Also, you're not going to get good results by creating an internal RAID; if you're serious about increasing drive performance you're going to get a *real* external RAID enclosure, whether it's eSATA, SCSI or Fiber.

[Dana Salsbury]

Thanks Robert,

There is an adapter, and Western Digital told me it had to be internal, but I believe you over them. What would you recommend? We've got to be able to capture and edit at the same time, and to the same drive(s). I was impressed by the 10000 rpms of the Velociraptor...

[Robert Lane]

WD is correct; the only way the V-raptor can connect is with a standard SATA/power cable going to an internal SATA controller card or to an addtional SATA mainboard connector. However in a MacPro this won't work because there's no way to put any addtional connectors on the back of the drive because the drive only has room to interface with the built-in connections on the "backplane" of the MB. You simply can't use the V-raptor in a MacPro, it would have to be in an external enclosure connected via eSATA host card.

You *can't* capture AND edit to the same drive at the same time; the only way you could pull that off is if you had a very large and *fast* RAID array with a LOT of bandwidth to handle two systems hitting the same array at the same time. (I'm assuming you're talking about using more than one Mac, right?)

That means you'd need either a SCSI or Fiber array because no eSATA array of any size would have the ability to handle all the HDD cache required for multi-user editing.

My suggestion would be to get with one of the A/V suppliers and or a FCP specialist locally and contract them to help you figure out how to accomplish what you need. Without knowing your budget and your exact workflow - and exactly what you're trying to accomplish it would be difficult to make direct suggestions.

[Giroud Francois]

to capture and edit at the same time, you just need to capture to another disk, whatever speed it is, no needs for RAID.
it is not even sure that a fast RAID would be ok to capture and edit on same disks, because you can be never sure if the sustained write would not drop below requirement if a big sustained read would occur at same time.
with two disk , there would be no question.

[Mark Keck]

Originally Posted by Robert Lane

Dana,

Also, you're not going to get good results by creating an internal RAID; if you're serious about increasing drive performance you're going to get a *real* external RAID enclosure, whether it's eSATA, SCSI or Fiber.

Robert, can you elaborate on this some??? Is this the differnce between a internal software RAID vs an external hardware RAID??? If not, what is the limitation for an internal RAID??? Also, what would you expect to be a reasonable data rate an internal RAID be capable of supporting??? I know, I know... too many variations... but could you ballpark it??? And how would this compare to an external RAID performance???

Mark

[Robert Lane]

(laughs) Yes, many questions but I'll try to simplify:

The best you can hope for (real world numbers, not the hyped-up theoretical numbers) in an internal array is around 110-130 MB/s, but that's not a constant because just like with external eSATA arrays there is no controller cache available to offload the HDD cache when they fill up, so as you work on edits the data-rate will slow down to about the speed of a single HDD.

There's tons of info (including tests that I've done) posted on the forum; if you use the search feature you'll find more info than you handle.

The only real benefit to creating in internal RAID is to make one large single-partition, but it will not be fast. If you do go that route DO NOT put your OS on an internal array, make sure your boot-drive has it's own physically separate HDD, not a partition on the array.

[Giroud Francois]

A raid is a raid. Does not matter if it is internal or external.
The bottleneck is at different place. Depends your hardware.
the first one could be the disk. An average SATA 300 drive is about 60Mb/sec so you can expect with Raid 0 a minimum of 120 Mb/sec.
For example my Shuttle barbone X38Pro uses 2 Western digital 320 gig sata drive in Raid 0
they are average quality disks and the Blackmagic utility for testing disk speed reads 145Mb/s.
that is on an empty disk. a full disk can loose as much as 50% of its speed. so you better had to plan twice the capacity needed so you are sure to never end up with full/slow disks.
in my case , a got 2x320=over 600gig of disk space, so i am ok in the first 200gig.
(i am supposed to capture components directly out from the camera, bypassing HDV compression).
when i reach this capacity, the captured clips are transferred to another drive (one internal and one external for backup).
This pc has also E-Sata plug (one the same controller, so there would be no difference if the disk would be inside or outside the case).
The second bottleneck could be the controller . if the card is on a PCI bus (limited to 133Mhz) there no chance you could get faster than a SATA150 disk could give.
if the chip is on the motherboard, you have to check how it is linked the the CPU (kind of chipset used)
It can use a PCI bus, a PCIe bus , at 1x or 16X. It could share the bus (bandwith) with something else (USB controller) .
here is a picture of the architecture of my chipset (X38)
http://www.intel.com/Assets/Image/di...ck_Diagram.jpg
here you can see that if any link between a SATA connector is 3G/s, the main link between the other part of the chipset is limited (and shared with USB, audio, Ethernet ) to 2G/s .
that is what is called Northbridge and Southbridge by chipset specialists.



The last bottleneck, is the way the RAID is managed, the kind of command it can handle, some controller being a way better than other just because the firmware.

but again, even the fastest raid could get problem if the head must read video and write at the same time. the displacement of the head will probably be so heavy, that you will not be able to get sustained bandwidth either for read or for write.
While a simple cheap harddisk dedicated for capture will do the trick , even on an USB2 port.

[Mark Keck]

Hmmm... I've got a 5 disk internal RAID 0 using bays 2 thru 4 and the 2 extra SATA ports with drives located in the spare optical drive bay. The drives are WD2500YS enterprise drives forming a 1.1TB RAID. It's about 2/3 full. Using the BM Disk Speed Test I'm getting about 290MB/s reads and writes. In real terms, I've been able to capture 10 bit 4:2:2 YUV 30fps uncompressed without dropping a frame. But I'm always looking for more, so your comments made me wonder.

[Robert Lane]

There are a few third-party kits that allow you to install up to (8) SATA drives in a MacPro however this is not an ideal situation. (Be very wary of the "MaxUpgrades" kits as they are *not* approved by Apple and there have been documented MB failures as a result of those "upgrades")

That scenario is not "hot-swap", so if you have a drive failure you've got a lot of work to do to replace it. You're also at a near-overload situation with respect to the internal power supply for the machine not to mention adding unplanned-for extra heat to the entire tower that has to be moved out. HDD's generate a great deal more heat than optical drives and the airflow routing for that area was not designed to handle that additional heat.

Mac's have always been very carefully designed and Apple has several white-papers about the data-flow design of the MB, heat-dissipation, and power-supply load-levels.

With respect to your data-rates, they will initially test very high but as your drives are used during an edit the data-rate will drop because there is no hardware RAID controller offering mid-level cache to offload the internal HDD cache.

Just because an accessory has been created for internal modification doesn't mean it's well thought out. In fact, Apple has clearly stated that certain internal upgrade "kits" will void both the standard and APP warranties if installed. Buyer beware.

[Giroud Francois]

again , having one Ferrari is useless if you need to go at to different location at the same time. Getting 2 cheap cars will be more efficient. (or getting 2 ferraris even better).
in your case, getting 2 raid (one for capture and one for editing ) will be more efficient that having only one superfast raid.
I never understood the Blackmagic utility screen. do they display in Pic/sec or frame /sec ?
for example if it says 37 in the 1080, doest it means you are ok (over 30 im/sec) or do you need to have over 60 (frame/sec) ?

[Dana Salsbury]

I don't see the need for a Raid-0 for capturing. It's realtime, so my firewired extrernals do fine. Rendering is where Raid-0 would shine, but I'm not sure what to look for. It sounds like I need to find an external Raid-0 enclosure w/a firewire 800 out. (recommendations?)

If I capture to a HDD (A) and render to the Raid-0 extrenal (B), can I capture again to (A) while (B) does more rendering? Wouldn't the render be pulling from (A) to post to (B), and would that be too much for (A) and lead to dropped frames?

If it worked I could clean out the older capture scratch files to keep (A) light and archive from (B) to keep (B) light.

[Mike Bisom]

Let's start with the idea that software managed RAIDs are not a good idea. Hardware managed RAIDs are expensive for a (good) reason. A RAID 0 will NOT help with rendering. Rendering is a frame by frame processor intensive activity- it doesn't take much to write one-frame at a time. You can decrease rendering times on a Mac by using Compressor and multiple Macs and essentially creating a render farm.

Today's drives are essentially fast enough to handle most any capture without needing a RAID. In terms of workflow, having something like a MacGurus Burly Bay, with swappable drives is great: each project goes on it's own drive. This allows you to capture to one drive while editing a project on another drive (provided you have the software to allow you to do both). Or, have one computer as a capture station to a single drive and then remove that drive and insert into tower for editing. Bottom line is that solutions are out there but typically, a RAID is NOT a solution. And for what it's worth: I have had software and firmware (but not hardware) RAIDs.

Mike

[Mark Keck]

Robert, you take all the fun out of life :-). Yup, I'm violating several rules with this one; and nope, I wouldn't recommend it to anyone. I'm not using the maxupgrade kit, you are correct that the maxupgrade kit is not kind to the air flow. My design is a little better in that regard. Also, my power supply calculations almost reach the limits of it capabilities.... almost, but it is within the capabilities of the power supply as specified. I'm confident that the good engineers at Apple over designed and under spec'd it. That's how I design anyway. Yup, it pretty much nulls the warranty. I'm a EE in my day job and I could write a book on how customers find creative ways to violate warranties. Please point me to these white papers of which you speak, I've been searching and can not find.

Giroud, I've been running with this for about a year. When I first installed it I partitioned it as 2 RAID 0 drives and ran a number of test rendering files sourced on one drive and targeted to the other. Didn't really see a difference in render times from a single drive configuration. Doesn't make sense to me either, but that what I found to be the case. As Mike mentioned it probably won't help render times.

Mike, No need for RAIDs??? Not sure I agree with that. I suppose if all you do is ingest via firewire then you would be correct. However, I wasn't able to capture uncompresses 1080i until I started using a RAID.

Dana, my apologies for taking your post off topic. I believe your solution will work just fine.

[Dana Salsbury]

No worries, Mark. This is all great stuff.

>You can decrease rendering times on a Mac by using Compressor and multiple Macs and essentially creating a render farm.

I want a render farm! Would you post a new thread on how you can utilize other networked comps using compressor? That is too cool.


MacGurus Burly Bay sounds great, but swapping drives continually concerns me per wear and tear. It seems a violent process to do one time, let alone several times each week. It seems safer to switch firewire plugs. Even better would be a switcher hub, but I haven't seen any.