To RAID or not to RAID?

[Paul Tauger]

I finally put together a nice quad-core 3.0 GHz 4 Gig RAM (soon to be 8) editing machine so I can handle HDV more easily, and picked up an Intensity Pro on sale, just for fun.

I was trying to capture component to mjpeg and found that, at either 1080i or 720p, I could only capture for between 30-45 minutes at a time. Both the capture software included with the Intensity Pro and Premiere Pro CS4 would simply stop capturing, and always at a different place.

I tested my drive speed and, though it's fast enough according to the Intensity Pro specs, it just barely bests the required speed. I'm thinking of going to some sort of RAID solution to improve capture performance. Here are my choices:

1. External eSATA-based RAID device, probably only 2-drives.
2. RAID card using 2 external eSATA drives (I already have the card, so this is the most appealing, economically speaking).
3. Internal RAID using the motherboard's built-in RAID capabilities.

Is there any benefit from the standpoint of getting the Intensity Pro to capture component for more than 45 minutes or so?

Thanks!

[James Brill]

Honestly for I would do the internal RAID 0. If you have all of your stuff backed up on tapes and your project is saved to a different drive then there isn't much need to get an external RAID unless you want terabytes of storage, 8+ terabytes depending on your motherboard, or speed and reliability from a RAID 5 which is too much for HDV.

[Harm Millaard]

The two pictures below show you a single drive and a 12 disk raid30.

Noteworthy is the performance degradation with increasing fill rates on the single disk, but none of that on the raid30. The same performance degradation happens with a 2 disk raid0. Notice that a 12 disk raid30 is for most of us overkill, but a 4 disk raid3, possibly with a hot spare added, is definitely a serious performance improvement. Not many raid controllers support raid3 however, so the alternative may be to use raid5 or even raid6, although for video editing raid3 is probably the best.

Maybe you need some drive cages to fit those disks in your system, but that is IMO better than externally. No cables, no extra power supply, no external housing.

Maybe also have a look at these guides, there is valuable information in there that may be of interest to you.
http://forums.adobe.com/thread/436215

[Mike Barber]

Originally Posted by Paul Tauger

I tested my drive speed and, though it's fast enough according to the Intensity Pro specs, it just barely bests the required speed. I'm thinking of going to some sort of RAID solution to improve capture performance.

What is the required speed for your needs?

You are looking for a RAID to use strictly as scratch space? Are you looking strictly for performance or security (or balance between the two)?

You're sticking with HDV or planning on moving up to something with a richer data rate?

[Ron Evans]

Firstly, if you have already recorded HDV there is no point in using the Intensity Pro. The Intensity Pro is only of use to record directly live from the camera from the HDMI connector. Once recorded to tape the signal has been compressed and the cleanest way of getting this video to the PC is to capture with iLINK as this is just a file copy.Using Intensity will invoke two more conversions. Firstly in the camera to HDMI and then in the Intensity/PC to whatever format is selected. HDV requires about 3.5MBps. Most modern drives should easily do 50MBps even 100MBps, even external USB 2.0 can manage 20MBps. Even 5400rpm laptop drives will be adequate to capture HDV. IF you want to ease the editing load on your CPU then you could convert the captured file to Cineform or if your CPU ( not your hard drive) is fast enough convert to Cineform on capture. If you do use Cineform then the file is bigger and you will now need a fast hard drive to feed the easier to edit Cineform file to your editing software. But even for this a single stream is easily handled by a modern hard drive. Secondly using a single 2 disc RAID 0, under some circumstances can be slower than two individual drives if it must both read and write at the same time, as it would if it were the only storage. Bottom line is if your editing just one stream of HDV you do not need a RAID or Intensity Pro.
For your information I run Q9450( 2.66Ghz), 8G RAM, Vista 64, 250G OS, 250G TEMP, 2x 750G for video editing and several external 1T drives as needed on eSATA. I have no RAID and regularly ( weekly at times) edit using EDIUS multicam with two tracks HDV native and 2 tracks of Canopus HQ converted from AVCHD. I have no speed problems and a check of performance always shows the CPU as the bottle neck. I also use Vegas 8 and this too has no problems using HDV native even AVCHD native though does slow the frame to the preview monitor at times. Using a RAID 0 would be convenient to not have to select drive for the files but I have found placing the files on the drives to be no real problem( read from one drive and write to another).
I too got an Intensity Pro as a means of editing my AVCHD video when this was not well supported. The Intensity has been on the shelf for over a year now. I guess I should sell it!!!

Ron Evans

[Harm Millaard]

Ron,

You have made some typo's I think. In the numbers you mentioned G should be M, as in MB/s (mega bytes per second).

While I agree with your argument about speed, there are some advantages to raids. Not only do they increase the speed and responsiveness of the system, but the R of redundancy (notice that I call a Raid0 an AID0 because of the lacking redundancy) is a significant advantage. Sure disks fail at some moment, but with single disks you lose all data, with a Raid when you lose a disk, replace the disk and it will be rebuilt in the background, without losing any data. With raid levels 6, 10, 30 or 50 you can even lose two disks at the same moment without losing data (to be completely correct, in the case of 10, 30 or 50, one from each array), and how often do you lose two disks at the same moment? The safety can be further increased by using hot spares.

With the low prices of hard disks, there is no reason not to invest in raids. More speed, more storage, more security.

[Ron Evans]

OOps. Yes Harm you are correct mixed up hard drive size with speed. My point about RAID is that you need more than two discs to make it worthwhile. Also for DV or HDV it really isn't worth it for speed as a single disc is fast enough by quite a margin. RAID makes life easier as there is no need to worry about where to put the files ( they are all on a very big disc) but for RAID 0 with just 2 drives ( which a lot of people use) there is likely no performance advantage ( could even be slower) and an increased risk to data ( all will go if either of the discs fail) rather than just one discs worth if left as two discs. A move to larger disc arrays may be worth while if there is a need for redundant storage or speed for uncompressed video etc. I contend that for the amateur or semi amateur as lots are on this forum, with only two or three discs in their systems RAID adds complexity and vulnerability for very little if any speed advantage. It makes managing files easier on one big disc for a downside of vulnerability. Your comments are valid for someone prepared to install upwards of 6 discs plus OS discs in their system. I have gone the other route by keeping it simple and backing up important files to tape. I have no speed problems using HDV and AVCHD. The CPU is still the bottleneck on performance. Biggest issues are encoding/transcoding, CPU and memory are the bottleneck there for me.

Ron Evans

[Andrew Clark]

Is there a link any of you can post as to what all the different RAID levels mean and do?

Will much appreciate it!!

[Harm Millaard]

AC&NC - RAID.edu - RAID Tutorial & Benchmarking Tools

[Tripp Woelfel]

Unless you just want more speed out of your computer, there is no reason to go RAID for HDV and DV. I can do a multi-cam edit in Premier Pro with three streams of HDV from a single drive and a fourth QT/Anim stream coming from another drive with good performance. It will drop some frames from the display but that doesn't effect my editing capabilities. Scrubbing the footage is pretty effortless.

I'd be an idiot to go contrary to Harm's comments because he's right in what he says. My thought is, do you really want to add to the complexity and maintenance requirements of your system for a little more speed? As with everything, there's a price/prize equilibrium that will work for you. Figure out your current and short-term future needs, and factor in how much money you want to part with and how much system admin time you want to spend and you should have your answer.

My guide is, if I don't need it, I don't get it. If I need it in the future, I wait. It'll likely be cheaper then anyway.

[Dimitris Mantalias]

I use Vegas to capture HDV (I consider it much more stable for the particular job than Premiere) and most times I capture on external USB drive! As mentioned, HDV is a pretty low-demanding format, since its data rate is even slower than USB's. Needless to say, a good SATA drive is more than enough to work with HDV with excellent speeds (never had a single problem, unless the color correction begins, but thats a CPU/memory issue).

[Harm Millaard]

Tripp,

As I said in post #6, there are three advantages to a raid configuration, the triple S law:

Speed, safety and space. There is no drawback in terms of maintenance or complexity.

Safety is a major advantage that outweighs IMO the trouble of setting up a raid. If you use single disks any of them can fail and occasionally do. That means you just lost all your data on that disk. If you have a raid, a disk can also fail, but after replacing the disk with a new one, you have not lost your data, it will be reconstructed in the background.

[Nicholas de Kock]

I have 5 drives in RAID 5 and it works great! I've lost three drives this year on separate occasions due to the firmware issue Seagate had, luckily I lost no data, swapping out the faulty drive was all I had to do. Data is money, get a redundant raid system! It's faster and makes sense.

[Tripp Woelfel]

Originally Posted by Harm Millaard

Tripp,

As I said in post #6, there are three advantages to a raid configuration, the triple S law:

Speed, safety and space. There is no drawback in terms of maintenance or complexity.

Yeah... I think I went a little deep on the maintenance overhead point. To be honest, it's been over a decade since I used a RAID and longer than that since I was configuring software and hardware RAIDs. Undoubtedly, things are much lower maintenance now.

To go to the OP's issue of the capture stopping prematurely, a RAID may or may not subvert the problem but it will not resolve his underlying system problem because any contemporary machine should be able to capture DV or HDV without a RAID. Thinking a RAID would resolve the problem is like thinking a poultice will fix a broken leg.

[Peter Moretti]

I think the speed argument actually can go out the window when only two physical drives are used. If your media is stored on drive #1 and you render to drive #2, I believe this will be a faster arrangement than if you were to RAID dives #1 and #2 together into a RAID0 configuration and store media on and render to the same RAID array.

Of course this doesn't hold if more drives are added to the equation. But as rule you want to keep your media on a different physical drive than you render to. So depending on how many drives you want to throw at the problem, RAID may or may not be the correct choice. If the answer is you're only going to use two physical drives for video editing, then RAID is not a good choice, FWIU.