eSata for connecting external RAID drive sets

[Andrew Smith]

I'm looking at purchasing an external RAID system and I'm mindful that the newfangled USB 3 is much faster than eSATA (which itself is still quite speedy).

The question is ... am I ever likely to exceed the bandwidth that you get with an eSATA connection to the external array? Am I getting worried over nothing? Should I go for a unit that supports USB 3 instead?

(I have a feeling that USB3 will tax the CPU a bit, but I need to research that one to know for sure.)

Thoughts?

Andrew

[Bart Walczak]

The question is what kind of files, resolutions, and compression you're going to work with, which eSata (1.5/3/6 Mbit) version you're using, what kind of motherboard you have, and how many other drives are connected via eSata.

[Andrew Smith]

Thanks Bob,

The computer is yet to be built, but will be in the very near future. I'm keen to do it right and this is part of the research.

I've purchased my upgrade to Adobe Production Premium Suite 5.5 and I'm about to purchase a Matrox MXO2 (with Max technology for super speedy h264 encoding). The MXO2 is connects via PCIe.

I've got a large collection of HDV recordings and will continue to record to this format with my Sony V1 camera. From time to time there will also be content in the AVCHD format from the Sony HXR NX5 camera.

With the Matrox MXO2 gear, there is the option to capture to 25, 50 and even 100Mbit/sec mpeg2 files for super quality work. I think this option is there with a view to having higher quality video to do good chromakey work with, etc.

Andrew

[Magnus Helander]

We use Highpoint RocketRaid eSATA cards with connections to two cabinets each holding 4x SATA drives running RAID5. No performance issues for editing several 4:2:2 iframe 150 mbit/s streams.
Problem is not in disc read/write, bottleneck is processing/CPU power.

HighPoint Global website
/m

[Andrew Smith]

Interesting. Amazing what we have access to these days. Not that I am expecting to be editing raw uncompressed streams.

Has anybody had any experience with the G-Tech units? See Videoguys.com - G-TECH from Hitachi

Andrew

[Bart Walczak]

Originally Posted by Andrew Smith

I've got a large collection of HDV recordings and will continue to record to this format with my Sony V1 camera. From time to time there will also be content in the AVCHD format from the Sony HXR NX5 camera.

With the Matrox MXO2 gear, there is the option to capture to 25, 50 and even 100Mbit/sec mpeg2 files for super quality work. I think this option is there with a view to having higher quality video to do good chromakey work with, etc.

With HDV and AVCHD footage you'd best work natively, without any transcoding, and no capture will make the footage better than it already is compressed, so you can just get those files via firewire and/or USB in their original format. HDV and AVCHD have pretty low bitrates, so I highly doubt you'll ever get to saturate your eSata connection. Even Matrox'es codecs shouldn't be much of a problem if you choose to capture live, although personally having had my own share of problems with Matrox cards, I tend to avoid their solutions.

[Andrew Smith]

I've used Matrox products for a very long time now. With all the internal resource intensity that editing video demands of a computer there are plenty of ways that an editing experience can go wrong.

What I've observed with Matrox is that they took it upon themselves to do extensive testing with various motherboards, sound cards etc to validate what will work successfully in a video editing system with one of their cards. Additionally, they only release their drivers for the latest edition of PremierePro etc when they are ready for production and not before. I've seen it consistently over many years, and it's the sort of thing that takes a privately owned (not shareholder profit orientated) company to do.

The last time I built an editing computer, I used components from their tested/validated list and spent well on the editing box as a whole. The result for me was a rock solid editing machine which has served me faithfully.

Thus here I am, researching so that I can spend well on my next edit computer.

Andrew

[Andrew Smith]

Also, thanks for the feedback regarding the potential for saturating the eSATA channel.

Andrew

[Magnus Helander]

I forgot to mention, we use Stardom cabinets for the drives - Taiwanese all aluminium with zero issues.
STARDOM STORAGE SOLUTIONS - SOHOTANK FEATURE
/m

[Andrew Smith]

Forgive me if I'm a bit clueless ..... but I'm beginning to get the idea that some of the external RAID units are merely physical enclosures to hold and power the hard drives, and some additionally have the hardware included for handling the RAID functionality internally?

I've been looking through all the G-Tech section on the VideoGuys site and it's a little bit hard to figure out as to why some units are priced in the hundreds of dollars and yet others are priced in the early thousands of dollars.

Andrew

[Magnus Helander]

Here is my limited understanding of the situation to why the costs differ:

Challenge is how to connect the external drive bays/units in one or several cabinet/s to your motherboard so that you can get the data from the drives at the highest possible rate.

I know of three kinds of connectivity
- SAS - Serial Attached SCSI o
- SATA - Serial ATA
- iSCSI over Gigabit Ethernet

You need to get the data from drives to your CPU over the PCI-express PCIe bus - PCIe is operating at 250 megabyte/sec per channel. Videocards operate at full 16x speed, RAID cards only needs 1x, as you will have a hard time to get 250 megabyte / sec transfer rate from your drives.

So, some have all the RAID logic in the cabinet and present a single logical drive to the operating system across eSATA or a (possibly, i'm not sure...) SAS connection.
Others put a RAID card/logic in the PCIe bus and then connect "dumb" drives in cabinets.

So, it depends if you have SAS / SATA / iSCSI connectivity to your drives, and if you need a PCIe card, and ... well.. the overall performance.

Factors to consider
- Read & write rates for the physical drives
- Combined read & write of the logical RAID unit/drive
- Ability to expand.
- RAID 5 or other configuration...

And check for "burst" which is what the unit can deliver as a single peak/burst, and "sustained data rate" which is what it can deliver continually during playback

This is a nightmare, so ask some storage professionals, I work with people who is fascinated by this stuff - I just want it to work. 150 mbit / sec I-frame 4:2:2 is approx 19 megabyte / sec datarate.

/magnus

[Gary Bettan]

A very good thread. Thanx to Andrew for giving me the heads up. As was posted, the G-tech RAIDs come in several levels.

G-Raids & G-Speed Q attach vie eSata (or firewire800/400/USB). eSata is the fastest connection and the one we recommend. We plug ours directly into the integrated eSata of our DIY motherboards. G-RAID and G-Speed Q offer the same throughput and datarates. The only difference is that you can configure the 4 drives in the Q as a RAID 5 for data protection. This protection costs you 25% of you total storage. So an 8TB Q with 4/x2TB drvies gives you 6TB of storage. A Graid is a RAID0, which gives you the full amount of storage of the 2 drives, but an drive failure results in all data lost.

G-Speed eS uses a G-Tech PCIe x4 RAID controller which gives you higher throughput capable of multi-stream ProRes 422 playback, but also ads costs.

G-Speed Es Pro uses an expensive mini-SAS high performance PCIe x8 IOP RAID card. Even higher thoughputs and capable of multi-stream ProRes 422 HQ playback.

We have a video storage FAQ on our website with some additional info Videoguys Blog - Videoguys NLE Video Storage FAQ

Gary

[Andrew Smith]

Thanks Gary.

To be honest I've had my heart on a RAID5 system whereby you get both the throughput speed as well as the protection that you mentioned.

On the other hand, I've had none of my regular consumer HDDs fail in the previous 6 years that they have been running in the computer. (I know, I know ... getting a bit old for hard drives.)

I was thinking: I could possibly just purchase a couple of 'enterprise' level drives and RAID 0 them together and it might be some sort of a solution? Just a thought.

(off to read the storage FAQ now)

Andrew

[Gary Bettan]

Here is the thing about hard drives. They don't just all of a sudden fail out of nowhere. Either an event causes them to fail, such as a power surge or physical damage, or they die a slow death.

What I mean by this is that if you pay attention to your system performance, you will know something is going wrong with your storage. Look and listen for any of these:

- clicking or loud whirring
- drives spinning hard for no apparent reason
- delays in starting clips or opening files
- windows explorer acting kind of sluggish
- heat or fans having to work extra hard to keep cool

When you see the first signs of any of these symptoms, back up your work and start planning on replacing the drives. The sooner the better.

Gary

[Andrew Smith]

Gary,

In the spirit of not having to pay 10% GST on items brought in to Australia from overseas when the value of the goods is less than $1000 AUD .... it is possible to purchase the 4-drive G-Speed eS unit with no drives in it? (Would also make it lighter for freight too, I imagine.)

Andrew