Hi there,
I currently edit SD (on a Mac powerbook, using Avid Xpress Pro and an assortment of fw drives), but plan to get an EX1 and a MacBook Pro very soon, and want to start editing HD.
I've been told that you can edit HD with fw 800 drives, but a friend strongly advised RAIDED SATA drives - - this would mean eSATA for me I guess.
Would people on this forum be kind enough to advise me as to what my best purchasing decision/workflow would be re hard drives?
I've read lots of very good things about CalDigit's HD Pro set up... it's very expensive, but CalDigit just came out with a cheaper, extremely fast set-up called HDOne. It has removeable drives (up to 8 of them), built-in PCI Express, supports RAID 5 and 6...
It SOUNDS very good to me, but I'd love some advice.
If RAIDED SATA drives ARE the way to go... what would my best modus operandi be...
Would I come back from a shoot, transfer footage from my MacBook Pro to the eSATA drives AND AT THE SAME TIME transfer to a fw drive (as a backup)? Can you do this at the same time?
Thanks for any advice.
Malcolm

Well, aside from caldigit, there is
http://maxxdigital.com/shop/index.php?cPath=58_78
I have heard these drives are some of the best and they are slightly cheaper than most other brands. They may not look like much, but its the inside that counts, right.
http://www.dulcesystems.com/
This company also makes some great raid setups.
g-technology.com
these are great also and are more expensive.

All of these are great options that you can't go wrong with. Just make sure you check your system specs and know what your getting into. They all offer raid protection.

You might check this out: http://www.amazon.com/Micronet-MDN1000-G-force-MegaDisk-Ethernet/dp/B000VEJGXC
I'm using 2/ 500 gig. Sata drive OWC kit. They are not raided but I've had no problems with this over the last 1 year or more with my MacBook Pro. The G Force was recommended by a good friend and editor, so, I thought I'd pass it on. I don't have it so can't comment on it's stability.

I've been told that you can edit HD with fw 800 drives, but a friend strongly advised RAIDED SATA drives - - this would mean eSATA for me I guess.

Are you planning to edit XDCAM native, or convert to another format? I'm not convinced there's a significant advantage to an eSATA RAID with XDCAM, especially if you're editing on a Macbook Pro. 35mb/s is less than 5Mb/second - the latest 1GB SATA drives are sustaining 75-100Mb/s read/write speeds (http://www.barefeats.com/hard94.html). Thus a single one of those drives in a fw800 enclosure should easily support 10+ streams of XDCAM, which is more than the processor in the Macbook can decode smoothly.

If you're converting to ProRes it makes more sense, but even then it's not absolutely necessary. At the lower data rate, 145mb/s or ~18Mb/s you should still be able to comfortably run 3-4 streams from a single drive, and at the higher data rate (220mb/s or ~27Mb/s) at least 2 streams.

Are you planning to edit XDCAM native, or convert to another format? I'm not convinced there's a significant advantage to an eSATA RAID with XDCAM, especially if you're editing on a Macbook Pro

Hi Evan, and Mark and Dean - -

Thanks so much for getting back to me. I understand what you're saying, Mark and Dean... it seems there are a number of good drive-makers out there (I know G-Tech is good, but I did have a G-Tech fw drive fail on me a few months ago... which is why I'm now smitten with the idea of RAID).

Evan, I get a bit lost when it comes to your advice (my fault, not yours)... but I think I get the drift: eSATA might be more than I need (i.e., faster than what the MacBook Pro is capable of), but here's the thing - - I'd like to ”future-proof“ myself a little bit... e.g., if I get a Mac Pro six months from now (honestly, it's the Mac Pro I WANT to get for editing... I'm getting the MacBook Pro just for the sake of being able to download in the field; if there were an inexpensive way to download in the field, I'd use it, and do my editing on a Mac Pro back in my humble office), I don't want to have to upgrade my hard drive “array”.

I want to come up with a solid solution not just for now... but (if this is even possible) for the next few years.

I get lost re the number of ”streams“ of data I'll be running (oh, and re editing XDCAM native... does this mean the 35Mb/s stuff? If so, yes, I'd like to be able to, if I can... can I?)... but aside from that... given that I might be adding a Mac Pro to my equipment list in six or eight months time... does eSATA make sense? I don't know if it makes a difference (re your reference to ProRes) that I use Avid Xpress Pro as my main nle...
Thanks so much for your help on this,
Malcolm

One reason I mentioned G Force raid is that it's ethernet cable. As far as I know, someone correct me if I'm wrong, it's the fastest of all the connections.
http://www.answers.com/topic/g-force-raid

Are you planning to edit XDCAM native, or convert to another format? I'm not convinced there's a significant advantage to an eSATA RAID with XDCAM, especially if you're editing on a Macbook Pro. 35mb/s is less than 5Mb/second - the latest 1GB SATA drives are sustaining 75-100Mb/s read/write speeds (http://www.barefeats.com/hard94.html). Thus a single one of those drives in a fw800 enclosure should easily support 10+ streams of XDCAM

Are you sure about the streams? my understanding of hard drives, especially single ones, is that those sustained speeds are for sequential data only. that means if you are dealing with more streams at one time your drive will be seeking, bringing your 100MBps Hard drive down as low as 3MBps.

One reason I mentioned G Force raid is that it's ethernet cable. As far as I know, someone correct me if I'm wrong, it's the fastest of all the connections.
http://www.answers.com/topic/g-force-raid

what do you mean? gigabit ethernet? its about 2-4x slower than esata, and in most consumer NAS'es it is, in reality, about 10x slower.

personally id say, if you need a lot of speed for cheap, esata is absolutely the way to go. and if you can afford a RAID, it is absolutely worth it if you are doing a lot of editing. the extra speed, capacity, and security they provide (depending on the configuration) are extremely useful. the only problem is finding exactly the right one for your needs (extra redundancy, extra speed, portability, ease of use, low price, etc)

Are you sure about the streams? my understanding of hard drives, especially single ones, is that those sustained speeds are for sequential data only. that means if you are dealing with more streams at one time your drive will be seeking, bringing your 100MBps Hard drive down as low as 3MBps.

On the link I provided ( http://www.barefeats.com/hard94.html ) you can see the drop in speed for small random vs. large sequential - it is significant, cutting the read speeds into the 30MBs range - which is still plenty for multiple streams of XDCAM. My experience with HDV has been that for any single dual-core processor (i.e when editing on a Macbook) the bottleneck is the processor's ability to decode multiple streams rather than the drive's ability to stream them.

On an 8-core mac pro the situation would be reversed and there would be an advantage to an eSATA array if you are doing a lot of compositing with multiple streams of video. If you're mostly doing basic editing I don't know that you'll see a significant difference.

I'd like to ”future-proof“ myself a little bit... e.g., if I get a Mac Pro six months from now (honestly, it's the Mac Pro I WANT to get for editing... I'm getting the MacBook Pro just for the sake of being able to download in the field; if there were an inexpensive way to download in the field, I'd use it, and do my editing on a Mac Pro back in my humble office), I don't want to have to upgrade my hard drive “array”.

I want to come up with a solid solution not just for now... but (if this is even possible) for the next few years.

Absolutely, that makes sense, and as I mentioned above you would potentially see an advantage with a Mac Pro and eSATA. However, whenever it comes to future-proofing my advice is to be realistic about your time frame. Are you going to get a Mac Pro in six months? If not - if it's closer to a year or more - you should probably just worry about what you need right now, not what you'll need then. In a year you'll be able to get a faster array with greater capacity for less money.

oh, and re editing XDCAM native... does this mean the 35Mb/s stuff? If so, yes, I'd like to be able to, if I can... can I?)... but aside from that... given that I might be adding a Mac Pro to my equipment list in six or eight months time... does eSATA make sense? I don't know if it makes a difference (re your reference to ProRes) that I use Avid Xpress Pro as my main nle...

Yes, XDCAM native is 35 megabits/second or approximately 4.5 megabytes/second. In terms of drive performance that's probably the easiest format to edit in. I don't really know Avid - my understanding is DNxHD is comparable to ProRes in terms of data rate, but I don't know if Xpress Pro supports it.

I use the Sonnet Fusion F2 eSATA Raid drive. Its a 640GB RAID drive in a tiny enclosure (2 X 320GB drives in there) and takes power from your Firewire port on the laptop. It connects to the Macbook Pro via an expresscard 34 eSATA card - so my workflow is thus: shoot, then insert the SxS card into the Macbook Pro directly - copy the card to a portable LaCie rugged drive via Firewire 800 - this acts as my backup - then swap the SxS card for the eSATA card, connect the Fusion F2 - copy across the files from the rugged drive to the Fusion and start editing.

Its a great solution for me - means that I have an instant backup and I can edit in the field without the need for a separate power source.

Simon Cox

what do you mean? gigabit ethernet? its about 2-4x slower than esata, and in most consumer NAS'es it is, in reality, about 10x slower.

personally id say, if you need a lot of speed for cheap, esata is absolutely the way to go. and if you can afford a RAID, it is absolutely worth it if you are doing a lot of editing. the extra speed, capacity, and security they provide (depending on the configuration) are extremely useful. the only problem is finding exactly the right one for your needs (extra redundancy, extra speed, portability, ease of use, low price, etc)

Looks like eSata will transfer 6 gigs per second and ethernet 1 gig per second according to the literature! Of course, I wonder what people are really getting? I'm using a Mac Book Pro intel with a teribit eSata drives in a kit set-up and they are fast but ... don't know the exact speeds.

Evan - thank you for your thoughtful advice.
I understand now that, with a MacBook Pro, the eSATA speeds of a RAID array (like the HDOne) might be wasted... any fw (800 anyway) drive would be just as fast.
(this, I presume, would apply not just to basic editing, but also rendering effects and this sort of thing?)

But what about security (eg., RAID 5)? I had a firewire drive fail on me a few months ago (I lost three weeks of work), and swore that my next hd setup would be RAIDed so that this would never happen again.
If fw 800 is just as fast as RAID, what do people do for peace of mind... just make a habit of always transferring footage to two fw drives?

If the above is a perfectly good working solution, I might just get a few more fw 800 drives (I guess I could use my collection of fw400 drives as my ”backup/redundancy“ drives)... but if I DO end up going eSATA because of my plans to get a Mac Pro in the fall (or by December, anyway), I have two questions for Simon, or someone like Simon who uses eSATA... does swapping an SXS card for an eSATA card involve a re-boot?
And... do they make ExpressCard hubs... so you don't have to swap?

Thanks so much,
Malcolm

All hot swappable so no need to reboot between the SxS card and the Fusion F2 for me.

... but do unmount the volumes first; don't just yank the card!

George/

P.S. If you find any ExpressCard extenders (hubs), I'd like to know. I haven't turned up any.

Malcolm:

I have a 6 TB Caldigit HDPro for a few weeks now. I got my from B&H New York. It works great so far. HDOne is similar to HDPro but with with some cost saving measures. (1) HDOne does not have a removable RAID card. (2) The memory on the RAID card is not ungradlbe. (3) No redundant power supply (I think).

Caldigit is retiring its S2VR HD.

Mark

Malcolm:
I have a 6 TB Caldigit HDPro for a few weeks now. I got my from B&H New York. It works great so far. HDOne is similar to HDPro but with with some cost saving measures. (1) HDOne does not have a removable RAID card. (2) The memory on the RAID card is not ungradlbe. (3) No redundant power supply (I think).Mark

Hi Mark... this is great, to hear from someone who's got something very much like the HDOne. (by the way, George... if I do come across an ExpressCard hub, I'll let you know)... Mark - - if I do end up going for the HDOne, am I a fool for pinching pennies (actually, quite a few pennies) and not getting the HDPro? How important is a removeable RAID card (and how important is it to be able to upgrade the memory on the RAID card)? As you perhaps read from one of my previous posts, if I sink money into a RAID array, I want to make sure I get something that will last me quite a while.
Cheers,
Malcolm

Hi Malcolm:

From what I gathered from speaking with Caldigit supports, it really comes down to how the product can be supported. For example, if RAID card is bad then you can simply send the RAID card back instead of the entire unit. The chasis is very heavy even without any drive in it. About the cache memory, he told me unless you are doing 2K or your drive is about 75% full, you will likely not experience a significant impriovement over the standard 256 MB cache. Either way you will probably ends up spending +/- $1 per GB. HDOne is still 8-bay units so you will get the benefit from having multiple drive spindles. I configured my 6TB unit in RAID-5 with usable space just about 4.7TB. I've tried RAID-6 already. I think the performance penalty is just too much. With 8-drive unit, you can certainly spit the drive into different RAID array with different RAID configuration. I personally finds a big RAID-5 is sufficient.

When you buy a unit with drive already populated, the entire unit has 3 years warranty with a single point of contact. When you have problem with drive youc all Caldigit. When you have problem with RAID card you call Caldig hit. You get drive replacement from Caldigit and not from the hard drive manufacture. I find that a great value.

Now about using your own drive. Yes you can. Technically you can. But one of a big side effect is getting false positive on your RAID status or at worst multiple drive failure simply because the different hard drives don't work well together. This really has to do with different drive firmware and drive self-correction .... yada.. yada. I know this because I've worked in a big IT data centers. The same theory apply to big HP SAN or Dell PowerVault. And yes, take out the 750 GB drive out of Caldigit drive cartridge and put your own 1 TB drive does void the warranty on the 750 GB drive itself.

I say get the largest drive that your budget can allowed. If you work on different uncompressed videos from differernt projects, 6TB is not really out of this world. It gives you some decent breathing room and that is about it. The 8 TB unit to me is just simply cost prohibitive. The 4 TB unit probably has a very short usable life before I start looking for a replacement RAID again. So I picked a 6 TB unit.


IF, that is big IF, if you ever need to have multiple computers sharing one external RAID array, then Caldigit is the answer for you. Very shortly you will see a "switch" device from Calidigit. You can have multiple HDPro units or computers all connected to this "switch". All HDPro units will be visible to all computers at the same time.

I hope I have answered your question. If not then please let me know.


Thanks

Mark

But what about security (eg., RAID 5)? I had a firewire drive fail on me a few months ago (I lost three weeks of work), and swore that my next hd setup would be RAIDed so that this would never happen again.
If fw 800 is just as fast as RAID, what do people do for peace of mind... just make a habit of always transferring footage to two fw drives?

The choice isn't between RAID and fw800, it's between fw800 and eSATA - you could go with a 2-drive fw800 RAID1 enclosure for redundancy. If you want RAID 5 you're probably better off just getting an eSATA solution.

The choice isn't between RAID and fw800, it's between fw800 and eSATA - you could go with a 2-drive fw800 RAID1 enclosure for redundancy. If you want RAID 5 you're probably better off just getting an eSATA solution.

Now you have me wondering why I got it into my head that RAID 5 is what I want - - I'm sure I read this somewhere... that RAID 5 is both safe AND fast.
But from what you said earlier, maybe RAID 1 is just as fast (and just as safe)... at least when it comes to a MacBook Pro (THAT'S where things bottleneck). But maybe not quite as good, though, if I get a Mac Pro?
Thanks for your patience on this...
Malcolm

Malcolm:

RAID-5 is actually "slower" than RAID-1 and RAID-0. RAID-0 is the fastest but it offers no data protection. RAID-1 is fast but there is a 50% over head of loosing space to the mirroring.

Check out Sonnet Tech here:
http://www.sonnettech.com/product/storagesolutions/index.html

If you care, this is my HDPro pkg:
http://gallery.mac.com/huang.mark#100021

Good luck and happy shopping.

Mark

On the link I provided ( http://www.barefeats.com/hard94.html ) you can see the drop in speed for small random vs. large sequential - it is significant, cutting the read speeds into the 30MBs range - which is still plenty for multiple streams of XDCAM. My experience with HDV has been that for any single dual-core processor (i.e when editing on a Macbook) the bottleneck is the processor's ability to decode multiple streams rather than the drive's ability to stream them.

With regard to the barefeats graphs, the "small random write" test may not be as useful as you think. they dont say how small or how random and i believe it is only meant to compare the drives listed based on this arbitrary test. this may or may not correlate well with playing multiple simultaneous video streams from a single drive. i have seen certain data transaction patterns bring drives to only 3MBps, and booting to your OS likely happens at about 10MBps because there are so many small files for the system to load into RAM.

Looks like eSata will transfer 6 gigs per second and ethernet 1 gig per second according to the literature! Of course, I wonder what people are really getting? I'm using a Mac Book Pro intel with a teribit eSata drives in a kit set-up and they are fast but ... don't know the exact speeds.

I believe you mean you have a Terabyte drive, not Terabit.

I'm not sure 6 "gigs" per second* is an accurate number, maybe 2.5Gigabits per second at most. the theoretical max speed for esata using sata2 is 2.5Gbps and 1.25Gbps using sata1, and the theoretical max speed of Gigabit ethernet (GigE) is, well, 1 Gigabit per second. That works out to 300MBps for sata and 125MBps for GigE. however, in practice, youll likely not see much more than about 250MBps over sata (probably less if it is attached using a single pci-express lane which itself maxes out at 250MBps). and probably the only way youll see that kind of speed over a single esata connection, since hard drives run quite a bit slower than that, is with a port multiplied RAID or a RAID that has an internal controller and outputs over a single esata. GigE on the other hand in practice wont get you much more than 60-80MBps. and i believe you will have a hard time finding a NAS under $5k that breaks 20MBps.

*note the difference between bits (b) and Bytes (B)