Archiving EX files

[Alex Raskin]

I own a WD Book of 2TB capacity, and it has 2 built-in drives (supposedly user-replaceable), and an internal controller that can be configured as either Raid0 or Raid1 (with 1TB total capacity for latter.)

It has USB, FW, and eSata connectivity, and is very small in size.

Comparing to Drobo, what would be disadvantages of this solution, if any?

[Mike Chandler]

I wonder, Keith, why you feel the need for an online solution at all? Isn't the magnetism loss over time the same whether you're using the drives or not? I would think that except for the time that you're adding files to the drive, isn't it better to leave it alone? As for blu-ray video decay, is the risk the same if you're just archiving your BPAV files? It's not a video blu-ray, just a data one. As for a raid array, I'm thinking it would be just as easy to get a couple of raid drives, one backing up the other. That gives you the performance for editing of Raid 0, and the redundancy of raid 1 mirroring. Once I run out of space, just get another pair.

Here's my archival/work storage thinking so far:

Get a couple of mini raid drives for location; that improves copying time by having a couple raid drives in one enclosure, and peace of mind having the second unit mirrored. Minis keep the weight down.

Do the same for editing, except they don't have to be minis. When they're full, get another pair.

Dump the minis onto blu-ray to back up dailies and onto a single (shelved) hard drive as well, just to be safe (& neurotic--that way two different storage media need to fail in two different ways. ;) . Maybe the blu-ray (as opposed to just HDD) is giving me comfort in a retro way: it will be nice to actually have something real to pick up off the shelf, like a tape! I also like that it's discrete: the first shoot on the first half-dozen (or so) disks, the next shoot on the next half-dozen, etc.

[Keith Moreau]

Originally Posted by Mike Chandler

I wonder, Keith, why you feel the need for an online solution at all? Isn't the magnetism loss over time the same whether you're using the drives or not? I would think that except for the time that you're adding files to the drive, isn't it better to leave it alone? As for blu-ray video decay, is the risk the same if you're just archiving your BPAV files? It's not a video blu-ray, just a data one.

The magnetism loss issue probably does happen with an online drive, although online drives do periodically check data loss and rewrite a marginal sector if found. The act of reading data on a drive initiates this. Also an online solution such as a Drobo will alert you (via email or other means) if the drive goes bad. A drive on the shelf can't alert you.

Blue Ray: It's not video decay, it's physical aluminum substrate on an optical disc that have a chance of becoming unreadable over the very long term (5 years or more). This is an industry secret but you can look it up. The truth is most affordable optical storage isn't 100% reliable over time. It, however is probably on of the better 'shelf' solutions available, as it's not susceptible to mechanical failure or erasure from a magnetic field.

No solution is going to be 100%, its a matter of economy vs convenience vs safety. It seems some people think an offline drive has a better chance of staying intact than an online because an online is just used more, reducing it's effective life. I suggest that an online solution can alert you if things go bad and is safer. I don't say throw away the offline archive solution, just realize it's not 100% reliable. If I had unlimited resources and a full time tech to monitor it I'd back everything up to Blue Ray Gold disks, plus have RAID 1+0 s that have health monitoring on all the time and periodically run a script the does a read of every bit of data on these drives. Mike, this is similar to what you have, and I like it. For me, right now, this isn't practical so I currently have a bunch of Raid 0 drives backed up to on the shelf drives but I'm transitioning to Drobos backed up to on the shelf drives. My online work stays on Raid 0 for speed. The inherent (theoretical) safety of the Drobo solution means I can be a bit lax on my off-the shelf backups.

I'm not pretending to be the be-all end all authority on this issue, but through a couple of decades of experience and experiencing the pain of data loss, when it happens you wish you had archived in the best way possible.

Part of these discussions is to learn new things and perhaps better ways of doing things. I'm open to better solutions that offer the a better combination of speed, safety, ease of use and economy so please keep the comments, challenges and questions coming!

[Alex Raskin]

Keith, question: what kind of HD video do you work with?

I use Cineform Prospect HD at High, and I'm able to edit up to 4-5 streams off of my RAID1 system.

[Keith Moreau]

Originally Posted by Alex Raskin

I own a WD Book of 2TB capacity, and it has 2 built-in drives (supposedly user-replaceable), and an internal controller that can be configured as either Raid0 or Raid1 (with 1TB total capacity for latter.)

It has USB, FW, and eSata connectivity, and is very small in size.

Comparing to Drobo, what would be disadvantages of this solution, if any?

Hi Alex

I think it's a pretty good deal if you just need 1 TB and a small, self-contained solution. I considered inexpensive solutions like this.

Here's a small analysis of the cost: you'd need 3 of these for a cost of about $1,000 to equal one 4TB Drobo. Drobo is about $500, 4 x 1 TB Drives another $600. Total $1,100.

That's 930GB once formatted and you'll run out of space pretty quickly archiving the EX1. It's not that fast in mirrored mode (but neither is the Drobo), replacing a drive if it does go bad is not as easy as the Drobo, I don't know about what monitoring software it has if a drive goes bad... will it flash red or something, what if you have it in a closet, will it email you if the drive goes bad?

It's for these reasons that I decided on the Drobo solution as a longer term solution. I was getting tired of buying small RAID drives and piling them up one after another. But if your storage needs are modest I don't necessarily think it's bad. You may want to buy 2 of these, keep them in Raid 0, then you have the benefits of a large capacity, Firewire 800 / eSata-worthy speed, and a good backup in case one goes bad. It's not a mirrored solution so it's only backed up as of your last daily backup, but not terrible.

The thing that's also kind of appealing about the Drobo is you upgrade the drives pretty easily. As drives become high capacity (I have 4 x 1TB drives in mine, but 1.5 TB drives are becoming more common and affordable) you can take out the old drives, use them for 'shelf' archive, and pop in a newer higher capacity drive. I anticipate that will happen to me about once a year or so.

I hope I don't seem like a shill for Drobo. I deliberated a bit before getting one, and had a couple little issues with it when I first got it, but I just like the idea, the smartness of the software, and the simplicity.

Maybe in 5 years hard drives will be relegated to the museum and we'll have flash or other types of cheap, stable archival media that will make all this discussion moot.

[Keith Moreau]

Originally Posted by Alex Raskin

Keith, question: what kind of HD video do you work with?

I use Cineform Prospect HD at High, and I'm able to edit up to 4-5 streams off of my RAID1 system.

Alex, I edit with HDV, Prores 422 or XDCAM ex. Much lower bandwidth requirements than your Cineform I believe (25, 35, 145 mbit/second). I usually work online with my RAID 0 inside my Mac Pro. I keep other stuff on a variety of RAID 0 drives on a Gigabit ethernet NAS. The NAS can get me 70 MB/sec transfer rates. So I'm a Mac and you're a PC I assume?

I wouldn't use the Drobo for online video editing unless in a pinch, if that's what you're asking. I've clocked about 20- 30MB second, which theoretically would work but would result in some random delays, I think. Haven't actually done much real editing like that though.

[Alex Raskin]

No, actually I was just wondering why do you need RAID0 for online edits at all.

Last time I needed RAID0 was when I was capturing *uncompressed* HD 1920x1080 live.

With Cineform, there's simply no need for RAID0 from speed viewpoint.

So my production system (yes, a PC - will you make me into a stereotype now? ;) has internal RAID10 (essentially RAID1 x 2, for double the capacity) and works fine.

I think controller card is extremely important, so I had to shell out for Areca (after not so good experience with others), and am very happy with it so far.

Of course all my PCs are powered via UPSs (uninterruptible power supplies), so power loss/brownouts/blackouts/spikes so not affect my data.

[Keith Moreau]

Originally Posted by Alex Raskin

No, actually I was just wondering why do you need RAID0 for online edits at all.

Last time I needed RAID0 was when I was capturing *uncompressed* HD 1920x1080 live.

With Cineform, there's simply no need for RAID0 from speed viewpoint.

So my production system (yes, a PC - will you make me into a stereotype now? ;) has internal RAID10 (essentially RAID1 x 2, for double the capacity) and works fine.

I think controller card is extremely important, so I had to shell out for Areca (after not so good experience with others), and am very happy with it so far.

Of course all my PCs are powered via UPSs (uninterruptible power supplies), so power loss/brownouts/blackouts/spikes so not affect my data.

I use RAID 0 not because I need it necessarily but because it's fast. I guess I like to have the fastest available system for online editing. I do a lot of disk intensive work, effects, filters and such, and the RAID 0 seems to make a difference. Although SATA is theoretically fast, the disk is the bottleneck. RAID 0 makes throughput faster. Maybe that's unwise.

Sounds like you have a great reliable system, even though it's a PC :)

[Jay Gladwell]

I think this thread has gotten off track.

[Mike Chandler]

Thanks, Keith, for your reasoning.

Originally Posted by Alex Raskin

No, actually I was just wondering why do you need RAID0 for online edits at all.

Alex, editing in FCP with XDCam footage the speed difference is not insignificant. I did a comparison with a raid 0 (two drives in one enclosure) and non-raid drive and was getting about a 30% increase in speed. Transferring a 13.5 minute clip via XDCAM transfer (that's the program's default for keeping files under 4gb) took 4 mins with the non-raid and 2.75 with it.

Don't think the thread's gotten off-track, Jay, as it's very hard to separate out one's online system with one's archiving system, since in many cases they overlap.

[Alex Raskin]

Originally Posted by Mike Chandler

it's very hard to separate out one's online system with one's archiving system, since in many cases they overlap.

In my case they're the same :) *

As for the speed, yes, I actually use RAID10 which is two RAID1's coupled into RAID0. Makes it faster than just RAID1.

I agree that of course file transfers are much faster on RAID0-based systems.

------------------------
* Well, almost. I have two 2TB RAID10 arrays... one production, which I use for current projects. Another to dump the *finished* project on for archiving. This way archiving system does not get taxed as much as production one by constant HDD use, and I feel this is safer for storage.