What's your hard drive storage solution?

[Mike Watson]

I have a 12TB Raid ("Client Projects") and a 13TB backup ("Client Projects Backup"), one housed in an OWC 4 bay (with 4 - 3TB drives) and backup housed in an off-brand 4-bay.

I keep a folder for each client. In each folder is a series of years. In each year is various projects for that year. Many of my clients use footage from some years ago even in current projects.

At the end of each year, I buy a couple of 3TB bare drives, offload whatever I don't think I'm going to use again, and put them in a stack in my storage closet. I dupe them, so there is one "original" and one "backup" for each year. If I need something from this stuff, I just pull it from the closet and put it in a bare drive reader.

The whole system works pretty well.

Two weaknesses I want to address:

1) The RAID thing is touch and go. I have probably 15 years working with RAIDs now, and I have seen (many times) whole RAIDs crap out for no particular reason. Back when the biggest drive you could get was 250GB, you pretty much had no choice to have a RAID or three, but now that 4, 5, 6, even 8TB single drives are available, I'd rather get back to a single drive solution, even if it meant two 6TB single drives, split into A-L and M-Z client names or something.

2) Two backups isn't enough for me... I'd like to have some kind of tertiary off-site backup and swap them out every week or so. Especially with RAIDs, where a mouse fart can corrupt a whole RAID, I sweat bullets every time the RAID craps out and I have to restore from backup... because at that moment, I only have one copy of basically everything I've ever shot.

I'm curious to know what everyone else is using, and if your system is better than mine.

[Mike Watson]

Is there no solution to this? Or is everyone as stumped as I am?

[Chris Soucy]

Hi Mike...................

I didn't respond before as your particular application is not up my street, however, some suggestions to mull over.

I take it from your comment re raid that you are running a completely striped system,where, indeed, the dreaded mouse fart can drop it on its head in a blink.

OK, a bit of history on my part. I've had about 6 different systems over the years and the one thing that has taken them all out (bar the last) was hard drive failure.

On my previous system build I was determined this was never going to happen again, so had it built with 4 X Hitachi drives in a Raid 10 configuration which gave me a pretty stonky C drive for the time - 1TB from memory. Hey, this was 12 years ago!

Guess what? Not one terminal system failure in 11 years. Then I lost the first of my hard drives, had it replaced and the array rebuilt. Then I lost a second 2 months later, no big issue as the system soldiered on even with a disk failed.

By this time my original XP system was looking pretty sad and a complete rebuild was on the cards, so a rebuild it was.

This time it was rebuilt (absolutely everything except the case, a massive Coolermaster CM Stacker) with 2TB WD Blacks, again in Raid 10.

SO, what am I getting at?

Consider changing you striped arrays for Raid 10's but using BIG drives, money seeming no object, which means no single drive failure will bring the system down.

Raid 10 your backup system as well, double protection.

Benefits:

1. You have double security as two discreet disk systems have Raid 10 protection, thus a drive failure in either will not bring down the array nor lose data.

2. Using 6 or 8TB drives will really allow that Sata 3 interface to come into its own, the higher the packing density the more effective Sata 3 is.

3. Further backup to a suitable "cloud" provider, off site, though as the volumes will be small compared to your work load not unreasonably expensive.

Downside:

1. Because you'll be using Raid 10, even with 4 X 8TB drives in each box you'll probably only end up with 14 TB usable in each one. Another box or two may be in order.

Best I can do in such short order.

Regards,


CS

[Mike Watson]

Appreciate your comments Chris, and I will take it all under consideration. Indeed, it's always the drive failure that concerns me.

What I have right now are two 4-bay RAIDS, and what I'm considering is switching them to JBOD instead of RAID... then replacing the 3TB drives with 6TB drives, and having eight discreet drives - four for "primary" and four for backup. How to separate the clients, I'm not sure, but I'll find a way. At least then the death of one drive would only be the death of that one, and I wouldn't have to bite my nails not to lose another... because the loss of any two out of the 8 when configured as RAID 0 would be a 100% data loss.

[Chris Soucy]

Further musings on the subject................

I don't know anything about JBOD compared to Raid so can't comment, but on the subject of your 2 X 4 drive "boxes" a supposition: presumably each has internal Raid built in as standard?

If so AND your system Raid controller can do it? you COULD (money no object)

1. Buy a third box with "hot swap" ability and separate UPS, load it with 4 X 6TB drives in Raid 0 then stream your current "primary" box to it.

2. Replace the hard drives in your primary box with 4 X 6 Tb drives in Raid 0 and stream the contents of box 3 back to box 1 in Raid 1 (hang on to those 4 X 3TB "primary" drives BTW, just in case).

You now have two independent 20 odd TB mirrored boxes giving you the ability to survive a total box failure (blown power supply- it happens) and more.

What you do with your second / backup box is up to you.

A 4 X 6TB Raid 0 is gonna be pretty damn quick, two of them even faster, for if the controllers are up to it the array that gets the data first, gets it first.

There's nothing I enjoy more than spending other peoples money!


CS

[Mike Watson]

I read the first part of your reply and thought... man, this guy knows how to spend my money like he was my wife!! Then got to the end of the post and at least you own up to it! :-)

If I do a third box it'll be JBOD, but then how to backup onto that box is the million dollar question. It'd be a manual system and that's a whole can of worms.

[Chris Soucy]

Mike..............

I know you know this so this is aimed more at anyone else following this thread than yourself.

The bottom line here is this question and your answers to it:

Is my livelihood financially dependent on either total data protection or "work through failure"?

If the answer to either option is "Yes" then duplicating your data in a Raid 10 or similar is not a frivolous luxury but a fundamental business imperative, tax deductible and should have been done yesterday.

If the answer to BOTH is no you can afford to do it the cheap but hard way no matter how much hair you extract in the process.

As for backing up completed projects it may make it easier is you adopt a stringent "Job No." approach, where you create a new directory for each job and make sure all files pertaining to same are retained within that directory, no matter where on your vast array of drives they are physically located.

Backing up then becomes as simple as copying the entire directory and it's contents to your Raid 10 backup system then deleting it from your main system if the space is required.

It probably won't work out quite that simply but you get my drift.

I shall leave you to ponder.

Regards,

CS