Storage- How Are You Handling 5D files

Jim Newberry · October 24th, 2009, 01:28 PM

Yeah, it's important to have offsite backups. Hard drives and optical media (DVDs and CDs) are both imperfect; there's no great archival solution for digital files. There's a good argument for having your files on two different types of media, as they each have their own strengths and weaknesses. So my advice is one copy of each file on a hard drive, and another on a DVD or Blu-ray disc. And be sure to store one set offsite.

As for hard drives, I agree with the others who have suggested the hard drive docks, so you can buy multiple internal drives without buying multiple enclosures. Regarding DVD-Rs, Taiyo-Yuden media is highly rated, and not super expensive. Just be sure to verify your burns, don't burn at max. speed, and don't write on the disc (you can use an archival pen to write the disc number on the clear part around the disc's hole).

There might be a good argument for capturing to mini-dv tape as well (from computer).

Jim Newberry · October 24th, 2009, 01:29 PM

Quote:

Originally Posted by Evan Donn

Right now I'm just keeping my source & other project related files on a mirrored array and all the prores files on a raid0. Closing in on about 4TB total from this past year and I just figure I'll add another similar pair of drives to carry me through the next year. At a cost of just a few hundred dollars it seems the cheapest & easiest way to go.

Why mirrored?

Kalulu Ngilo · October 24th, 2009, 08:48 PM

it duplicates the same data from the original source. Two mirrors are better than one right? You break one, you can use the other to put your makeup on right? j/k

thanks.

Allan Tabilas · October 25th, 2009, 11:35 AM

I keep two copies of my source video on my main machine, and my backup file server. My file server is a Norco DS-4220 case, which is common with the HD movie enthusiasts on sites like AVSforum.com and Hardforums.com. Some are actually video editors, but many are movie guys with 40TB+ of space. It's a 4U rack mountable case, and fits 20 SATA hard drives via a SAS backplane. I put a Gigabyte AMD 785G motherboard and a low power Athlon 64 X2 5050e cpu.

The gigabyte motherboard has at least 6 SATA ports, and at least two PCIE x16 slots. So you canput in cheap Supermicro AOC-SASLP-MV8 cards, which takes two SFF8087 connectors, or basically 8 drives. The case has essentially 5 backplanes, so you get two of the Supermicro SASLP-MV8 cards, which takes care of 4 backplanes, or 16 drives. Then you get a SATA to SAS reverse breakout cable, which will plug into the 5th backplane, and your sata ports on the motherboard, adding to the total of 20 SATA drives.

Some good reviews of the case are here. You can get the case through newegg.com, or mwave.com through its ebay store for around $300. The AOC-SASLP-MV8 cards are around $110 through provantage.com (and also get SAS-SAS SFF8087 cables and SATA-SAS reverse breakout cables there too).

I have four 1.5TB drives, and three 2TB drives in there, with 13 SATA hard drive trays left for tons of expansion. What's interesting is that My HDV and AVCHD files (old Sony HDR-FX7 and Panasonic AG-HMC150, both around 21Mbits/sec in 1440x1080i or 1920x1080 PH mode) are a little smaller than my old 25Mbit DV files from a DVX100 and DVC30 (typically 12GB/hour). The 5D Mark II files are around 40Mbits/sec, or 5MB/sec (18GB/hour approximately), still much smaller than say a 100Mbit P2 DVCPRO HD files. A single 2TB drive can fit somewhere in the neighborhood of 100 hours of raw 5D Mark II h.264 MOV files, 160 hours of raw DV files, and between 180-200 hours of raw HDV and raw 21MBit/AVCHD files (rough rough estimates, don't quote me).

More links for the case.

Hands On: Norco RPC-4220 Rackmount Chassis | We Got Served - Windows Home Server & Your Digital Home

New Norco RPC-4220 Server Case pictures - AVS Forum

Data Storage Systems - [H]ard|Forum

Christian Majeres · October 25th, 2009, 01:25 PM

Quote:

Originally Posted by Chris Barcellos

Well for the next Terabyte, I bought a Vantec NexStar hard drive dock, for $40, and then bought an Hitachi terabyte drive for $70, both at Frys. The Dock works like a popping the hard drive in like a toaster. It all hooks into the SATA set up on my mother board. Should be just like an internal drive, except you can change drive easily by just ejecting one and replacing with another. I will be installing this weekend.

See picture below.

i use a similar device... as for storage i still have a few 700giggers lying around... makes backup easy. I copy the files to 2 HD drives tho, so that i still have the data if one of the HDs fail.

As for Allans fileserver: i have a very similar setup, with a Chenbro 3U (16xS-ATA) on a fancy hardware RAID6 controller. Bad part about that experience was that my RAID controller decided to die and take my array with it. So if you go for big fileservers, my recommendation is that you do make copies somewhere else (just like Allan does)... RAID servers are only for uptime, never for data security :)

Allan Tabilas · October 26th, 2009, 04:59 PM

A expensive $500 hardware raid card is nice because you have redundancy and can create very large partitions (10TB+). The complexity of setup and configuration can be difficult. The cost of the hardware raid cards (decent ones with dual core Intel IOP cards are well over $500 like Areca and Adaptec) plus the loss of capacity due to parity is pretty high.

The Supermicro SAS cards are around $100, and you need only two or three of them in this 4U Norco case. They are cheap enough, and with the money saved over true Hardware RAID, some people have essentially two Norco cases, one backing up the other one (40 disks max, but in effect almost RAID 1 without the synchronization realtime).

Evan Donn · October 27th, 2009, 12:17 AM

Quote:

Originally Posted by Jim Newberry

Why mirrored?

Because I don't trust a single drive for mid-term (next few years) storage of source files, and don't want to take the time to burn everything to optical. Raid1 is simple & cheap and doesn't require any additional work once it's set up. I'm sure optical is a better choice for long term storage, but I'm also sure that our capacity & format choices will be significantly different in 5 years then they are now so - that's the archival window I'm most concerned with at this point.

Jim Newberry · October 27th, 2009, 05:36 PM

I think it's a great idea to not rely on a single hard drive for storage, but RAID only without offsite storage is risky. If the RAID device gets stolen or damaged from a flood/fire/power surge or corruption, you've lost your files. To me it makes more sense to have one copy in the office, and another offsite, preferably on two types of media (optical/hard drive/tape).

Peer Landa · October 27th, 2009, 10:57 PM

Quote:

Originally Posted by Jim Newberry

I think it's a great idea to not rely on a single hard drive for storage, but RAID only without offsite storage is risky. If the RAID device gets stolen or damaged from a flood/fire/power surge or corruption, you've lost your files.

That's why I try to always keep an updated redundant hard drive backup in my car -- don't think it's very likely my car gets stolen the very same time my office burns down.

-- peer

Peter Burke · October 28th, 2009, 04:04 PM

Quote:

Originally Posted by Mike Williams

I back up "critical" footage to DVD, DVD DL, and BluRay disks. ... I heard long term is not a good idea on HDD. ...

Corporate policy where I work forbids archive back up to optical discs. CDs and DVDs have proven to be far too unreliable.

I have experienced many corrupted 5 to 10 year old discs - and one was faulty the very next day !!! - even though it tested ok the previous night !!! (was a cheap brand). A computer guru mate ran a data recovery app that continuosly attempted to read the lost files for 48 hours, and approx. 60% of the data was recovered.

Do I trust optical media ? - NO. The dyes degrade over time.

Christian Majeres · October 28th, 2009, 05:02 PM

Quote:

Originally Posted by Peter Burke

Do I trust optical media ? - NO. The dyes degrade over time.

I wholeheartedly agree. Optical has been bad to me, even when using quality dvd-r's.

As stated above, HD's don't strike me as safe for long-term storage either. The problem are the disk head readers, if they dont start up correctly, the HD can't be read anymore.

This whole situation will change when SSD is getting cheaper. Heck, i have to admit, i'm already contemplating getting 64gbyte USB sticks, or some of the cheaper SSD "harddrives" for really reliable long-term backups... as flash-based storage degrades only on writes, not on reads.

But for now, price-wise, tape is still interesting for long-term storage... LTO-2 or LTO-3, depending on your data amounts. But even those need recopying to new tapes every 5 years or so. Keeping 2 copies can be handy... and the tape drives can be finicky.

Pick your poison :)

Bill Binder · October 28th, 2009, 06:09 PM

I wouldn't necessarily hold up SSD as the answer. Check out this recent SlashDot story:
Slashdot Ask Slashdot Story | Reliability of PC Flash SSDs?

Beyond that, my motto is more backups, at least two if not three, and then to migrate forward with technology as it happens. It always seems that right when I'm about to burst my seems, the latest large drive comes out, LOL. I'm kind of praying for some insane 10T drives to drop in the next few years where I'll just migrate off my old three copies onto new ones over time.

Biggest thing in my book is don't rely on any one copy, and make sure something is off site, and check your backups on occasion.

BTW, I personally agree that tape is awesome, especially when verified, but it's too expensive and too much of a PITA for the solo hobbiest. If I made my living on this, I would seriously consider going that route myself.

Jim Newberry · October 28th, 2009, 09:09 PM

Quote:

Originally Posted by Peter Burke

Corporate policy where I work forbids archive back up to optical discs. CDs and DVDs have proven to be far too unreliable.

I have experienced many corrupted 5 to 10 year old discs - and one was faulty the very next day !!! - even though it tested ok the previous night !!! (was a cheap brand). A computer guru mate ran a data recovery app that continuosly attempted to read the lost files for 48 hours, and approx. 60% of the data was recovered.

Do I trust optical media ? - NO. The dyes degrade over time.

There's no digital media that has proven to be completely reliable for very long periods of time. But I'm not sure that optical is much worse (if at all) than hard drives. I make my living as a still photographer shooting mainly digital. I have all image files on both hard drives and DVD-Rs (I just recently switched to Blu-ray). I have over 450 DVD-Rs burned with digital images. I've never had a single one fail (though I'm sure I will eventually), probably at least in part due to the fact that I take these precautions:

- use high quality media (authentic Taiyo-Yuden)
- burn at less than fastest speed
- don't write anywhere on disc (only center clear part with cd safe pen)
- don't store discs in sleeves--that can cause scratching. store them in DVD cases (I use slim double cases to avoid taking up too much space)

Hard drives are great, but eventually nearly every one will crash or get corrupted. Plus you can get a virus on hard drive or accidentally erase data; this can't happen on an optical disk (unless of course there's already a virus present before burning). I think the key is to have at least two backups on at least two different media types, with at least one stored offsite.

There's a good book and forum on digital asset management here:
The DAM Forum - Index

Dan Brockett · October 29th, 2009, 03:54 PM

Quote:

Originally Posted by Tom Daigon

My approach is to convert H.264 files with MPEG Streamclip to Prores 422 then store those on a double sided dvd.I then assign a volume name and number to the dvd and log basic info into a database. This gives me 8 gigs of storage. For Larger projects Blu-ray would be a better way to do it.

Hi Tom:

I have done the same but you should know that almost all optical media is prone to digital rot. I used to work for Circuit City back in the 1980s at the dawn of digital consumer audio, namely CDs. I own thousands of them. Approximately 5% of my collection has become oxidized, meaning that they basically rusted. They fade, then turn lighter orange, then eventually turn a deep red/orange and are unreadable. Some of these CDs were as recent as the late 1990s, some were made way back in the late 1980s.

DVDs are the same, except I have not seen the same oxidation, I have seen literal rot, where you hold the DVD up to a strong light source and you see holes in the substrate. The only optical media category I have not experienced digital rot or oxidation on are the Mobile Fidelity 24k gold CDs.

Hard drives are just as bad, I own a total of 168 hard drives. Most date from the dawn of digital consumer video with DV from late 1990s. I boot them on a regular basis to spin and lubricate the platter bearings and they are stored in a dark, climate controlled environment. On my last round of exercising, out of an attempt to boot 30 drives, 8 of them would not boot. I think the data is still on the platters, but the drives could only be rescued by a data recovery service, which costs a fortune.

I have had the least amount of long term storage degradation from tape. I still regularly watch VHS video tapes I recorded back in the 1980s. They have degraded since then but at least they will still play. I also have a lot of tapes from orphan formats like Hi-8, U-Matic, U-Matic SP and Beta SP, most of which still play decently.

I don't have any solution for you, but just beware that all of us are deluding ourselves if we think that any of the content we are generating today will be playable/recoverable in as little as 10 years from now. I sincerely believe that most of us in our old age will not have access to much media from our lives unless it was shot on film or possibly archived to a tape format like LTO or DLT, etc. And that is a big maybe since there will not be machines around to decode the data off of said tapes, even though the tapes may be intact.

Short term storage, we are fine but long term storage is not looking promising at all.

Dan

Don Hauerken · November 17th, 2009, 01:45 AM

For a few years, in addition to keeping two copies on two separate hard disk drives, I also burned to two DVDs. Then I discontinued the HDD copies and went with two burned DVDs of every file. When I got up to about 1300 DVDs I realized that migrating that data from mollasses-slow discs to the next generation of storage media (not to mention accessing files quickly and easily) would be a nightmare. Reading from disc takes forever and forever... copying them all to the next media format would be hellish feeding of discs and waiting and waiting. In the meantime, large capacity HDDs got fairly inexpensive.

So now I've ditched the disc copies and I'm back to using HDDs (mostly 1TB) with two onsite copies and an offsite copy (or two!). The desktop toaster docks for running raw 3.5" SATA drives are awesome!

And when it comes time to transfer the data to another drive or the next storage media format, HDDs are a easy to connect and to read from. And day to day access and searching is so quick and easy.

It is KEY, however, as others have mentioned, to expect that all media/devices will fail sooner or later, and you have no idea when, so redundancy is vitally important. As is on and off-site copies of anything you don't want to lose. And smart to run all of the drives a little, now and then, to try to maintain the mechanisms.

Last year I decided not to buy any drives smaller than 1TB. Now I am starting to buy some 1.5TB and a couple of 2TB drives. But I still think 1TB drives are a good balance between capacity and risk.

October 24th, 2009, 08:48 PM	#18
Kalulu Ngilo Regular Crew Join Date: Jan 2007 Location: Riverside, California Posts: 112	it duplicates the same data from the original source. Two mirrors are better than one right? You break one, you can use the other to put your makeup on right? j/k thanks. __________________ dcrin3.com we don't know alot of genres, at least as of now.

October 25th, 2009, 11:35 AM	#19
Allan Tabilas Regular Crew Join Date: Jan 2005 Location: Philadelphia, PA Posts: 81	I keep two copies of my source video on my main machine, and my backup file server. My file server is a Norco DS-4220 case, which is common with the HD movie enthusiasts on sites like AVSforum.com and Hardforums.com. Some are actually video editors, but many are movie guys with 40TB+ of space. It's a 4U rack mountable case, and fits 20 SATA hard drives via a SAS backplane. I put a Gigabyte AMD 785G motherboard and a low power Athlon 64 X2 5050e cpu. The gigabyte motherboard has at least 6 SATA ports, and at least two PCIE x16 slots. So you canput in cheap Supermicro AOC-SASLP-MV8 cards, which takes two SFF8087 connectors, or basically 8 drives. The case has essentially 5 backplanes, so you get two of the Supermicro SASLP-MV8 cards, which takes care of 4 backplanes, or 16 drives. Then you get a SATA to SAS reverse breakout cable, which will plug into the 5th backplane, and your sata ports on the motherboard, adding to the total of 20 SATA drives. Some good reviews of the case are here. You can get the case through newegg.com, or mwave.com through its ebay store for around $300. The AOC-SASLP-MV8 cards are around $110 through provantage.com (and also get SAS-SAS SFF8087 cables and SATA-SAS reverse breakout cables there too). I have four 1.5TB drives, and three 2TB drives in there, with 13 SATA hard drive trays left for tons of expansion. What's interesting is that My HDV and AVCHD files (old Sony HDR-FX7 and Panasonic AG-HMC150, both around 21Mbits/sec in 1440x1080i or 1920x1080 PH mode) are a little smaller than my old 25Mbit DV files from a DVX100 and DVC30 (typically 12GB/hour). The 5D Mark II files are around 40Mbits/sec, or 5MB/sec (18GB/hour approximately), still much smaller than say a 100Mbit P2 DVCPRO HD files. A single 2TB drive can fit somewhere in the neighborhood of 100 hours of raw 5D Mark II h.264 MOV files, 160 hours of raw DV files, and between 180-200 hours of raw HDV and raw 21MBit/AVCHD files (rough rough estimates, don't quote me). More links for the case. Hands On: Norco RPC-4220 Rackmount Chassis \| We Got Served - Windows Home Server & Your Digital Home New Norco RPC-4220 Server Case pictures - AVS Forum Data Storage Systems - [H]ard\|Forum Attached Thumbnails

October 24th, 2009, 01:28 PM	#16
Jim Newberry Regular Crew Join Date: May 2007 Location: Los Angeles, California Posts: 170	Yeah, it's important to have offsite backups. Hard drives and optical media (DVDs and CDs) are both imperfect; there's no great archival solution for digital files. There's a good argument for having your files on two different types of media, as they each have their own strengths and weaknesses. So my advice is one copy of each file on a hard drive, and another on a DVD or Blu-ray disc. And be sure to store one set offsite. As for hard drives, I agree with the others who have suggested the hard drive docks, so you can buy multiple internal drives without buying multiple enclosures. Regarding DVD-Rs, Taiyo-Yuden media is highly rated, and not super expensive. Just be sure to verify your burns, don't burn at max. speed, and don't write on the disc (you can use an archival pen to write the disc number on the clear part around the disc's hole). There might be a good argument for capturing to mini-dv tape as well (from computer).

October 26th, 2009, 04:59 PM	#21
Allan Tabilas Regular Crew Join Date: Jan 2005 Location: Philadelphia, PA Posts: 81	A expensive $500 hardware raid card is nice because you have redundancy and can create very large partitions (10TB+). The complexity of setup and configuration can be difficult. The cost of the hardware raid cards (decent ones with dual core Intel IOP cards are well over $500 like Areca and Adaptec) plus the loss of capacity due to parity is pretty high. The Supermicro SAS cards are around $100, and you need only two or three of them in this 4U Norco case. They are cheap enough, and with the money saved over true Hardware RAID, some people have essentially two Norco cases, one backing up the other one (40 disks max, but in effect almost RAID 1 without the synchronization realtime).

October 27th, 2009, 05:36 PM	#23
Jim Newberry Regular Crew Join Date: May 2007 Location: Los Angeles, California Posts: 170	I think it's a great idea to not rely on a single hard drive for storage, but RAID only without offsite storage is risky. If the RAID device gets stolen or damaged from a flood/fire/power surge or corruption, you've lost your files. To me it makes more sense to have one copy in the office, and another offsite, preferably on two types of media (optical/hard drive/tape).

October 28th, 2009, 06:09 PM	#27
Bill Binder Major Player Join Date: Jun 2005 Location: San Francisco Posts: 542	I wouldn't necessarily hold up SSD as the answer. Check out this recent SlashDot story: Slashdot Ask Slashdot Story \| Reliability of PC Flash SSDs? Beyond that, my motto is more backups, at least two if not three, and then to migrate forward with technology as it happens. It always seems that right when I'm about to burst my seems, the latest large drive comes out, LOL. I'm kind of praying for some insane 10T drives to drop in the next few years where I'll just migrate off my old three copies onto new ones over time. Biggest thing in my book is don't rely on any one copy, and make sure something is off site, and check your backups on occasion. BTW, I personally agree that tape is awesome, especially when verified, but it's too expensive and too much of a PITA for the solo hobbiest. If I made my living on this, I would seriously consider going that route myself.

November 17th, 2009, 01:45 AM	#30
Don Hauerken New Boot Join Date: Nov 2009 Location: Vancouver, WA USA Posts: 12	For a few years, in addition to keeping two copies on two separate hard disk drives, I also burned to two DVDs. Then I discontinued the HDD copies and went with two burned DVDs of every file. When I got up to about 1300 DVDs I realized that migrating that data from mollasses-slow discs to the next generation of storage media (not to mention accessing files quickly and easily) would be a nightmare. Reading from disc takes forever and forever... copying them all to the next media format would be hellish feeding of discs and waiting and waiting. In the meantime, large capacity HDDs got fairly inexpensive. So now I've ditched the disc copies and I'm back to using HDDs (mostly 1TB) with two onsite copies and an offsite copy (or two!). The desktop toaster docks for running raw 3.5" SATA drives are awesome! And when it comes time to transfer the data to another drive or the next storage media format, HDDs are a easy to connect and to read from. And day to day access and searching is so quick and easy. It is KEY, however, as others have mentioned, to expect that all media/devices will fail sooner or later, and you have no idea when, so redundancy is vitally important. As is on and off-site copies of anything you don't want to lose. And smart to run all of the drives a little, now and then, to try to maintain the mechanisms. Last year I decided not to buy any drives smaller than 1TB. Now I am starting to buy some 1.5TB and a couple of 2TB drives. But I still think 1TB drives are a good balance between capacity and risk.