Safe and Long Term Backup Solutions at DVinfo.net

Bill Vincent · October 8th, 2009, 06:09 AM

I threw out this question at a WEVA panel of experts - the question was, what is a failsafe method for backing up our project data in the long-term? Their answer was multiple sources of backup. While I concur, using different media (drives, cards and/or tape backup) can be very expensive.

My question today is whether RAID technology is the way to go - since it is a redundant system if done properly, and if one drive fails in the array, you can still recover the data from the redundancy on the other drives. It seems to me that if the RAID array is properly stored and maintained that it could theoretically provide safe and long-term backup storage.

Anyone doing this? If not, what long-term solution are you using for backing up your HD projects?

Jim Andrada · October 9th, 2009, 02:48 PM

Generally hard drives (RAID or not) are not designed as long term shelf storage devices. Tape (well, Linear tape anyhow) is designed for long term shelf life. Things like using special coatings on the backside of the tape so it won't adhere to the front side when tightly wrapped on the reel and stored for years and years.

However, Backup and Archive are really two different animals. Backup is to protect you in case of failure - including operator error, which I believe still accounts for the vast majority of restore requests. So it's an appropriate way to protect current versions of changeable data.

Archive is a way to keep assets for the long term. So far magnetic tape is the most well proven digital archival medium. At least Linear (as opposed to helical scan) tape is. I don't know anything about helical scan (ie what you shoot in your videocam) characteristics. Ideally you would have more than one copy of archival data. Issues around archive also include "future proofing" by periodically migrating from older technology to newer so 20 years from now you aren't frantically looking for a drive that will read your old media since the current drives will have been out ofproduction for a decade or more.

Having said all that, tape is just not particularly familiar to most people these days so if you do use HDD, you should periodically copy data to a different HDD every year or two.

Disclaimer: I work in the LTO tape business so I might be biased!

Taky Cheung · October 9th, 2009, 10:25 PM

I am using harddrives for all backups. Then pull them offline using these removable trayless system

Trayless Removable Harddrive System | L.A. Color Blog

Steve Kalle · October 9th, 2009, 10:52 PM

Kind of good timing(for you) that this issue comes up. Just last night, I was editing a 2 week old video that had some how become corrupted. They are XDCAM EX files and I have a 2-drive Raid 1 array which is used to import from the SxS cards and then I copy it to another 2-drive Raid 1 array. The video had a couple of small "glitches" during playback but the copy on the 2nd Raid 1 array was fine. I am still somewhat new to the video production world but not to computers and Raid, especially. I am setting up a 4-drive Raid 10 array for editing in addition to 2 external 1TB drives for backup/archival AND 2 copies to DL DVDs. My video only needs to be stored on the computer for editing for no more than 3 months back so I will remove files as I fill up the Raid 10 array.

Oh, and let me tell you about the time 900GB of data dissappeared from a 4-drive Raid 5 array. That was FUN restoring. Took me about 40hrs to finally get about 95% restored - all thanks to a corrupted partition file which happened due to a brownout(I think). And I had to use a virtual machine to get the restore program to work.

I have been using a 3ware 9650SE-8 for 2-3yrs without a single problem. I am ordering an Areca 1680ix and external 4-bay case for the Raid 10 and plan on moving the 3ware card to another computer. I also had a 4-drive Raid 10 array with 4 10k rpm WD Raptors for the OS & Apps connected to the onboard Intel Raid. After less than 2yrs, 2 of those drives failed within 3 wks of one another. This is why I dont use Raid 5 anymore. However, if you are a PC guy like me, then don't trust your data to the onboard Raid and also get a UPS.

I agree 100% with what Jim said about tape and hard drives for archival.

Hope this helps.

Bill Vincent · October 13th, 2009, 05:27 AM

Thanks for the replies, everyone.

Yes, I was specifically talking about archiving long-term as opposed to just backing up. Both are important, but archiving is a more prescient concern as I'm trying to figure out what to do with these HUGE HD projects that need to go into storage of some kind.

I spoke with a DIT for film production yesterday who said they rely on tape backup systems for Long-term storage. It takes a horribly long time to back up the data to tape, but once it is there it's a fairly safe storage medium.

Jim, your comment about future-proofing is right on target. It's going to be necessary to migrate archives to new technology every few years - but that's okay as long as I know the format I have used will be at least readable when it comes time to migrate. Again, tape data backup seems to be the safest of all mediums right now, although it seems so "old school"... lol.

Perrone Ford · October 13th, 2009, 05:47 AM

LTO or Optical would be my choices. I don't have LTO so I use BluRay. Seems to be working nicely for me.

Good luck with your choices.

Ron Little · October 13th, 2009, 07:12 AM

Tell me more about how you use blue ray? Do you use them just like a data dvd? And where are you getting them?

Perrone Ford · October 13th, 2009, 07:35 AM

I've posted on here a few times about my use of BluRay, but yes, I use them as data discs for archival purposes.

Essentially, my 1hr projects go onto a 25GB BluRay using Jpeg2000 encoding. This gives an excellent encoding in minimum space. It also leaves enough room for me to place both a high bandwidth and low bandwidth web version (h.264) on the same disc. I can even bundle a player on the disc so each disc becomes a self-contained playing system.

For larger projects I move to 50GB BluRays (or multiples thereof). Most of these longer projects are just conferences, so I don't need to save them at best quality. I can usually put 4 hours worth onto a 50GB BluRay without much loss.

Stored on BluRay, they are susceptible to fewer issue than tape, in my opinion. Finding a computer to read them later is also going to be quite a lot simpler than finding a tape machine. And unlike saving them in a media that requires not only unique hardware, but a unique codec, using these as data discs only, abstracts that.

Forum rules prevent me from divulging where I am buying my media from, but my last purchase of 25GB BluRays was $2.67 per disk for TDKs, and the 50GB disks were $17.99 each. Compared to ANY tape based format that can store a 1920x1080 image, this is DIRT cheap.

Brian Standing · October 13th, 2009, 09:36 AM

I'm using LTO tape. I bought a refurbished LTO-2 drive for about $500, and I'm very happy with my choice. It's much faster than you'd expect for a linear recording mechanism.

I work in long-form documentary, so it's not unusual for me to have 60 hours or more of footage for a 90-minute project. I'm still shooting HDV, and use Cineform Neo Scene to convert to Cineform codec for editing. When I'm done with a project, the HDV originals go into a vault, the Cineform files, project files and capture logs go onto LTO tape, so I have double redundancy. If I ever shift to a tapeless acquisition system, I'll simply make two copies of the LTO tape.

I looked at Blu-Ray, but after you factor in the smaller media size (25gb for Blu-Ray disks vs. 200gb-800gb for LTO tape), Blu-Ray just wasn't practical. Who wants to sit around and swap disks all day? Plus, LTO tape has a proven lifespan and the standard requires that all hardware is backward-compatible for two generations. So, when I upgrade to a LTO-4 deck, my LTO-2 tapes should still be fine.

Perrone Ford · October 13th, 2009, 10:07 AM

Brian makes and excellent point. If you have copious amounts of raw data, LTO is about the only thing that makes a lot of sense. My needs were very different.

Jim Andrada · October 13th, 2009, 12:27 PM

By the way, we're withing spitting distance of the release of LTO Gen 5 with 1.6 TB (uncompressed) capacity per cartridge. So we're tracking quite closely to the originally announced roadmap of doubling capacity every two years.

LTO Technology

Another nice thing is that LTO generation X will read and write Generation X-1 media and will read (but not write) Generation X-2 media

In other words, if you have LTO 3 cartridges, they'll be readable on Generation 5 so there's at least a few years of future proofing built in. Our LTO drive reliability is also pretty good. I've seen LTO tapes that we use in our drive test labs survive over 20,000 load unload cycles, and the drives are rated for upwards of one hundred thousand cycles.

Jim Snow · October 13th, 2009, 01:27 PM

Quote:

Originally Posted by Perrone Ford

Stored on BluRay, they are susceptible to fewer issue than tape, in my opinion. Finding a computer to read them later is also going to be quite a lot simpler than finding a tape machine. And unlike saving them in a media that requires not only unique hardware, but a unique codec, using these as data discs only, abstracts that.

Forum rules prevent me from divulging where I am buying my media from, but my last purchase of 25GB BluRays was $2.67 per disk for TDKs, and the 50GB disks were $17.99 each. Compared to ANY tape based format that can store a 1920x1080 image, this is DIRT cheap.

Perrone, I agree that this an attractive price point. However, I wish I didn't know as much as I do about optical media. My concern is about its shelf life. Dye layer stability over time is a real issue. I know that with DVD media, there are some brands that are quite decent while other are virtually guaranteed to fail. There is very little data on the "good guy, bad guy" list of BluRay media suppliers.

Bill Vincent · October 15th, 2009, 10:09 PM

This is a fascinating topic for me. I actually got to go on a music video shoot this week with a DIT for a Red camera. He had a very thorough and interesting workflow which I only got to see firsthand once. I wish I could remember exactly what he was doing, but he did say that for long-term backup he's using LTO2.

I think that's where I'm headed too. I don't trust optical media for long-term storage, it's just burned me already too many times. Same with hard drives. Cards might be an okay option, but others have mentioned cards going bad that were recently recorded.

Overall, there is no perfect solution, because all methods will fail. The only functionally "infallable" solution is a thoroughly redundant set of backups over multiple formats, refreshed and updated as years go by.

Perrone Ford · October 15th, 2009, 10:50 PM

Quote:

"Overall, there is no perfect solution, because all methods will fail. The only functionally "infallable" solution is a thoroughly redundant set of backups over multiple formats, refreshed and updated as years go by. "

Exactly!

So knowing that you will have to walk that data forward in a few years do you:

A. Chose a solution that is specific to both a medium AND a codec (say like HDV which is tape specific and codec specific)

B. Choose a proprietary codec on common media (DVCProHD, XDCamEX, Cineform)

C. Choose a common codec on proprietary media (Mpeg2 on AIT), REDCode on LTO)

4. Choose a standards based codec on common media (VC1 or VC3 on HDD, JP2K on BluRay)

My own choice was #4. Current BluRay media may not have great longevity. But I know in 3 years time, every PC will have a burner/reader, media density will have quadrupled most likely, and we'll have more stable media technology.

I choose a standards based codec like VC1 (Windows Media), VC3 (Avid's DNxHD), or for all my work now Jpeg2000 which is an ISO standard wavelet technology.

I just didn't see the point in sinking a fortune into an "archive" strategy that would be obsolete in a few years. If you KNOW you going to move the data forward, go cheap. And make DARN sure that you have multiple readers for that data. Finding an LTO-3 machine might be kinda hard in 3-5 years. Finding a BluRay reader is as easy as going to any PC in the library. Apple may even have them by then.

October 8th, 2009, 06:09 AM	#1
Bill Vincent Major Player Join Date: Aug 2009 Location: Nashville, TN Posts: 253	Safe and Long Term Backup Solutions I threw out this question at a WEVA panel of experts - the question was, what is a failsafe method for backing up our project data in the long-term? Their answer was multiple sources of backup. While I concur, using different media (drives, cards and/or tape backup) can be very expensive. My question today is whether RAID technology is the way to go - since it is a redundant system if done properly, and if one drive fails in the array, you can still recover the data from the redundancy on the other drives. It seems to me that if the RAID array is properly stored and maintained that it could theoretically provide safe and long-term backup storage. Anyone doing this? If not, what long-term solution are you using for backing up your HD projects?

October 9th, 2009, 10:25 PM	#3
Taky Cheung Inner Circle Join Date: May 2007 Location: Los Angeles, USA Posts: 2,114	I am using harddrives for all backups. Then pull them offline using these removable trayless system Trayless Removable Harddrive System \| L.A. Color Blog __________________ LA Color Pros Blog RODE Authorized Reseller . Comer LED Camera Lights . TakyBox HTML5 Menu Generator

October 13th, 2009, 05:47 AM	#6
Perrone Ford Inner Circle Join Date: Nov 2006 Location: Tallahassee, FL Posts: 4,100	LTO or Optical would be my choices. I don't have LTO so I use BluRay. Seems to be working nicely for me. Good luck with your choices. __________________ DVX100, PMW-EX1, Canon 550D, FigRig, Dell Octocore, Avid MC4/5, MB Looks, RedCineX, Matrox MX02 mini, GTech RAID, Edirol R-4, Senn. G2 Evo, Countryman, Moles and Lowels.

October 13th, 2009, 07:35 AM	#8
Perrone Ford Inner Circle Join Date: Nov 2006 Location: Tallahassee, FL Posts: 4,100	I've posted on here a few times about my use of BluRay, but yes, I use them as data discs for archival purposes. Essentially, my 1hr projects go onto a 25GB BluRay using Jpeg2000 encoding. This gives an excellent encoding in minimum space. It also leaves enough room for me to place both a high bandwidth and low bandwidth web version (h.264) on the same disc. I can even bundle a player on the disc so each disc becomes a self-contained playing system. For larger projects I move to 50GB BluRays (or multiples thereof). Most of these longer projects are just conferences, so I don't need to save them at best quality. I can usually put 4 hours worth onto a 50GB BluRay without much loss. Stored on BluRay, they are susceptible to fewer issue than tape, in my opinion. Finding a computer to read them later is also going to be quite a lot simpler than finding a tape machine. And unlike saving them in a media that requires not only unique hardware, but a unique codec, using these as data discs only, abstracts that. Forum rules prevent me from divulging where I am buying my media from, but my last purchase of 25GB BluRays was $2.67 per disk for TDKs, and the 50GB disks were $17.99 each. Compared to ANY tape based format that can store a 1920x1080 image, this is DIRT cheap. __________________ DVX100, PMW-EX1, Canon 550D, FigRig, Dell Octocore, Avid MC4/5, MB Looks, RedCineX, Matrox MX02 mini, GTech RAID, Edirol R-4, Senn. G2 Evo, Countryman, Moles and Lowels.

October 13th, 2009, 09:36 AM	#9
Brian Standing Major Player Join Date: Dec 2003 Location: Madison, Wisconsin Posts: 621	I'm using LTO tape. I bought a refurbished LTO-2 drive for about $500, and I'm very happy with my choice. It's much faster than you'd expect for a linear recording mechanism. I work in long-form documentary, so it's not unusual for me to have 60 hours or more of footage for a 90-minute project. I'm still shooting HDV, and use Cineform Neo Scene to convert to Cineform codec for editing. When I'm done with a project, the HDV originals go into a vault, the Cineform files, project files and capture logs go onto LTO tape, so I have double redundancy. If I ever shift to a tapeless acquisition system, I'll simply make two copies of the LTO tape. I looked at Blu-Ray, but after you factor in the smaller media size (25gb for Blu-Ray disks vs. 200gb-800gb for LTO tape), Blu-Ray just wasn't practical. Who wants to sit around and swap disks all day? Plus, LTO tape has a proven lifespan and the standard requires that all hardware is backward-compatible for two generations. So, when I upgrade to a LTO-4 deck, my LTO-2 tapes should still be fine. __________________ http://www.prolefeedstudios.com/blog/ Documentary for the masses!

October 9th, 2009, 02:48 PM	#2
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Generally hard drives (RAID or not) are not designed as long term shelf storage devices. Tape (well, Linear tape anyhow) is designed for long term shelf life. Things like using special coatings on the backside of the tape so it won't adhere to the front side when tightly wrapped on the reel and stored for years and years. However, Backup and Archive are really two different animals. Backup is to protect you in case of failure - including operator error, which I believe still accounts for the vast majority of restore requests. So it's an appropriate way to protect current versions of changeable data. Archive is a way to keep assets for the long term. So far magnetic tape is the most well proven digital archival medium. At least Linear (as opposed to helical scan) tape is. I don't know anything about helical scan (ie what you shoot in your videocam) characteristics. Ideally you would have more than one copy of archival data. Issues around archive also include "future proofing" by periodically migrating from older technology to newer so 20 years from now you aren't frantically looking for a drive that will read your old media since the current drives will have been out ofproduction for a decade or more. Having said all that, tape is just not particularly familiar to most people these days so if you do use HDD, you should periodically copy data to a different HDD every year or two. Disclaimer: I work in the LTO tape business so I might be biased!

October 9th, 2009, 10:52 PM	#4
Steve Kalle Trustee Join Date: Aug 2009 Location: Chicago, IL Posts: 1,554	Kind of good timing(for you) that this issue comes up. Just last night, I was editing a 2 week old video that had some how become corrupted. They are XDCAM EX files and I have a 2-drive Raid 1 array which is used to import from the SxS cards and then I copy it to another 2-drive Raid 1 array. The video had a couple of small "glitches" during playback but the copy on the 2nd Raid 1 array was fine. I am still somewhat new to the video production world but not to computers and Raid, especially. I am setting up a 4-drive Raid 10 array for editing in addition to 2 external 1TB drives for backup/archival AND 2 copies to DL DVDs. My video only needs to be stored on the computer for editing for no more than 3 months back so I will remove files as I fill up the Raid 10 array. Oh, and let me tell you about the time 900GB of data dissappeared from a 4-drive Raid 5 array. That was FUN restoring. Took me about 40hrs to finally get about 95% restored - all thanks to a corrupted partition file which happened due to a brownout(I think). And I had to use a virtual machine to get the restore program to work. I have been using a 3ware 9650SE-8 for 2-3yrs without a single problem. I am ordering an Areca 1680ix and external 4-bay case for the Raid 10 and plan on moving the 3ware card to another computer. I also had a 4-drive Raid 10 array with 4 10k rpm WD Raptors for the OS & Apps connected to the onboard Intel Raid. After less than 2yrs, 2 of those drives failed within 3 wks of one another. This is why I dont use Raid 5 anymore. However, if you are a PC guy like me, then don't trust your data to the onboard Raid and also get a UPS. I agree 100% with what Jim said about tape and hard drives for archival. Hope this helps.

October 13th, 2009, 05:27 AM	#5
Bill Vincent Major Player Join Date: Aug 2009 Location: Nashville, TN Posts: 253	Thanks for the replies, everyone. Yes, I was specifically talking about archiving long-term as opposed to just backing up. Both are important, but archiving is a more prescient concern as I'm trying to figure out what to do with these HUGE HD projects that need to go into storage of some kind. I spoke with a DIT for film production yesterday who said they rely on tape backup systems for Long-term storage. It takes a horribly long time to back up the data to tape, but once it is there it's a fairly safe storage medium. Jim, your comment about future-proofing is right on target. It's going to be necessary to migrate archives to new technology every few years - but that's okay as long as I know the format I have used will be at least readable when it comes time to migrate. Again, tape data backup seems to be the safest of all mediums right now, although it seems so "old school"... lol.

October 13th, 2009, 07:12 AM	#7
Ron Little Major Player Join Date: Feb 2003 Location: Pensacola Fl. Posts: 627	Tell me more about how you use blue ray? Do you use them just like a data dvd? And where are you getting them?

October 13th, 2009, 10:07 AM	#10
Perrone Ford Inner Circle Join Date: Nov 2006 Location: Tallahassee, FL Posts: 4,100	Brian makes and excellent point. If you have copious amounts of raw data, LTO is about the only thing that makes a lot of sense. My needs were very different. __________________ DVX100, PMW-EX1, Canon 550D, FigRig, Dell Octocore, Avid MC4/5, MB Looks, RedCineX, Matrox MX02 mini, GTech RAID, Edirol R-4, Senn. G2 Evo, Countryman, Moles and Lowels.

October 13th, 2009, 12:27 PM	#11
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	By the way, we're withing spitting distance of the release of LTO Gen 5 with 1.6 TB (uncompressed) capacity per cartridge. So we're tracking quite closely to the originally announced roadmap of doubling capacity every two years. LTO Technology Another nice thing is that LTO generation X will read and write Generation X-1 media and will read (but not write) Generation X-2 media In other words, if you have LTO 3 cartridges, they'll be readable on Generation 5 so there's at least a few years of future proofing built in. Our LTO drive reliability is also pretty good. I've seen LTO tapes that we use in our drive test labs survive over 20,000 load unload cycles, and the drives are rated for upwards of one hundred thousand cycles.

October 15th, 2009, 10:09 PM	#13
Bill Vincent Major Player Join Date: Aug 2009 Location: Nashville, TN Posts: 253	This is a fascinating topic for me. I actually got to go on a music video shoot this week with a DIT for a Red camera. He had a very thorough and interesting workflow which I only got to see firsthand once. I wish I could remember exactly what he was doing, but he did say that for long-term backup he's using LTO2. I think that's where I'm headed too. I don't trust optical media for long-term storage, it's just burned me already too many times. Same with hard drives. Cards might be an okay option, but others have mentioned cards going bad that were recently recorded. Overall, there is no perfect solution, because all methods will fail. The only functionally "infallable" solution is a thoroughly redundant set of backups over multiple formats, refreshed and updated as years go by.