Storage/Backup strategy at DVinfo.net

Tran Bui · July 25th, 2007, 02:51 PM

Hi all,
Can't find a proper place to post this, so I post it here.
I store my captured avi clips (4.2GB each) in my hard drives, also one DVD copy of each. The original clips are still on tapes.
Both tape and DVD are not very reliable in the long run, so I still have to retain a copy on hard drives. Of course, this strategy requires hard drives to be purchased often, but that seems the most secure plan.
What is your strategy?

Jim Andrada · July 25th, 2007, 10:53 PM

Actually I think tape generally has longer archival life than hard disks, which can have problems over time just sitting on a shelf.

User writable (not stamped) DVD's probably don't have very long life. The problem arises because over time the medium wants to return to it's original and most stable state. There is a tough tradeoff between permanence and power because if the material is more stable it also takes much higher power to change it's state when writing and DVD writers (and hard drives for that matter) can't put out much power, particularly as the density gets higher and the heads get smaller.

One of the upcoming technologies for hard disks will use a laser to heat the area which is about to be written. This means that switching the magnetic field can be done with low power, but when the material cools it reverts to a more stable form and the magnetic field also remains stable.

The technology is known as HAMR or heat assisted magnetic recording.

Tran Bui · July 26th, 2007, 11:10 AM

Agree.
I'm sure there will be more reliable ways to archive our clips in the future, and hope they will be cheaper. At the moment, with my condition, I don't see any other than the tape-drive-dvd solution.
How are you archiving your clips so far?

Jim Andrada · July 26th, 2007, 04:59 PM

Honestly speaking, the best way to archive clips etc is to store them on a storage service. The fundamental problem is that, without a direct fibre connection, the upload time is prohibitive. I encrypt all my business data and store it on my web site by FTP upload and the ISP does the backups etc. I was really glad I had done this when my file server failed.

I think for now the most practical way is to save on hard disk with regular backup and keep the tapes themselves as an archive.

If you're really paranoid about things, it would be best to be sure the tapes are kept in a separate location from the hard drive. A small climate controlled public storage locker isn't terribly expensive if you have a lot of stuff. I think the fundamental priciple of backup is not that data doesn't disappear, but rather that the probability of two or more copies failing at the same time is vanishingly small.

I haven't gone quite that far yet, though - maybe because my clips aren't so valuable (at least not yet!)

Tran Bui · July 28th, 2007, 12:14 PM

Hi Jim,
I was responsible for IT equipment and procedures for a company several years ago. We had daily, weekly and monthly tape (I remember HP tape & drive) backup. The tapes were stored away from the server. I am now basically do the same thing with an additional of DVD copies. I said basically because in practice tape backup for video clips requires some process i.e capturing and copying. More over, it's more expensive but more reliale.
I think my strategy is ok for now. I am doing more research on tape, hopefully just tape-DVD copies is enough for the long run.
Cheers!

Jim Andrada · July 30th, 2007, 05:19 PM

Tran

I don't know too much technically about helical scan video tape, but I've been involved in the development of LTO tape drives for 10 years+ now so if you have any questions feel free to ask. If it's technical I won't be much help but I can probably find out for you.

Bob Warner · August 2nd, 2007, 11:59 AM

Couple of newbie questions. I'm not going to do anything too sophisticated, but I figured I would just have DVDs and ext. drive backup. I don't have anything that would be catastrophic to lose, but I would like to have reasonable security on keeping the movies. 1. Gold DVDs. Are these a worthwhile investment? Not sure how long they are expected to last without degradation. If I burned 3 of these it seems I'd be good in that area. 2. Although I'm using a fast external drive for editing, I was planning on using less expensive, slower drives (I think you can get 500GB drives for $120 or so) for storage and having storage on 2 drives. Does this make sense? 3. Is there an estimate of the safe life of an external drive on a shelf. I guess there are no hard rules on this but if one wants to play it on the safe side at what point should one put the data on new external drives (which, of course, would have to be bigger and bigger, but hopefully prices will be coming down). I'm just looking for a system that is redundant and as safe as possible without going to great expense. Bob

Jim Andrada · August 8th, 2007, 12:09 AM

Bob,

I've heard that recently there was an industry study in Japan to try to assess a useful lifetime for rewritable DVD using statistical analysis of accelerated aging tests.

Results were simply all over the map. There seems no clear trend or consensus on archival life. Basically quite a few of the samples indicated that they should be OK for in the ballpark of maybe 10 years, others looked like they might last longer (maybe 30 years), others couldn't even be properly evaluated and might not even be reliable for the 10 year period. Others looked like they could have extremely long lifetimes at room temperature, but elevating the temperature a few degrees would cause them to become unreliable very quickly.

I think my personal feeling is that I might want to refresh the data every 5 to 10 years and I would suggest making 2 or 3 copies on different branded media while you're at it. The so called "gold" DVD's claim higher data retention period, but again there were no consistent definitive results by the impartial study group. Whatever you do, don't store them in a hot attic or garage - maybe you should keep them in your wine cellar!

In so far as HDD archival life, there have been many studies of the lifetime of the media itself, but no definitive studies of the effects of storing an inactive HDD.

Amazing how much really isn't known about this subject. Part of the problem of course is that nobody can run a test that takes 30 to 50 years so everybody uses some kind of accelerated aging study where they stress the media at extremes of temperature and humidity and then try to extrapolate the data to get an idea of practical lifetimes.

Bob Warner · August 8th, 2007, 11:33 AM

I got the picture. Thanks for the data. Bob

July 25th, 2007, 02:51 PM	#1
Tran Bui New Boot Join Date: Jul 2007 Location: Rockville, MD Posts: 7	Storage/Backup strategy Hi all, Can't find a proper place to post this, so I post it here. I store my captured avi clips (4.2GB each) in my hard drives, also one DVD copy of each. The original clips are still on tapes. Both tape and DVD are not very reliable in the long run, so I still have to retain a copy on hard drives. Of course, this strategy requires hard drives to be purchased often, but that seems the most secure plan. What is your strategy?

July 25th, 2007, 10:53 PM	#2
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Actually I think tape generally has longer archival life than hard disks, which can have problems over time just sitting on a shelf. User writable (not stamped) DVD's probably don't have very long life. The problem arises because over time the medium wants to return to it's original and most stable state. There is a tough tradeoff between permanence and power because if the material is more stable it also takes much higher power to change it's state when writing and DVD writers (and hard drives for that matter) can't put out much power, particularly as the density gets higher and the heads get smaller. One of the upcoming technologies for hard disks will use a laser to heat the area which is about to be written. This means that switching the magnetic field can be done with low power, but when the material cools it reverts to a more stable form and the magnetic field also remains stable. The technology is known as HAMR or heat assisted magnetic recording.

July 26th, 2007, 11:10 AM	#3
Tran Bui New Boot Join Date: Jul 2007 Location: Rockville, MD Posts: 7	Agree. I'm sure there will be more reliable ways to archive our clips in the future, and hope they will be cheaper. At the moment, with my condition, I don't see any other than the tape-drive-dvd solution. How are you archiving your clips so far?

July 26th, 2007, 04:59 PM	#4
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Honestly speaking, the best way to archive clips etc is to store them on a storage service. The fundamental problem is that, without a direct fibre connection, the upload time is prohibitive. I encrypt all my business data and store it on my web site by FTP upload and the ISP does the backups etc. I was really glad I had done this when my file server failed. I think for now the most practical way is to save on hard disk with regular backup and keep the tapes themselves as an archive. If you're really paranoid about things, it would be best to be sure the tapes are kept in a separate location from the hard drive. A small climate controlled public storage locker isn't terribly expensive if you have a lot of stuff. I think the fundamental priciple of backup is not that data doesn't disappear, but rather that the probability of two or more copies failing at the same time is vanishingly small. I haven't gone quite that far yet, though - maybe because my clips aren't so valuable (at least not yet!)

July 28th, 2007, 12:14 PM	#5
Tran Bui New Boot Join Date: Jul 2007 Location: Rockville, MD Posts: 7	Hi Jim, I was responsible for IT equipment and procedures for a company several years ago. We had daily, weekly and monthly tape (I remember HP tape & drive) backup. The tapes were stored away from the server. I am now basically do the same thing with an additional of DVD copies. I said basically because in practice tape backup for video clips requires some process i.e capturing and copying. More over, it's more expensive but more reliale. I think my strategy is ok for now. I am doing more research on tape, hopefully just tape-DVD copies is enough for the long run. Cheers!

July 30th, 2007, 05:19 PM	#6
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Tran I don't know too much technically about helical scan video tape, but I've been involved in the development of LTO tape drives for 10 years+ now so if you have any questions feel free to ask. If it's technical I won't be much help but I can probably find out for you.

August 2nd, 2007, 11:59 AM	#7
Bob Warner Regular Crew Join Date: Mar 2007 Location: Los Angeles, CA Posts: 35	Couple of newbie questions. I'm not going to do anything too sophisticated, but I figured I would just have DVDs and ext. drive backup. I don't have anything that would be catastrophic to lose, but I would like to have reasonable security on keeping the movies. 1. Gold DVDs. Are these a worthwhile investment? Not sure how long they are expected to last without degradation. If I burned 3 of these it seems I'd be good in that area. 2. Although I'm using a fast external drive for editing, I was planning on using less expensive, slower drives (I think you can get 500GB drives for $120 or so) for storage and having storage on 2 drives. Does this make sense? 3. Is there an estimate of the safe life of an external drive on a shelf. I guess there are no hard rules on this but if one wants to play it on the safe side at what point should one put the data on new external drives (which, of course, would have to be bigger and bigger, but hopefully prices will be coming down). I'm just looking for a system that is redundant and as safe as possible without going to great expense. Bob

August 8th, 2007, 12:09 AM	#8
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Bob, I've heard that recently there was an industry study in Japan to try to assess a useful lifetime for rewritable DVD using statistical analysis of accelerated aging tests. Results were simply all over the map. There seems no clear trend or consensus on archival life. Basically quite a few of the samples indicated that they should be OK for in the ballpark of maybe 10 years, others looked like they might last longer (maybe 30 years), others couldn't even be properly evaluated and might not even be reliable for the 10 year period. Others looked like they could have extremely long lifetimes at room temperature, but elevating the temperature a few degrees would cause them to become unreliable very quickly. I think my personal feeling is that I might want to refresh the data every 5 to 10 years and I would suggest making 2 or 3 copies on different branded media while you're at it. The so called "gold" DVD's claim higher data retention period, but again there were no consistent definitive results by the impartial study group. Whatever you do, don't store them in a hot attic or garage - maybe you should keep them in your wine cellar! In so far as HDD archival life, there have been many studies of the lifetime of the media itself, but no definitive studies of the effects of storing an inactive HDD. Amazing how much really isn't known about this subject. Part of the problem of course is that nobody can run a test that takes 30 to 50 years so everybody uses some kind of accelerated aging study where they stress the media at extremes of temperature and humidity and then try to extrapolate the data to get an idea of practical lifetimes.

August 8th, 2007, 11:33 AM	#9
Bob Warner Regular Crew Join Date: Mar 2007 Location: Los Angeles, CA Posts: 35	I got the picture. Thanks for the data. Bob