Video archiving at DVinfo.net

Gary Chavez · May 18th, 2010, 01:50 PM

My company's IT guys laughed at me when I asked for 5TB storage on our system.
They suggested off site backup.
I was not really aware of such a possibility. My quick Goggle search as me reeling trying to dechipher that language. I have DVCAM and P2 HD QT and Content files that I need to save.

What should I be looking for in such a service?
Any recommendations?

Jim Andrada · May 18th, 2010, 02:22 PM

I think the IT guys' job is to worry about this kind of stuff for you. If they don't want to do it, then maybe your manager and their manager need to have a friendly talk. This all assumes of course that the video is work related.

Biggest hassle I think you're likely to run into is the amount of data you need to move over a network to an offsite service. Moving terabytes is really slow - much faster to send the service a couple of hard drives instead of transmitting the data. So maybe a criteria for choosing a service would relate to whether they can accept data this way.

Anyhow, this isn't something that non-IT professionals should have to figure out for themselves in a corporate setting. If the video is a digital asset of the company, I think the IT guys have to take responsibility for it. That would certainly be where I'd start from if I were in your situation

Gary Chavez · May 18th, 2010, 02:27 PM

You are preaching to the choir. It's a battle we lost. Thx for the suggestions.

Jim Andrada · May 18th, 2010, 03:17 PM

Wow - that's bad! They must not think there's much value to the asset!

OK then I guess if it were me I'd call some services and ask them for estimates based on amount of storage and expected volume of data transfer etc. I'd also check if they could handle receiving the data on disk vs transmission and if that would affect the cost. I'd be particularly alert to how they say they would ensure the accuracy of what winds up on their systems as errors in transmission/copying etc do occur. I'd also want to know how (specifically) they would propose to make sure that the data doesn't degrade while it's on their systems. (I know the Harvard University Library says they have a couple of hundred TB of digitized material and they keep two offsite copies and they scan all copies regularly looking for dropped/corrupted records and when (not if) they find them they recover the data from the other two copies.) Anyhow, I'd want the service to be really detailed and specific about how they would make sure there was no "bit rot" going on as the data sat there over time.

Then I'd take the estimates to management. I'd also try to get the IT group to at least review the estimates and look for obvious holes. I guess what I would want to be sure of is that their fingerprints are on whatever solution you wind up with so when the data disappears it isn't just you standing there all by yourself!

Other thing I'd look into (although again it should be the IT guys who do this or at least help you with it) would be doing it in-house with an LTO tape drive either standalone or in a small library. The latest version which is just hitting the market is Generation 5 and each cartridge holds 1.5TB uncompressed or approximately 3.75 TB using built in non-lossy compression. At most you'd need 4 or 5 cartridges. and data transfer time would be a lot faster than transmitting the data over the internet.

The new IBM G5 drives have a feature (which was demonstrated at NAB this year) that would let you drag and drop files to the drive without any specific backup software. Quantum has had a similar feature in their G4 drives I believe although their implementation is somewhat different/more kludgey than IBM's. I'd expect them to adopt the IBM methodology as it has been approved by the LTO Consortium and IBM has said that will make the design freely available.

If you used generation 4 drives which have been available for a couple of years now, the capacities would be from 800GB to 1.6TB again depending on the effectiveness of the built in compression algorithms. Tape cartridges run around $30 each for G4.

I think a couple of sets of cartridges would let you store one or two sets offsite and tape really has better shelf life characteristics than small HDD's.

Heck, you could even keep the cartridges in a couple of bank safe deposit boxes!

I'm glad I don't have your problem! Or your IT department.

Dean Sensui · May 19th, 2010, 12:12 AM

Quote:

Originally Posted by Gary Chavez

My company's IT guys laughed at me when I asked for 5TB storage on our system.

If your IT people laughed at you, then the matter should be taken out of their hands and put into your area of responsibility. They obviously don't take your requirements seriously, are unwilling to give you due consideration and are ignorant of what you do.

That happened to a company I was working in. Our photo archives were under the jurisdiction of the photo department. When new management came in, it was turned over to the IT department. The IT techs didn't take appropriate care of the data and, after ignoring repeated warnings and as a result of carelessness, lost tens of thousands of historic photos.

It happened two more times. Finally, upper management gave the responsibility back to the photo department.

What were the warnings? #1: put all critical equipment on uninterruptable power supplies. #2: all critical daily operations and archive data need to go on mirrored raids. Neither was done. In fact, it was later discovered the data drive in the photo server was simply laid into the case. It wasn't secured at all.

Upper management eventually took my recommendations seriously. Better late than never. Hope that doesn't happen to you and all your work.

Gary Chavez · May 19th, 2010, 06:42 AM

It must the nature of the IT personality to not value any needs outside their own.
I like the tape system and will explore that. The off site sounds risky. We didn't even discuss the disgruntled employee, the bank takeover, the data thief etc. Thx again for the help.

Jim Andrada · May 19th, 2010, 12:35 PM

Gary

Bear in mind that I'm a bit biased in favor of LTO because I've been involved with its development and manufacturing for over 10 years. I really believe it is seriously good stuff.

By the way - all media is not created equal. Fuji Film, Maxell, IBM (rebranded but sometimes cheaper than the actual maker's stuff) are all excellent. I've seen some cartridges last for 20,000 to 30,000 load/unload cycles in the manufacturing line although the spec is "only" 5,000 cycles.

I've been out of the IT arena for more years than most people have been alive and Dean's story sounds so much like what I was used to in the "old days" - I would have hoped things would have improved in all these years. Technology improves, human nature doesn't, I guess.

May 18th, 2010, 01:50 PM	#1
Gary Chavez Major Player Join Date: Mar 2003 Location: San Antonio Posts: 355	Video archiving My company's IT guys laughed at me when I asked for 5TB storage on our system. They suggested off site backup. I was not really aware of such a possibility. My quick Goggle search as me reeling trying to dechipher that language. I have DVCAM and P2 HD QT and Content files that I need to save. What should I be looking for in such a service? Any recommendations?

May 18th, 2010, 02:22 PM	#2
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	I think the IT guys' job is to worry about this kind of stuff for you. If they don't want to do it, then maybe your manager and their manager need to have a friendly talk. This all assumes of course that the video is work related. Biggest hassle I think you're likely to run into is the amount of data you need to move over a network to an offsite service. Moving terabytes is really slow - much faster to send the service a couple of hard drives instead of transmitting the data. So maybe a criteria for choosing a service would relate to whether they can accept data this way. Anyhow, this isn't something that non-IT professionals should have to figure out for themselves in a corporate setting. If the video is a digital asset of the company, I think the IT guys have to take responsibility for it. That would certainly be where I'd start from if I were in your situation

May 18th, 2010, 02:27 PM	#3
Gary Chavez Major Player Join Date: Mar 2003 Location: San Antonio Posts: 355	You are preaching to the choir. It's a battle we lost. Thx for the suggestions.

May 18th, 2010, 03:17 PM	#4
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Wow - that's bad! They must not think there's much value to the asset! OK then I guess if it were me I'd call some services and ask them for estimates based on amount of storage and expected volume of data transfer etc. I'd also check if they could handle receiving the data on disk vs transmission and if that would affect the cost. I'd be particularly alert to how they say they would ensure the accuracy of what winds up on their systems as errors in transmission/copying etc do occur. I'd also want to know how (specifically) they would propose to make sure that the data doesn't degrade while it's on their systems. (I know the Harvard University Library says they have a couple of hundred TB of digitized material and they keep two offsite copies and they scan all copies regularly looking for dropped/corrupted records and when (not if) they find them they recover the data from the other two copies.) Anyhow, I'd want the service to be really detailed and specific about how they would make sure there was no "bit rot" going on as the data sat there over time. Then I'd take the estimates to management. I'd also try to get the IT group to at least review the estimates and look for obvious holes. I guess what I would want to be sure of is that their fingerprints are on whatever solution you wind up with so when the data disappears it isn't just you standing there all by yourself! Other thing I'd look into (although again it should be the IT guys who do this or at least help you with it) would be doing it in-house with an LTO tape drive either standalone or in a small library. The latest version which is just hitting the market is Generation 5 and each cartridge holds 1.5TB uncompressed or approximately 3.75 TB using built in non-lossy compression. At most you'd need 4 or 5 cartridges. and data transfer time would be a lot faster than transmitting the data over the internet. The new IBM G5 drives have a feature (which was demonstrated at NAB this year) that would let you drag and drop files to the drive without any specific backup software. Quantum has had a similar feature in their G4 drives I believe although their implementation is somewhat different/more kludgey than IBM's. I'd expect them to adopt the IBM methodology as it has been approved by the LTO Consortium and IBM has said that will make the design freely available. If you used generation 4 drives which have been available for a couple of years now, the capacities would be from 800GB to 1.6TB again depending on the effectiveness of the built in compression algorithms. Tape cartridges run around $30 each for G4. I think a couple of sets of cartridges would let you store one or two sets offsite and tape really has better shelf life characteristics than small HDD's. Heck, you could even keep the cartridges in a couple of bank safe deposit boxes! I'm glad I don't have your problem! Or your IT department.

May 19th, 2010, 06:42 AM	#6
Gary Chavez Major Player Join Date: Mar 2003 Location: San Antonio Posts: 355	It must the nature of the IT personality to not value any needs outside their own. I like the tape system and will explore that. The off site sounds risky. We didn't even discuss the disgruntled employee, the bank takeover, the data thief etc. Thx again for the help.

May 19th, 2010, 12:35 PM	#7
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Gary Bear in mind that I'm a bit biased in favor of LTO because I've been involved with its development and manufacturing for over 10 years. I really believe it is seriously good stuff. By the way - all media is not created equal. Fuji Film, Maxell, IBM (rebranded but sometimes cheaper than the actual maker's stuff) are all excellent. I've seen some cartridges last for 20,000 to 30,000 load/unload cycles in the manufacturing line although the spec is "only" 5,000 cycles. I've been out of the IT arena for more years than most people have been alive and Dean's story sounds so much like what I was used to in the "old days" - I would have hoped things would have improved in all these years. Technology improves, human nature doesn't, I guess.