Archiving video

Geoff Jak · January 13th, 2011, 08:31 AM

I need to know what systems people are using to archive video. Are they storing online, if so, where?
Are they storing on hard drives, if so, using what software (please let me know a range if you know them)

Any advice much appreciated if you can help. thx

Casey Krugman · January 13th, 2011, 12:21 PM

I always liked the Blu-ray option. Get some archival discs, burn the files 25gb at a time, then store in peace.

Geoff Jak · January 13th, 2011, 04:51 PM

Thx Casey

I've never used Blue Ray but dont discard the idea....

What about a simple terra SATA II drive. That would be the most cost effective medium wouldn't it?

Is anyone dumping footage online?

And what about software to manage the scenes. If I want to catalogue all scenes - eg, landscsapes, cityscapes, cars, trucks ...is there software I can use to put things in order.
Thx

Jim Andrada · January 13th, 2011, 08:48 PM

Just a quick question - how do you define an archive?

In other words, are you expecting it to retain it's contents for a couple of weeks, a couple of months, a couple of years, a decade or more, etc? It makes a difference

Generally for long term storage nothing you can burn on a computer is truly archival, nor is any hard drive. Hard drives are designed to spin, not sit on a shelf for a year or two or more, and despite the talk about archival DVD's they aren't really to be trusted for long term storage.

LTO tape is probably technically the best medium around for long term storage, as it IS designed to sit around for long periods of time, but it isn't particularly cheap, and not particularly user friendly, although there have been some developments recently that are addressing that.

Maybe the real key is that for an archive to be secure, it has to be accessed and checked regularly and there needs to be a backup archive or two from which you can recover problems with "bit rot"/file corruption, which will happen.

But it does really depend on your specific needs.

Kind of a glum picture I'm afraid, although we're working on it. (I'm really in the storage business and we're very seriously looking at ways to structure so called "archive appliances" that in addition to keeping data safe also provide automated and non-disruptive ways to migrate data to newer technologies as the older technologies age. Which is another requirement of "archive" that isn't often appreciated.)

John C. Chu · January 15th, 2011, 10:07 AM

Ever since I got into this Canon DSLR video thing, I've been trying to figure a way to offload and archive the huge amount of files I've generated.

I finally decided to get a eSATA PCI card for my PowerMac G5 and a Kingwin brand eSATA/USB dock from Amazon.

Amazon.com: Kingwin 3.5IN Esata Plus USB 2.0 Docking Station: Electronics

This to me, is an affordable and convenient solution, not to mention how wickedly fast eSATA is.

I will just buy more bare naked drives as I need them in the future.

I have no idea of the durability of long term storage on hard drives, but I can't beat the price.

Jim Andrada · January 16th, 2011, 01:02 AM

Yes, price is pretty attractive - until you lose your video one day.

That's the problem in a way, lots of cheap storage available. If you go this route, please make at least two copies and preferably three of each disk and store them in different places. It's not IF you will lose something, but WHEN you will lose it. And occasionally mount the old disks and cross compare the two or three copies. Bit rot happens.

David W. Jones · January 16th, 2011, 07:37 AM

Backing up to raw 1TB drives.

Geoff Jak · January 16th, 2011, 04:56 PM

Thx to all for your responses.

John C Chu's solution looks a good starting point and maybe the end one too.But what about software to access a particular shot. Is anyone using a program that catalogues each scene so I can find a particular shot when needed.

Hard Disk is the way to go for now - on a big terra drive it seems but I definitely take the point that making two or three copies and checking them periodically is a necessity. We, all of us, really are in a precarious position given the immature stage the archive solutions business is at. There is plenty of room for this business to grow provided it is a cost effective siolution.

Jims comment "Bit rot happens"...is kinda scary, but accurate - just slowly lurking back there is the slimy shadows...beware!

Jim Andrada · January 16th, 2011, 06:07 PM

One thing that IBM is talking about on their LTO tape drives that implement a file system is the possibility of making the drive itself "MXF" aware - the idea would be to recognize MXF files and store the metadata in the directory partition - this would at least allow retrieval by timecode range without having to transfer the whole file to get a section of it. Not sure how you'd do it on HDD though - somehow the user level metadata would have to be kept in a data base of some kind that would reference the file names. Basically logging the shots, which I think most people do already.

Ah yes,"cost effective" - hard to define cost effectiveness. $250k cameras and lenses are certainly "cost effective" for their users or nobody wold buy them. "Cost Effective" for one man or small outfits is another thing altogether.

Let me turn it around and ask what you would consider a cost effective solution?

Just for kicks, assume a solution at price points of say 10TB/50TB/100TB/500TB/1PB

I'd be curious to know what people would consider cost effective at these levels. And what level of storage would be needed.

Or think of it another way - if there were a really good complete solution for archive (and backup while we're at it) out there, would it be reasonable to spend about the same on such a system as on a pro camera/lens combination?

Dean Sensui · January 16th, 2011, 07:12 PM

I've been using SATA mirrored RAIDs for the past 7 years and haven't lost anything yet.

I'm using Firmtek systems with Hitachi SATA drives.

Jim Andrada · January 16th, 2011, 10:56 PM

You're lucky. I know a few people who aren't as lucky. Do you periodically mount the drives and check them for errors - corrupted files, etc?

Dean Sensui · January 16th, 2011, 11:29 PM

Jim... the RAID software lets me know if there is a problem with one of the drives. If there is, I make yet another backup, replace the bad drive and sync the RAID.

I've done that twice. Perfect recoveries.

Mirrored RAIDs are used for mission-critical operations for everything from banks to air traffic control.

Jim Andrada · January 16th, 2011, 11:54 PM

The first example of mirrored RAID I came across was before anyone had thought of the term RAID - the Airlines Control Program developed by IBM in the early sixties maintained mirrored drives across the airline reservation system complex. It worked well then and works well now.

But if drives are stored on the shelf (ie not spinning) the controllers won't detect the out of sync issue and it's possible for them to drop data. If the system is always or regularly powered up, the scan is pretty effective. Unless of course you get a controller failure which wipes out both copies (or the equivalent in a RAID 5 config0. Controller failures do happen (I've seen them fail and corrupt all the members of a RAID array)

More expensive commercial array systems usually have dual controllers to avoid this kind of failure.

You actually made an important comment - ie you referred to making a new backup - typically RAID is thought of as a means of protecting against a single (or more) failure, but it won't protect against losing data intentionally - ie making a human error.

RAID protects against errors in online (ie active) storage but in and of itself it isn't an archiving methodology.

One thing that's happening these days is that drives are getting so big that recovering a bad drive can take many hours - or more, and the exposure to a second (fatal) error increases wit recovery time. Some products like IBM's XIV system (developed by an Israeli company that IBM bought a couple of years back) gets around this by scattering data all around the system so no one drive has a copy of any other singe drive. This allows the recovery to read several drives in parallel and write to several other drives in parallel, thus reducing this kind of exposure.

Patrick McLoad · January 17th, 2011, 09:54 AM

For what it's worth, I save all clips to an external hard drive AND burn them to BD, one copy for my files and another I send to the client. Although most of my prior footage is on BetaSP and i'm sort of new to flash card recording, I have NEVER had a client come back 3 or 5 or 7 years later looking for footage. By that time, most of it has been relegated for demo reel use. After a few years, I send it to my clients and make it their problem. But certainly a good idea to exercise a hard drive every now and then. I do like the idea of LTO tape drives, but seem too pricey right now for me.

Would be interesting to know what NASA uses for image storage, though I expect it to be on the order of a dedicated Cray computer.

My 2 cents only.

Jim Andrada · January 17th, 2011, 12:43 PM

Interesting question - I don't know what NASA uses. Although a Cray wouldn't be particularly relevant for data storage.

I just saw where the Kennedy archives are now available online in digitized format - they didn't say too much about the system but it was a cooperative effort that included EMC the big storage array maker.

Harvard is in process of digitizing everything in their library system and I know that they keep three copies of all the digitized data in different physical locations. They scan the archives on a regular schedule verifying checksums and when (not if) they detect a file corruption they recover by looking at the other two copies. Not sure of the exact details though.

I guess one of the real questions (which you address) that people have to answer for themselves is how long to retain the archived data - or rather, at what point they wouldn't be impacted by losing it. Clearly something like the Hollywood film archive is intended to be kept forever as are university library and museum collections, presidential archives, exploration data held by the oil companies, etc etc. (There is an archive in Stavanger Norway that has decades of tape cartridges that hold all the data recorded by the North Sea petroleum exploration ships. Oil companies reprocess old tapes with newer algorithms and faster machines, and do indeed discover oil that was missed by earlier analyses,)

But how long do we really need to keep the data around - ie at what rate does its value decrease to the point where losing it isn't a big deal or it isn't worth the cost of keeping the media around anymore. Once you have an answer to this, the archive decision gets simpler - if you think you'll only need it for a few years, then the odds of the media on which you stored it becoming obsolete becomes less important - but if you want to keep it for 10 or 15 years or more, then future proofing becomes important.

One of my Japanese clients had a sudden request from one of their customers for two reel to reel tape drives that had been out of production for - years and years. Apparently the customer had some important data that needed to be accessed and no working drives to access the data with. So my client built two new drives for them pretty much by hand. Would you like to hazard a guess how expensive that was? Don't even ask!

So just how long do you think you want to keep those old (digital) photos of your grandmother before just deciding to throw them out???

January 13th, 2011, 08:31 AM	#1
Geoff Jak Regular Crew Join Date: Oct 2005 Location: Perth Australia Posts: 121	Archiving video - best solution needed I need to know what systems people are using to archive video. Are they storing online, if so, where? Are they storing on hard drives, if so, using what software (please let me know a range if you know them) Any advice much appreciated if you can help. thx

January 15th, 2011, 10:07 AM	#5
John C. Chu Major Player Join Date: Sep 2003 Location: Rego Park , NYC Posts: 665	Ever since I got into this Canon DSLR video thing, I've been trying to figure a way to offload and archive the huge amount of files I've generated. I finally decided to get a eSATA PCI card for my PowerMac G5 and a Kingwin brand eSATA/USB dock from Amazon. Amazon.com: Kingwin 3.5IN Esata Plus USB 2.0 Docking Station: Electronics Amazon.com: Kingwin 3.5IN Esata Plus USB 2.0 Docking Station: Electronics This to me, is an affordable and convenient solution, not to mention how wickedly fast eSATA is. I will just buy more bare naked drives as I need them in the future. I have no idea of the durability of long term storage on hard drives, but I can't beat the price.

January 16th, 2011, 07:37 AM	#7
David W. Jones Trustee Join Date: Dec 2006 Location: Little Rock Posts: 1,383	Backing up to raw 1TB drives. __________________ David W. Jones www.joneshdfilms.com

January 16th, 2011, 07:12 PM	#10
Dean Sensui Inner Circle Join Date: Oct 2001 Location: Honolulu, HI Posts: 2,054	I've been using SATA mirrored RAIDs for the past 7 years and haven't lost anything yet. I'm using Firmtek systems with Hitachi SATA drives. __________________ Dean Sensui Exec Producer, Hawaii Goes Fishing

January 16th, 2011, 11:29 PM	#12
Dean Sensui Inner Circle Join Date: Oct 2001 Location: Honolulu, HI Posts: 2,054	Jim... the RAID software lets me know if there is a problem with one of the drives. If there is, I make yet another backup, replace the bad drive and sync the RAID. I've done that twice. Perfect recoveries. Mirrored RAIDs are used for mission-critical operations for everything from banks to air traffic control. __________________ Dean Sensui Exec Producer, Hawaii Goes Fishing

January 13th, 2011, 12:21 PM	#2
Casey Krugman Regular Crew Join Date: Jan 2007 Location: Brooklyn, NY Posts: 135	I always liked the Blu-ray option. Get some archival discs, burn the files 25gb at a time, then store in peace.

January 13th, 2011, 04:51 PM	#3
Geoff Jak Regular Crew Join Date: Oct 2005 Location: Perth Australia Posts: 121	Thx Casey I've never used Blue Ray but dont discard the idea.... What about a simple terra SATA II drive. That would be the most cost effective medium wouldn't it? Is anyone dumping footage online? And what about software to manage the scenes. If I want to catalogue all scenes - eg, landscsapes, cityscapes, cars, trucks ...is there software I can use to put things in order. Thx

January 13th, 2011, 08:48 PM	#4
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Just a quick question - how do you define an archive? In other words, are you expecting it to retain it's contents for a couple of weeks, a couple of months, a couple of years, a decade or more, etc? It makes a difference Generally for long term storage nothing you can burn on a computer is truly archival, nor is any hard drive. Hard drives are designed to spin, not sit on a shelf for a year or two or more, and despite the talk about archival DVD's they aren't really to be trusted for long term storage. LTO tape is probably technically the best medium around for long term storage, as it IS designed to sit around for long periods of time, but it isn't particularly cheap, and not particularly user friendly, although there have been some developments recently that are addressing that. Maybe the real key is that for an archive to be secure, it has to be accessed and checked regularly and there needs to be a backup archive or two from which you can recover problems with "bit rot"/file corruption, which will happen. But it does really depend on your specific needs. Kind of a glum picture I'm afraid, although we're working on it. (I'm really in the storage business and we're very seriously looking at ways to structure so called "archive appliances" that in addition to keeping data safe also provide automated and non-disruptive ways to migrate data to newer technologies as the older technologies age. Which is another requirement of "archive" that isn't often appreciated.)

January 16th, 2011, 01:02 AM	#6
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Yes, price is pretty attractive - until you lose your video one day. That's the problem in a way, lots of cheap storage available. If you go this route, please make at least two copies and preferably three of each disk and store them in different places. It's not IF you will lose something, but WHEN you will lose it. And occasionally mount the old disks and cross compare the two or three copies. Bit rot happens.

January 16th, 2011, 04:56 PM	#8
Geoff Jak Regular Crew Join Date: Oct 2005 Location: Perth Australia Posts: 121	Thx to all for your responses. John C Chu's solution looks a good starting point and maybe the end one too.But what about software to access a particular shot. Is anyone using a program that catalogues each scene so I can find a particular shot when needed. Hard Disk is the way to go for now - on a big terra drive it seems but I definitely take the point that making two or three copies and checking them periodically is a necessity. We, all of us, really are in a precarious position given the immature stage the archive solutions business is at. There is plenty of room for this business to grow provided it is a cost effective siolution. Jims comment "Bit rot happens"...is kinda scary, but accurate - just slowly lurking back there is the slimy shadows...beware!

January 16th, 2011, 06:07 PM	#9
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	One thing that IBM is talking about on their LTO tape drives that implement a file system is the possibility of making the drive itself "MXF" aware - the idea would be to recognize MXF files and store the metadata in the directory partition - this would at least allow retrieval by timecode range without having to transfer the whole file to get a section of it. Not sure how you'd do it on HDD though - somehow the user level metadata would have to be kept in a data base of some kind that would reference the file names. Basically logging the shots, which I think most people do already. Ah yes,"cost effective" - hard to define cost effectiveness. $250k cameras and lenses are certainly "cost effective" for their users or nobody wold buy them. "Cost Effective" for one man or small outfits is another thing altogether. Let me turn it around and ask what you would consider a cost effective solution? Just for kicks, assume a solution at price points of say 10TB/50TB/100TB/500TB/1PB I'd be curious to know what people would consider cost effective at these levels. And what level of storage would be needed. Or think of it another way - if there were a really good complete solution for archive (and backup while we're at it) out there, would it be reasonable to spend about the same on such a system as on a pro camera/lens combination?

January 16th, 2011, 10:56 PM	#11
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	You're lucky. I know a few people who aren't as lucky. Do you periodically mount the drives and check them for errors - corrupted files, etc?

January 16th, 2011, 11:54 PM	#13
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	The first example of mirrored RAID I came across was before anyone had thought of the term RAID - the Airlines Control Program developed by IBM in the early sixties maintained mirrored drives across the airline reservation system complex. It worked well then and works well now. But if drives are stored on the shelf (ie not spinning) the controllers won't detect the out of sync issue and it's possible for them to drop data. If the system is always or regularly powered up, the scan is pretty effective. Unless of course you get a controller failure which wipes out both copies (or the equivalent in a RAID 5 config0. Controller failures do happen (I've seen them fail and corrupt all the members of a RAID array) More expensive commercial array systems usually have dual controllers to avoid this kind of failure. You actually made an important comment - ie you referred to making a new backup - typically RAID is thought of as a means of protecting against a single (or more) failure, but it won't protect against losing data intentionally - ie making a human error. RAID protects against errors in online (ie active) storage but in and of itself it isn't an archiving methodology. One thing that's happening these days is that drives are getting so big that recovering a bad drive can take many hours - or more, and the exposure to a second (fatal) error increases wit recovery time. Some products like IBM's XIV system (developed by an Israeli company that IBM bought a couple of years back) gets around this by scattering data all around the system so no one drive has a copy of any other singe drive. This allows the recovery to read several drives in parallel and write to several other drives in parallel, thus reducing this kind of exposure.

January 17th, 2011, 09:54 AM	#14
Patrick McLoad Regular Crew Join Date: Mar 2003 Location: Houston, Texas Posts: 52	For what it's worth, I save all clips to an external hard drive AND burn them to BD, one copy for my files and another I send to the client. Although most of my prior footage is on BetaSP and i'm sort of new to flash card recording, I have NEVER had a client come back 3 or 5 or 7 years later looking for footage. By that time, most of it has been relegated for demo reel use. After a few years, I send it to my clients and make it their problem. But certainly a good idea to exercise a hard drive every now and then. I do like the idea of LTO tape drives, but seem too pricey right now for me. Would be interesting to know what NASA uses for image storage, though I expect it to be on the order of a dedicated Cray computer. My 2 cents only.

January 17th, 2011, 12:43 PM	#15
Jim Andrada Inner Circle Join Date: Feb 2007 Location: Tucson AZ Posts: 2,211	Interesting question - I don't know what NASA uses. Although a Cray wouldn't be particularly relevant for data storage. I just saw where the Kennedy archives are now available online in digitized format - they didn't say too much about the system but it was a cooperative effort that included EMC the big storage array maker. Harvard is in process of digitizing everything in their library system and I know that they keep three copies of all the digitized data in different physical locations. They scan the archives on a regular schedule verifying checksums and when (not if) they detect a file corruption they recover by looking at the other two copies. Not sure of the exact details though. I guess one of the real questions (which you address) that people have to answer for themselves is how long to retain the archived data - or rather, at what point they wouldn't be impacted by losing it. Clearly something like the Hollywood film archive is intended to be kept forever as are university library and museum collections, presidential archives, exploration data held by the oil companies, etc etc. (There is an archive in Stavanger Norway that has decades of tape cartridges that hold all the data recorded by the North Sea petroleum exploration ships. Oil companies reprocess old tapes with newer algorithms and faster machines, and do indeed discover oil that was missed by earlier analyses,) But how long do we really need to keep the data around - ie at what rate does its value decrease to the point where losing it isn't a big deal or it isn't worth the cost of keeping the media around anymore. Once you have an answer to this, the archive decision gets simpler - if you think you'll only need it for a few years, then the odds of the media on which you stored it becoming obsolete becomes less important - but if you want to keep it for 10 or 15 years or more, then future proofing becomes important. One of my Japanese clients had a sudden request from one of their customers for two reel to reel tape drives that had been out of production for - years and years. Apparently the customer had some important data that needed to be accessed and no working drives to access the data with. So my client built two new drives for them pretty much by hand. Would you like to hazard a guess how expensive that was? Don't even ask! So just how long do you think you want to keep those old (digital) photos of your grandmother before just deciding to throw them out???