Robert: how are you archiving?

[Peter Richardson]

Now that I've started shooting A LOT of footage for my doc, I'm curious to know how you archive your footage Robert. Right now I'm backing up each card to a Dual Layer DVD, and working with the footage in my edit off of a G-Raid Pro in a RAID-5 configuration. As you might imagine, the archiving is a very time consuming proposition. I'm not looking at HDD as a viable long-term archival method, so I'm wondering how you are archiving footage. One thing I've been checking out is this AIT drive from Sony:

http://b2b.sony.com/Solutions/subcat...Drives/AIT-3Ex

They will be releasing a version in May that has a SATA interface, so I'm thinking I might be able to connect it to my MBPro via an ExpressCard SATA adapter. Curious to hear your thoughts on this solution and, of course, how you are archiving. Thanks!

Peter

[Robert Lane]

Hi Peter,

It's unfortunate that you're not here in AZ; tomorrow I'll be giving my P2/Apple Workflow Keynote at EAR in Phoenix and a great deal of the discussion is around archiving methodologies. I'll do my best to give a nutshell version of what's being covered:

First, it's important to note that there is no single, all-perfect archival method or media; it mostly comes down to three factors: What your current infrastructure is built around, what you own comfort level is with a particular media type and, what your clients or end-users need to receive your work. The media-type archiving conundrum isn't a new one; I've been through this once before.

Prior to migrating my company into video/film production I was a commercial-print & fashion shooter for about 20 years. Even before the first *viable* pro-DSLR ever hit the market we had digital backs for medium and large format cameras. They were huge, extremely costly and the file sizes they output dwarfed the then common 3.5" inch floppies. Most of the time those digi-files were sent off to a service bureau to be output as a transparency and that became the "archival" copy, but we always needed a way to store the original digital file too, and all we had to choose from were either Syquest drives, Zip (later Jazz), Rockwell-tape backup systems and a few others I can't even remember now. Back then HDD's were painfully expensive and there was no such thing as even a 1GB HDD - and a 250MB drive was thousands of dollars! And, back then HDD's were notorious for failure in less than 1 year. How times have changed.

And that's more to my point; technically you could use anything from DVD, HD-DVD (PC only right now), Blu-Ray, Ultrium & Quantum-type DLT's and of course HDD's. Whatever media you use you must ALWAYS create a diverse backup of the original archive, regardless what it is.

For my company's productions and for 90% of my clients - who range from indie producers to fortune 100 companies - HDD's make the most sense. They are the most cost effective, verstatile and universal storage method currently available. The remaining 10% are using DLT-type archival methods because they already have existing tape libraries along with the cataloging software to keep track of it all.

Cataloging, by the way, is one of the most overlooked aspects when it comes to choosing an archival method. For someone who doesn't shoot professionally meaning, it's not their sole source of income and don't shoot often then cataloging my not be such an issue. But if you're a full-time content provider of any type, then one project could easily turn into dozens if not hundreds of DVD's or even Blu-Ray disks, for example. And if you don't have a cataloging/library scheme or software already in place then you're going to have a major hassle on your hands trying to figure out which disc has what clips, scenes or "rolls" on it.

The debate about HDD reliability is endless and the opinions rage on both sides of the arguement but when you consider cost, ease of implementation, MTBF rates and the fact that cataloging becomes much less an issue because every file or folder has a time/date stamp on both it's creation and "last used" time, the HDD's really become the hands-down winner. Today, that is. 5 years from now, who knows.

Lastly, here's one interesting statistic to ponder: One of my clients runs a data-warehouse in southern AZ where they store literally tens-of-thousands of Terabytes of data between countless server racks (the actual number of both is a closely guarded secret). In the last 5 years less than 1% of the HDD's in these racks have failed and most of the failures were due to a software bug in their data-management software (proprietary) that for some odd reason forced the heads to bang against the stops repeatedly until the drive physically stopped working.

So as I say, HDD's are my preferred media of choice but regardless which one you choose make sure you create your diverse, redundant copy of your archive so that you're not putting all your precious files in one single place or disk/tape.

[Douglas Villalba]

Originally Posted by Peter Richardson

Now that I've started shooting A LOT of footage for my doc, I'm curious to know how you archive your footage Robert. Right now I'm backing up each card to a Dual Layer DVD, and working with the footage in my edit off of a G-Raid Pro in a RAID-5 configuration. As you might imagine, the archiving is a very time consuming proposition. I'm not looking at HDD as a viable long-term archival method, so I'm wondering how you are archiving footage. One thing I've been checking out is this AIT drive from Sony:

http://b2b.sony.com/Solutions/subcat...Drives/AIT-3Ex

They will be releasing a version in May that has a SATA interface, so I'm thinking I might be able to connect it to my MBPro via an ExpressCard SATA adapter. Curious to hear your thoughts on this solution and, of course, how you are archiving. Thanks!

Peter

Originally Posted by Robert Lane

Hi Peter,

It's unfortunate that you're not here in AZ; tomorrow I'll be giving my P2/Apple Workflow Keynote at EAR in Phoenix and a great deal of the discussion is around archiving methodologies. I'll do my best to give a nutshell version of what's being covered:

First, it's important to note that there is no single, all-perfect archival method or media; it mostly comes down to three factors: What your current infrastructure is built around, what you own comfort level is with a particular media type and, what your clients or end-users need to receive your work. The media-type archiving conundrum isn't a new one; I've been through this once before.

Prior to migrating my company into video/film production I was a commercial-print & fashion shooter for about 20 years. Even before the first *viable* pro-DSLR ever hit the market we had digital backs for medium and large format cameras. They were huge, extremely costly and the file sizes they output dwarfed the then common 3.5" inch floppies. Most of the time those digi-files were sent off to a service bureau to be output as a transparency and that became the "archival" copy, but we always needed a way to store the original digital file too, and all we had to choose from were either Syquest drives, Zip (later Jazz), Rockwell-tape backup systems and a few others I can't even remember now. Back then HDD's were painfully expensive and there was no such thing as even a 1GB HDD - and a 250MB drive was thousands of dollars! And, back then HDD's were notorious for failure in less than 1 year. How times have changed.

And that's more to my point; technically you could use anything from DVD, HD-DVD (PC only right now), Blu-Ray, Ultrium & Quantum-type DLT's and of course HDD's. Whatever media you use you must ALWAYS create a diverse backup of the original archive, regardless what it is.

For my company's productions and for 90% of my clients - who range from indie producers to fortune 100 companies - HDD's make the most sense. They are the most cost effective, verstatile and universal storage method currently available. The remaining 10% are using DLT-type archival methods because they already have existing tape libraries along with the cataloging software to keep track of it all.

Cataloging, by the way, is one of the most overlooked aspects when it comes to choosing an archival method. For someone who doesn't shoot professionally meaning, it's not their sole source of income and don't shoot often then cataloging my not be such an issue. But if you're a full-time content provider of any type, then one project could easily turn into dozens if not hundreds of DVD's or even Blu-Ray disks, for example. And if you don't have a cataloging/library scheme or software already in place then you're going to have a major hassle on your hands trying to figure out which disc has what clips, scenes or "rolls" on it.

The debate about HDD reliability is endless and the opinions rage on both sides of the arguement but when you consider cost, ease of implementation, MTBF rates and the fact that cataloging becomes much less an issue because every file or folder has a time/date stamp on both it's creation and "last used" time, the HDD's really become the hands-down winner. Today, that is. 5 years from now, who knows.

Lastly, here's one interesting statistic to ponder: One of my clients runs a data-warehouse in southern AZ where they store literally tens-of-thousands of Terabytes of data between countless server racks (the actual number of both is a closely guarded secret). In the last 5 years less than 1% of the HDD's in these racks have failed and most of the failures were due to a software bug in their data-management software (proprietary) that for some odd reason forced the heads to bang against the stops repeatedly until the drive physically stopped working.

So as I say, HDD's are my preferred media of choice but regardless which one you choose make sure you create your diverse, redundant copy of your archive so that you're not putting all your precious files in one single place or disk/tape.

I also use HDD as my archiving method. DVDs are are an option but I've had more DVDs go bad than HDD.

I can tell you that 100% of the HDDs that had gone bad on me were in FW cases. I've had LaCie 500GB, some Acomm and CompUSA cases go bad.
One thing for sure is that with this type of media, P2 and FS-100 you need to transfer your original to two separate storage solutions. I personally transfer to a HDD and to a SATA Raid. The SATA Raid I use to edit and the HDD replaces the tapes.

[Mike Schrengohst]

You can also get swappable storage....
Once you fill up a drive, it is a good idea to clone
it and store it in a secure location in a different place
than the original. Pull the drive out every once in a while
and clone it again. As drives get bigger and cheaper this
is one way to accomplish saving your data. Tape drives are
not 100% either....I have had AIT and DLT and the biggest
problem is retrieving just what you need without restoring the
data to a drive anyway. Perhaps the future will be somekind
of holographic storage array?

[Peter Richardson]

Thanks guys. I have pretty much ruled out HDD's as a long-term storage method (each to his own, I guess) but man am I getting tired of burning Dual Layer DVD's. Mike what did you think of AIT vs. DLT? I think you're actually spot on about the holographic storage coming sooner than anyone thinks, so maybe in a year this conversation will be moot. Hoping so. I'd be curious to hear your thoughts though on the Sony AIT 3ex drive I linked to. Is the drive Mac compatible? Couldn't find any info on Sony's site about this. Thanks,

Peter

[Mike Schrengohst]

Either tape drive has it pros and cons.
The AIT is more delicate, the DLT worked all the time
it did not have as good as software as the AIT drive
for finding files.
Like Robert said, file management is key. Proper
naming conventions and a .xls log helps.

[Peter Richardson]

Thanks Mike. What do you mean by delicate? Would it be appropriate for taking in the field, to do offloads at night at a basecamp? Is AIT Mac compatible? Thanks!

Peter

[Jeffrey Butler]

Remember, though, that the failure rate for HDD's is 100%. They will all go the way of the wind...great interim, but don't have everything on HDD. I'm not sure I have a better immediate solution, but optical is probably in there somewhere. I guess it depends on your product...

[Douglas Villalba]

I must be lucky. I still have 40GB HDD that I bought for $350 when it first came out about 7 years ago.
It runs in a computer that never shots off. I keep my sound & titles folder in it.

[Mike Schrengohst]

Originally Posted by Peter Richardson

Thanks Mike. What do you mean by delicate? Would it be appropriate for taking in the field, to do offloads at night at a basecamp? Is AIT Mac compatible? Thanks!

Peter

I don't think that would be a good solution in the field. To slow.....
We use 100 gig portables on set and then at night we copy everything
to another external drive. If I have time I will also burn the cards to DVD.
That will get to be more of a problem as 8 and 16 gig cards come into use.
The 4 gig cards are perfect but you must have many and learn the swap discipline.

For long term storage? I am sure some great minds are at work. I would pay more for higher quality drives that don't crap out. Maybe the new solid state drives will be a better solution when prices come down.

http://www.adtron.com/newsroom/25fb-...tate-Disk.html

[Jeffrey Butler]

Originally Posted by Douglas Villalba

I must be lucky.

Lucky? No - it's ONLY seven years old....as long as that would seem, and it is midly impressive, seven years is only the beginning for an archive. For a backup, it's probably pretty good.

[Robert Lane]

Jeffrey,

The perspective of "100% failure rate" isn't relevant because at some point literally everything on this planet will either cease to stop working, wear-out or die, nothing is eternal. The concept of archival storage is something that will have a viable shelf life for a specified time period.

The problem is that we are used to physical storage for archives, obviously analog terms we have archives that are literally thousands of years old; papers, etchings, inks, metals etc. Films have also proven to be an archival media - when stored properly, however in archival terms there is no digital-based media that has yet proven itself to true archival standards for 2 simple reasons; not enough time has passed since the creation of the very first digital media to prove it's true, long-term archival worthiness when compared to analog/physical methods and, because of the nature of consumer "consumable" technology media types keep evolving at an exponential rate and we keep getting newer/faster/better media all the time.

Let's not forget that the second half to the archival-storage question is future access. I know for a fact that any human generations from now will be able to dig up my film archives and look at them. Will they also be able to access my HDD's or optical media or tapes? Absolutely not - not unless some historical society dedicated themselves to keeping around devices and power supplies that would allow it.


Let's face it, we - the end users - have created our own conundrum of media choices. We constantly demand faster/better/more. That means those brilliant engineers are always thinking up something new, so that means that we have NO choice but in another 10-15 years be forced into migrating our archives - from whatever media they live on - onto something completely different and new, just so we can still access it.

Remember guys, that the concept of "archive" means that device - or media - is being written to only a few times and put away on a shelf and not touched unless needed for a backup or re-copy or original files.

In the case of HDD's that means *in theory* the device would never fail because 99.99% of it's life would be in a disconnected, powered-off state not sitting in a system constantly spinning. The notion that a HDD would not for some reason not spin-up in the future has long since gone away; those fears are based on drive-spindle technology that is more than 12 years old, not HDD's made with current spindle designs.

But again, a true archive methodology also means that you're not putting your eggs into one basket, so your archive should have a diverse, redundant copy regardless what media it's lives on. You don't want to rely on just one HDD to spin up later or not to be dropped hard and ruin the heads, you don't want to trust that any single optical media won't become scratched/cracked etc. The sad fact is, the failure for any media to perform in the distant future is almost always due to human error or mis-use, not failure of the media itself.

The moral of the "archive conundrum": Pick the solution that fits your current infrastructure, client requirements, budget and comfort level. Then go shoot something so you'll have something to archive later. (^_^)