SxS Workflow: Archive at DVinfo.net

Chuck Spaulding · November 3rd, 2008, 02:45 PM

Sorry, this is a bit off topic.

We are using several EX3 and our current workflow is:

Shoot
Backup the SxS (BPAV) by making a direct copy on set.
Archive the entire SxS using ClipBrowser 2x
Encode and distribute Quicktimes for FCP
Edit XDCAM as normal.
Hopefully send to color...

We need to archive about 2 to 3 terabytes per month, anyone have any suggestions on the best sort of media to accomplish this? I am thinking about LTO3 or LTO4 anyone using this?

We originally thought we would try Blu-Ray disk but I think it would take way to long and end up being cost prohibitive.

Any thoughts about this would be greatly appreciated.

Chris Leong · November 3rd, 2008, 07:22 PM

Hey Chuck
LTO is definitely the way to go.
I started out with an older LTO2 and am onto an LTO4 now.

However, I do operate a post facility - I'm doing an HD TV special right now with over 8TB of HDCam HD in FCP and have a dedicated Mac to handle the backups.

At work, they're onto the next step, which is an automated backline server system. This takes the media you flag for archiving and puts it into an LTO queue for backup to tape, and also has a request queue for unarchiving from LTO, all fully automated. (Well, before they had a full time assistant, which I guess one could call automaton, to an extent...)

Can't remember the name of the system offhand, but it holds 8 LTO4 tapes at a time, which would seem like a lot (at first - it always does...)

Be that what it may, the LTO system was designed to be backwards compatible, and I confirm that I was able successfully to unarchive some EX1 footage that I had backed up onto my LTO2 at home, but using the automated backline and one of the LTO4 drives at work.

Bottom line: functional, yes, reliable, very. Fast.... well... depends. Fast with respect to the 50 years' archival life of the tape?

HTH
Cheers
Chris

{edit} don't forget the double sided DVD-R for archiving. Very cost effective. I only use 8GB cards for that very reason. (Even though I do own a couple of 16GB cards, I only do scripted and commercials and come from film discipline-- ahem, background, so have only used the 16s as backups thus far).

Steve Gibbons · November 3rd, 2008, 07:46 PM

I would agree that an LTO4 based system is preferable over Blu-Ray data discs.

We have a Dell TL2000 24-tape LTO4 autolibrary that we use and it is very fast. Our workflow is a bit different though as we archive all our 8GB SxS raw video to HDCAM tape before ingest into our Avid Unity. The LTO4 library is more for short-term protection until the video is put to HDCAM. We'll probably re-visit this setup sometime soon when people are more comfortable editing Long GOP XDCAM stuff rather than DNxHD145.

SG

Jay Gladwell · November 4th, 2008, 07:10 AM

Chris, I'm just trying to learn/understand, so please don't think I have any agenda here...

Why is no one considering using 1TB hard drives as a form of archiving for massive amounts of video? One would think this would also address access times as well, both archiving and retrieving.

Mike Williams · November 4th, 2008, 07:58 AM

Hi Jay,

The main reason for not using HDs (hard drives) for archiving is the appearant issues that leaving them on the shelf may cause. There are some schools of thought that say an HD on the shelf will start to have issues after one year of no use. The tape is allegedly good for 20+ years. I hope that we will have an archival HD solution soon since backing up to tape just seems rediculous.

Jay Gladwell · November 4th, 2008, 08:07 AM

Quote:

Originally Posted by Mike Williams

There are some schools of thought that say an HD on the shelf will start to have issues after one year of no use.

Yes, I 've heard the same "rumor." However, my personal experience (and others I know personally) says otherwise.

Quote:

Originally Posted by Mike Williams

The tape is allegedly good for 20+ years. I hope that we will have an archival HD solution soon since backing up to tape just seems rediculous.

Agreed! Moving to solid-state recording only to store it on tape is, dare I say it? STUPID!

I was told directly by one of the engineers at SanDisk that they are working on a high capacity SD WORM card that will have an archive shelf live of 100 years. He he told me to watch for it in 2009.

Chuck Spaulding · November 4th, 2008, 10:24 AM

We must have twelve 1TB HD's that we've been using for about a year. Four of them have failed and a couple are having controller issues and you might have to cycle them on and off an indeterminate number of times before they come alive.

Although we have not lost any data yet, that's way too high a failure rate to trust for archiving. Also, not only do you have to worry about drives deteriorating but you also need to be concerned about obsolescence. I have my share of Bernoulli and Zip drives that work fine but the newer operating systems no-longer support.

I think part of the solution here depends on how much data you have to back up and the added importance that SxS cards adds to the equation. Currently we plan to do both tape and HD backup concurrently.

Chris Leong · November 4th, 2008, 10:35 AM

Jay
In my case I think it's more a case of monkey see, monkey do.

I have, I'd guess, around 16TB on line on an everyday basis. Around the same nearline (i.e. digitized on HDD but those drives not mounted on my desktop).

I needed something to back that up with, because the HDD access was getting, let's say, complicated for a home system. There's only so far you can go with eSATA backplanes, RAID cards, etc., before you have to think JBODs and network/Unity storage.

Since LTO was being used at work, and they were upgrading their systems, they offered me an older drive for cheap and since I knew their system worked well, it was a no-brainer.

And yes, I keep on buying more new drives for each new job, all the time, mainly 2TB RAIDs, because I'm at work most of the day and night and don't have much time spare for backups and restores. But when that job's done and dusted, I usually back the media and sessions up onto my LTOs and give those physical drives (and the responsibility for the long term data integrity) back to the client.

Oh, and like Chuck, I deal with HDD failures and near failures on a monthly if not a weekly basis. So I tend to have a spare HDD copy lying around for as long as I can, and when the time comes to trim down the HDD space, I will hold off for as long as possible and then get rid of all of it but still hang onto the actual final sessions and the media that those sessions use.

And one other thing. I usually only shoot 8GB cards, because I immediately back up the SxS media onto double sided DVD-Rs as soon as possible, then clone those DVD-Rs as soon as we get back to the studio. I try to have at least two copies of all original media at all times, stored in different places, and that's in addition to the two HDD copies and, eventually, the LTOs and single HDD stripped-down nearline backups.

So yes, the HDD system works very well, has done me grand.

At around 16x2TB drives and above, long term hicap storage became an issue for me that I solved the way most blue chip businesses and banks do - LTO.

But if push came to shove I'd say that my DVD-R system has been just as important to me, at least for short term media and session backup purposes.

Long term is LTO for now, though.

I'm working on the official documentary for the Library of Congress's new Packard Campus Audio Visual Conservation Center, which is interesting in that the LoC has finally gotten every single piece of their collection of everything audio visual from the 1800s onwards under one roof, and has started to digitize and store everything, and eventually make it all available online (copyrights etc permitting, of course).

Check it out at loc.gov.

I'll ask about their plans for long term storage, and I'll let you know what they say.

November 3rd, 2008, 02:45 PM	#1
Chuck Spaulding Trustee Join Date: Nov 2004 Location: Thousand Oaks Posts: 1,104	SxS Workflow: Archive Sorry, this is a bit off topic. We are using several EX3 and our current workflow is: Shoot Backup the SxS (BPAV) by making a direct copy on set. Archive the entire SxS using ClipBrowser 2x Encode and distribute Quicktimes for FCP Edit XDCAM as normal. Hopefully send to color... We need to archive about 2 to 3 terabytes per month, anyone have any suggestions on the best sort of media to accomplish this? I am thinking about LTO3 or LTO4 anyone using this? We originally thought we would try Blu-Ray disk but I think it would take way to long and end up being cost prohibitive. Any thoughts about this would be greatly appreciated.

November 3rd, 2008, 07:22 PM	#2
Chris Leong Major Player Join Date: Mar 2005 Location: Los Angeles CA USA Posts: 507	Hey Chuck LTO is definitely the way to go. I started out with an older LTO2 and am onto an LTO4 now. However, I do operate a post facility - I'm doing an HD TV special right now with over 8TB of HDCam HD in FCP and have a dedicated Mac to handle the backups. At work, they're onto the next step, which is an automated backline server system. This takes the media you flag for archiving and puts it into an LTO queue for backup to tape, and also has a request queue for unarchiving from LTO, all fully automated. (Well, before they had a full time assistant, which I guess one could call automaton, to an extent...) Can't remember the name of the system offhand, but it holds 8 LTO4 tapes at a time, which would seem like a lot (at first - it always does...) Be that what it may, the LTO system was designed to be backwards compatible, and I confirm that I was able successfully to unarchive some EX1 footage that I had backed up onto my LTO2 at home, but using the automated backline and one of the LTO4 drives at work. Bottom line: functional, yes, reliable, very. Fast.... well... depends. Fast with respect to the 50 years' archival life of the tape? HTH Cheers Chris {edit} don't forget the double sided DVD-R for archiving. Very cost effective. I only use 8GB cards for that very reason. (Even though I do own a couple of 16GB cards, I only do scripted and commercials and come from film discipline-- ahem, background, so have only used the 16s as backups thus far). __________________ "The content, not the container."

November 4th, 2008, 07:58 AM	#5
Mike Williams Major Player Join Date: Jun 2007 Location: USVI Posts: 232	Shelf life of the hard drives Hi Jay, The main reason for not using HDs (hard drives) for archiving is the appearant issues that leaving them on the shelf may cause. There are some schools of thought that say an HD on the shelf will start to have issues after one year of no use. The tape is allegedly good for 20+ years. I hope that we will have an archival HD solution soon since backing up to tape just seems rediculous.

November 4th, 2008, 10:35 AM	#8
Chris Leong Major Player Join Date: Mar 2005 Location: Los Angeles CA USA Posts: 507	Jay In my case I think it's more a case of monkey see, monkey do. I have, I'd guess, around 16TB on line on an everyday basis. Around the same nearline (i.e. digitized on HDD but those drives not mounted on my desktop). I needed something to back that up with, because the HDD access was getting, let's say, complicated for a home system. There's only so far you can go with eSATA backplanes, RAID cards, etc., before you have to think JBODs and network/Unity storage. Since LTO was being used at work, and they were upgrading their systems, they offered me an older drive for cheap and since I knew their system worked well, it was a no-brainer. And yes, I keep on buying more new drives for each new job, all the time, mainly 2TB RAIDs, because I'm at work most of the day and night and don't have much time spare for backups and restores. But when that job's done and dusted, I usually back the media and sessions up onto my LTOs and give those physical drives (and the responsibility for the long term data integrity) back to the client. Oh, and like Chuck, I deal with HDD failures and near failures on a monthly if not a weekly basis. So I tend to have a spare HDD copy lying around for as long as I can, and when the time comes to trim down the HDD space, I will hold off for as long as possible and then get rid of all of it but still hang onto the actual final sessions and the media that those sessions use. And one other thing. I usually only shoot 8GB cards, because I immediately back up the SxS media onto double sided DVD-Rs as soon as possible, then clone those DVD-Rs as soon as we get back to the studio. I try to have at least two copies of all original media at all times, stored in different places, and that's in addition to the two HDD copies and, eventually, the LTOs and single HDD stripped-down nearline backups. So yes, the HDD system works very well, has done me grand. At around 16x2TB drives and above, long term hicap storage became an issue for me that I solved the way most blue chip businesses and banks do - LTO. But if push came to shove I'd say that my DVD-R system has been just as important to me, at least for short term media and session backup purposes. Long term is LTO for now, though. I'm working on the official documentary for the Library of Congress's new Packard Campus Audio Visual Conservation Center, which is interesting in that the LoC has finally gotten every single piece of their collection of everything audio visual from the 1800s onwards under one roof, and has started to digitize and store everything, and eventually make it all available online (copyrights etc permitting, of course). Check it out at loc.gov. I'll ask about their plans for long term storage, and I'll let you know what they say. __________________ "The content, not the container."

November 3rd, 2008, 07:46 PM	#3
Steve Gibbons Regular Crew Join Date: Mar 2008 Location: Bridgenorth Posts: 116	I would agree that an LTO4 based system is preferable over Blu-Ray data discs. We have a Dell TL2000 24-tape LTO4 autolibrary that we use and it is very fast. Our workflow is a bit different though as we archive all our 8GB SxS raw video to HDCAM tape before ingest into our Avid Unity. The LTO4 library is more for short-term protection until the video is put to HDCAM. We'll probably re-visit this setup sometime soon when people are more comfortable editing Long GOP XDCAM stuff rather than DNxHD145. SG

November 4th, 2008, 07:10 AM	#4
Jay Gladwell Join Date: Jul 2002 Posts: 1,244	Chris, I'm just trying to learn/understand, so please don't think I have any agenda here... Why is no one considering using 1TB hard drives as a form of archiving for massive amounts of video? One would think this would also address access times as well, both archiving and retrieving.

November 4th, 2008, 10:24 AM	#7
Chuck Spaulding Trustee Join Date: Nov 2004 Location: Thousand Oaks Posts: 1,104	We must have twelve 1TB HD's that we've been using for about a year. Four of them have failed and a couple are having controller issues and you might have to cycle them on and off an indeterminate number of times before they come alive. Although we have not lost any data yet, that's way too high a failure rate to trust for archiving. Also, not only do you have to worry about drives deteriorating but you also need to be concerned about obsolescence. I have my share of Bernoulli and Zip drives that work fine but the newer operating systems no-longer support. I think part of the solution here depends on how much data you have to back up and the added importance that SxS cards adds to the equation. Currently we plan to do both tape and HD backup concurrently.