Archiving XD Cam EX Footage

Thomas Smet · August 9th, 2007, 06:56 AM

I agree with Alister. You would have to at least once a year hook up all your drives to check to make sure they still work. The older they get the more you will have to do this. The older the drives get the better the chances of the two backup drives not working either.

For short term storage such as less then five years hard drives may work very well. For long term however after five years the risk gets higher and higher of all drives failing.

As an example I recently put together an older system to use to experiment some of my own programs on. For this system I dusted off some older hard drives I kept packed in a box. Some of these drives were a little over five years old and sure enough some of them died right away. A few of the other ones died a few weeks after that. If the drives were a few years older perhaps none of them would have worked at all. Of course I do have one SCSI drive that is almost eight years old still working well but I do not trust it to anything important.

Of course that means actually hooking up all the drives to make sure they still work. How else will you know if one has died to quickly make a new backup of? Your shelf could be full of dozens of drives that have died and you may not even know it. By the time you go to use that drive it's backup may be dead as well and then you are SOL.

It's not the cost of hard drives that puts me off using them to store video, it's the constant checking them out and making new backups when they fail that freaks me out. I would have to eventually hire an IT guy just to check my drives all the time.

Kevin Shaw · August 9th, 2007, 07:37 AM

Fair enough, but isn't it also true that we don't know how reliable any of the recordable disc formats are going to be over time? I can see how even duplicate hard drives could go bad after sitting around for a while, but I wouldn't consider that any riskier than any single-copy archiving solution. The ideal thing would be duplicate backups on different types of media to avoid similar failure risks, but then that's a pain to manage.

By the way, a rigorous data protection plan would call for having multiple copies at physically separate locations, for what that's worth.

Alister Chapman · August 9th, 2007, 12:13 PM

I agree that we probably do not in reality know how long XDCAM (or any other recent format) disks will last. They should be good for 50 years, but by then XDCAM drives are probably going to be pretty rare! The beauty of XDCAM disks is that it is just the disk in a cartridge. The mechanics and electronics are in the drive, if the drive fails you can get another one and nothing is lost. XDCAM disks are waterproof and un affected by humidity so even if your home or office floods you footage should be safe. They can also stand high temperatures and cold, so all in all a very robust medium. I keep working copies of my material on hard drives with backups on XDCAM disks stored in separate locations. My most valuable footage is also duplicated and held by 3rd parties for peace of mind.

Chris Medico · August 9th, 2007, 02:42 PM

Also if we are just talking about raw data then don't discount other optical technologies that when they become fully main stream will be less expensive than a niche product like XDCAM.

A dual layer BlueRay disk will hold 50gig. Once fully main stream and low cost it will be a viable choice for backup of video files for the compressed formats including XDCAM.

Chris

Brett Sherman · August 19th, 2007, 04:10 PM

Quote:

Originally Posted by Chris Medico

Also if we are just talking about raw data then don't discount other optical technologies that when they become fully main stream will be less expensive than a niche product like XDCAM.

Yeah but by the time writeable Blu-Rays become common and cheap we'll be at $150 per Terabyte with hard disk drives. The march of cost/GB of hard drives will continue to obliterate other options. Even if you replace your hard drives every 6-7 years or so remember that by that time the capacity of the drives will have increased and price will have dropped significantly. Just think back to 2000 and what you thought was a "large" hard disk then is a joke now. Also, because the capacity of hard drives is so much greater than individual optical discs, it will also be a lot easier to manage.

So I'd go with hard drives for archiving and just plan to replace them periodically. Also get a hot swappable drive enclosure to make life easier.

Simon Wyndham · August 19th, 2007, 04:23 PM

Quote:

A dual layer BlueRay disk will hold 50gig. Once fully main stream and low cost it will be a viable choice for backup of video files for the compressed formats including XDCAM.

Not quite the same. Consumer Blu_ray doesn't have all the error correction and robust file system, or the dust sealing of the XDCAM discs.

Greg Boston · August 19th, 2007, 11:35 PM

Quote:

Originally Posted by Kevin Shaw

Any single archiving solution can't be considered infallible, so if you want security then duplicate backups are essential...and two hard drives are about as good as anything else. The odds of two drives both failing while sitting on a shelf are pretty slim.

Agree with multiple back-ups. But here's another little item I just remembered from my days as a systems builder. As you know, the drives have a low level format and high level format. The low level format was the drive and the controller being hitched up together. The high level format was what the OS wanted to do (FDISK on DOS/Windows). I found that drives which sat around for extended periods would develop read errors. It was explained to me that the low level formatting information can become corrupted due to the ever so slow loss of the magnetic charges on the disc.

Granted, this was many years ago so maybe the retenticity has improved over the years. But it was a potential liability then. The solution was to back up the drive annually and go do a low level format to refresh that part of the equation. But these were 20 mb drives and it didn't take all that long. BTW, here's the trivia portion. Most controllers had their low level format routine started by opening Debug and entering G=C800:5.

-gb-

Alister Chapman · August 21st, 2007, 07:02 AM

Bigger and bigger drives means more and more material will be lost because of a single failure. I would never fully trust a hard drive. I've had too many fail on me, I have a large box of dead HD's, many of them only 2 or 3 years old.

I want to put my footage on a medium I can put away in a box and forget about until I need it. That's the way it is with my tape and XDCAM disk footage. I don't want to keep dumping from one HD to another HD, that's how footage gets lost or corrupted. If you copy 500 gigs of footage from one drive to another are you going to go through every file to make sure none of them are corrupted each time? It happens, and every time you have to make another copy you risk adding more errors or loosing more files.

That's why Banks and others with crucial data still use tape for backups and companies like EXABYTE are still in business making tape backup systems. Failures are rarely catastrophic with tape, unlike hard drives.

Brett Sherman · August 23rd, 2007, 07:54 AM

I suppose it all depends on your risk/cost calculation goes. Yes, I've had hard drives fail. But that's why you'd backup to two drives. I'd never suggest a backup system that relied on one hard disk drive. I still believe the likelihood of two drives failing within 6 or 7 years is so small it's not worth worrying about. As far as corruption or losing files during transfer goes, if you do a verified backup it virtually eliminates that risk.

Magnetic tape also fails pretty frequently in about 20 years. I have played Betacam tapes from the early 90s that clog the heads within 30 seconds of playback making them virtually useless.

Alex Leith · August 24th, 2007, 07:36 PM

Yeah, the actual "rated" lifespan of magnetic tape is pretty short. You'll see dropouts (or at least lossy drop-out compensation) on DV tapes within 5 years - unless you have a 5°C 20%RH storage facility, of course!

Give me hard drives or optical media any day!

Dave Bingham · September 4th, 2007, 06:40 PM

This may not be a solution for everyone, but a few years ago I bought a Quantum LTO-2 tape backup (the successor to DLT). There is now LTO-3 which is even faster. It backs up onto a 200GB tape at about 1 GB per minute. The tape cartridges are only $34.00 (the unit is $1,200.00) and the tapes have a 30+ years life span. LTO is used by most Fortune 500 companies for reliable data backup. An Atto SCSI card is about $300.00-450.00. This really works well for me.

I went to the LA FCP users group meeting in January (2007) at MacWorld. There were two guys who had just finished editing the film Zodiac for Paramount. They were capturing directly onto hard drives via FCP and backing up onto LTO-3. At the same event the Panasonic rep showed everyone the new DVCPRO HD deck. It was a mere $25,000.00. Just think about it. We’ll never have to purchase another expensive deck again, never have to worry about tape dropout, etc.

Andrew Kimery · September 4th, 2007, 10:14 PM

Quote:

Originally Posted by Brett Sherman

I suppose it all depends on your risk/cost calculation goes. Yes, I've had hard drives fail. But that's why you'd backup to two drives. I'd never suggest a backup system that relied on one hard disk drive. I still believe the likelihood of two drives failing within 6 or 7 years is so small it's not worth worrying about. As far as corruption or losing files during transfer goes, if you do a verified backup it virtually eliminates that risk.

What I'm reading more and more though is, as others have said here, that HDDs need to be spun up/accessed occasionally to keep it in proper working order. Though, if you only have a few external HDDs in enclosures then having a "spin up day" worked into the calendar a few times a year shouldn't be to big a deal. I'd still be ready to rotate all the data onto new storage devices or mediums (depending on what's come out) every 5 years or so though.

Me personally, I'm looking into tape options like Dave Bingham talked about.

-A

Brad Vaughan · December 18th, 2007, 01:49 PM

Hmm, the LTO-3 seems rather interesting.

Now can you back up the full HD (1920) file or does it get compressed to 1440 rez when transferred to tape for back-up?

Piotr Wozniacki · December 18th, 2007, 01:55 PM

Quote:

Originally Posted by Brad Vaughan

Hmm, the LTO-3 seems rather interesting.

Now can you back up the full HD (1920) file or does it get compressed to 1440 rez when transferred to tape for back-up?

Why not transfer as data? Completely format-independent.

Brad Vaughan · December 18th, 2007, 03:03 PM

Quote:

Originally Posted by Piotr Wozniacki

Why not transfer as data? Completely format-independent.

Hmmm, didn't know that was possible with tape.

I have never transfered Data to Tape as a back up... other than the tape used for capturing footage in my camera of course.

August 9th, 2007, 12:13 PM	#18
Alister Chapman Inner Circle Join Date: Dec 2005 Location: Bracknell, Berkshire, UK Posts: 4,957	I agree that we probably do not in reality know how long XDCAM (or any other recent format) disks will last. They should be good for 50 years, but by then XDCAM drives are probably going to be pretty rare! The beauty of XDCAM disks is that it is just the disk in a cartridge. The mechanics and electronics are in the drive, if the drive fails you can get another one and nothing is lost. XDCAM disks are waterproof and un affected by humidity so even if your home or office floods you footage should be safe. They can also stand high temperatures and cold, so all in all a very robust medium. I keep working copies of my material on hard drives with backups on XDCAM disks stored in separate locations. My most valuable footage is also duplicated and held by 3rd parties for peace of mind. __________________ Alister Chapman, Film-Maker/Stormchaser http://www.xdcam-user.com/alisters-blog/ My XDCAM site and blog. http://www.hurricane-rig.com

August 9th, 2007, 02:42 PM	#19
Chris Medico Inner Circle Join Date: Aug 2006 Location: Efland NC, USA Posts: 2,322	Also if we are just talking about raw data then don't discount other optical technologies that when they become fully main stream will be less expensive than a niche product like XDCAM. A dual layer BlueRay disk will hold 50gig. Once fully main stream and low cost it will be a viable choice for backup of video files for the compressed formats including XDCAM. Chris __________________ http://www.LandYachtMedia.com

August 21st, 2007, 07:02 AM	#23
Alister Chapman Inner Circle Join Date: Dec 2005 Location: Bracknell, Berkshire, UK Posts: 4,957	Bigger and bigger drives means more and more material will be lost because of a single failure. I would never fully trust a hard drive. I've had too many fail on me, I have a large box of dead HD's, many of them only 2 or 3 years old. I want to put my footage on a medium I can put away in a box and forget about until I need it. That's the way it is with my tape and XDCAM disk footage. I don't want to keep dumping from one HD to another HD, that's how footage gets lost or corrupted. If you copy 500 gigs of footage from one drive to another are you going to go through every file to make sure none of them are corrupted each time? It happens, and every time you have to make another copy you risk adding more errors or loosing more files. That's why Banks and others with crucial data still use tape for backups and companies like EXABYTE are still in business making tape backup systems. Failures are rarely catastrophic with tape, unlike hard drives. __________________ Alister Chapman, Film-Maker/Stormchaser http://www.xdcam-user.com/alisters-blog/ My XDCAM site and blog. http://www.hurricane-rig.com

August 24th, 2007, 07:36 PM	#25
Alex Leith Major Player Join Date: Mar 2005 Location: Northampton, England Posts: 500	Yeah, the actual "rated" lifespan of magnetic tape is pretty short. You'll see dropouts (or at least lossy drop-out compensation) on DV tapes within 5 years - unless you have a 5°C 20%RH storage facility, of course! Give me hard drives or optical media any day! __________________ Alex

September 4th, 2007, 06:40 PM	#26
Dave Bingham New Boot Join Date: Sep 2007 Location: United States Posts: 8	Tape backup for video This may not be a solution for everyone, but a few years ago I bought a Quantum LTO-2 tape backup (the successor to DLT). There is now LTO-3 which is even faster. It backs up onto a 200GB tape at about 1 GB per minute. The tape cartridges are only $34.00 (the unit is $1,200.00) and the tapes have a 30+ years life span. LTO is used by most Fortune 500 companies for reliable data backup. An Atto SCSI card is about $300.00-450.00. This really works well for me. I went to the LA FCP users group meeting in January (2007) at MacWorld. There were two guys who had just finished editing the film Zodiac for Paramount. They were capturing directly onto hard drives via FCP and backing up onto LTO-3. At the same event the Panasonic rep showed everyone the new DVCPRO HD deck. It was a mere $25,000.00. Just think about it. We’ll never have to purchase another expensive deck again, never have to worry about tape dropout, etc.

August 9th, 2007, 06:56 AM	#16
Thomas Smet Trustee Join Date: Mar 2004 Location: Milwaukee, WI Posts: 1,719	I agree with Alister. You would have to at least once a year hook up all your drives to check to make sure they still work. The older they get the more you will have to do this. The older the drives get the better the chances of the two backup drives not working either. For short term storage such as less then five years hard drives may work very well. For long term however after five years the risk gets higher and higher of all drives failing. As an example I recently put together an older system to use to experiment some of my own programs on. For this system I dusted off some older hard drives I kept packed in a box. Some of these drives were a little over five years old and sure enough some of them died right away. A few of the other ones died a few weeks after that. If the drives were a few years older perhaps none of them would have worked at all. Of course I do have one SCSI drive that is almost eight years old still working well but I do not trust it to anything important. Of course that means actually hooking up all the drives to make sure they still work. How else will you know if one has died to quickly make a new backup of? Your shelf could be full of dozens of drives that have died and you may not even know it. By the time you go to use that drive it's backup may be dead as well and then you are SOL. It's not the cost of hard drives that puts me off using them to store video, it's the constant checking them out and making new backups when they fail that freaks me out. I would have to eventually hire an IT guy just to check my drives all the time.

August 9th, 2007, 07:37 AM	#17
Kevin Shaw Inner Circle Join Date: Sep 2004 Location: Sacramento, CA Posts: 2,488	Fair enough, but isn't it also true that we don't know how reliable any of the recordable disc formats are going to be over time? I can see how even duplicate hard drives could go bad after sitting around for a while, but I wouldn't consider that any riskier than any single-copy archiving solution. The ideal thing would be duplicate backups on different types of media to avoid similar failure risks, but then that's a pain to manage. By the way, a rigorous data protection plan would call for having multiple copies at physically separate locations, for what that's worth.

August 23rd, 2007, 07:54 AM	#24
Brett Sherman Major Player Join Date: Mar 2006 Location: Mount Rainier, MD Posts: 428	I suppose it all depends on your risk/cost calculation goes. Yes, I've had hard drives fail. But that's why you'd backup to two drives. I'd never suggest a backup system that relied on one hard disk drive. I still believe the likelihood of two drives failing within 6 or 7 years is so small it's not worth worrying about. As far as corruption or losing files during transfer goes, if you do a verified backup it virtually eliminates that risk. Magnetic tape also fails pretty frequently in about 20 years. I have played Betacam tapes from the early 90s that clog the heads within 30 seconds of playback making them virtually useless.

December 18th, 2007, 01:49 PM	#28
Brad Vaughan Major Player Join Date: Dec 2006 Location: Va Beach, Va Posts: 241	Hmm, the LTO-3 seems rather interesting. Now can you back up the full HD (1920) file or does it get compressed to 1440 rez when transferred to tape for back-up?