Tapless camera archiving solutions at DVinfo.net

Marcello Mazzilli · May 17th, 2009, 12:33 PM

I am wondering what are other people's workflows in handling tapless cameras. I shoot on an HM700 now. I have jobs that keep me out for 3-4 days so I bought 6 SDHC 16Gb cards (and I have 2 8 GB cards I got with the camera). I prefer not to fiddle with disks, portable PCs, etc.. in the middle of the job so I just go from card 1 to 2, from 2 to 3 etc... In this months I never had to download something half way through a job. When I get home I decided to archive everything on Blueray. 25 Gb are goot for almost 1h30' and having found Verbatim disks at around 5,56 € (+ VAT) I can say that I can archive an hour at 3,50 € almost what I was doing with tapes before. My problem. Reliability. How will these first generation Bluray disks last? Can I trust them? Some people have other choices?

1) External HD? (just little less than BD, faster, but maybe less practical)
2) Buy new cards every job (cost like a DigiBeta or so)
3) Copy every card to cheaper Class4 ones just for archiving
4) Archive only good (actually used in project) footage through "project compacting" features of the editing software ?
5) Don't archive footage.. just rendered master

Regarding solutions 4 and 5 I must tell you that in my life I never (never!) used twice a tape and kept all DVs (or HDVs) of my jobs and really I recommend taht to everybody. Many times clients called after years asking for a re-editing of a job or a new editing on old footage

Brett Sherman · May 17th, 2009, 03:41 PM

It's a brave new world. I haven't switched yet to tapeless, but for all practical purposes now tapes are a one time capture for me. I use hard drives with a non-realtime mirror. Meaning I have an active hard drive and a backup hard drive. The backup hard drive is synced periodically, daily if I'm currently working on a project. This is both my footage and project media. I find this method of backup protects against the most types of failure - media failure (covered as long as both drives don't fail at the same time), file system failure (this is actually quite common and RAIDs don't protect against this at all), accidental deletion, and once the projects on the drive are retired, I can store the mirror in a separate location protecting against fire.

Unfortunately hard drives don't last long, so you have to replace them every 5 years. An additional backup of the compressed project on Blu-ray would be a good idea in addition, I just don't have a Blu-ray burner yet. I don't have any idea about the reliability of BD-Rs, but at least there are no moving parts. I also don't have any idea on the long term viability of SD cards. My suspicion is that anything optical beats anything electronic, but it would be nice to have some data on real-world reliability.

Brian Standing · May 17th, 2009, 08:06 PM

If you want archival-quality data storage with a proven track record of reliability for years while sitting on the shelf, there's really only one answer: magnetic data tape.

It may seem crazy to go from tape to tapeless, just to archive back to tape, but it really makes a lot of sense. LTO tape drives are fast, the media is reasonably priced and hold up to 800GB per cartridge. Plus the LTO format is open-source, so you don't have to worry about the company disappearing and taking its proprietary format with it.

I picked up a reconditioned LTO-2 tape drive for $500 and use it to archive all my Cineform intermediates. Great investment and I breathe easier.

Doug Tessler · May 17th, 2009, 08:34 PM

This is the only way to go Get a cheap external hardrive. Tape drives are slow and expensive and you will be dealing with terabytes . You ca pick up a 1 terybyte drive for a couple of hundred bucks

Doug

Marcello Mazzilli · May 18th, 2009, 05:47 AM

LTO ? Oh.. no..not again tapes! Also... GBytes are almost the same than on external HD (Now you can find 1,5 Tb for 100 €). If you use it just for backup I think it can las "for-ever". I agree with Doug.. probably at present external HD are better than Blu-rays.. but I like the practical issue that I can separate single footage from different jobs, but it in a box a bring it to the client. Still.. probably, at least until BR have proven some reliabilty, I should make an exernal HD copy.... Mmmm....Maybe tapes were not so bad after all ! :)

Brian Standing · May 18th, 2009, 07:38 AM

Quote:

Originally Posted by Marcello Mazzilli

LTO ? Oh.. no..not again tapes! Also... GBytes are almost the same than on external HD (Now you can find 1,5 Tb for 100 €). If you use it just for backup I think it can las "for-ever". ... Mmmm....Maybe tapes were not so bad after all ! :)

I think the best strategy is not to put all your eggs in one basket. So, I've been backing up all my digital masters to an internal hard drive (I'm operating on the theory that firing up the driive on a regular basis will prolong its life, rather than letting it sit on a shelf). I have archived, trimmed Cineform files for each project, plus the digital masters also saved to LTO tape. When (not if) the drives fail, I can replace them from the LTO tape.

I'm still using HDV tape, so I actually have a triple-redundant backup of my most important files. Maybe I'm paranoid, but since much of my work is irreplaceable documentary footage, I'd rather be safe than sorry.

Brett Sherman · May 18th, 2009, 07:16 PM

Tape drives work for active backups, but aren't good for archiving IMHO.

The problem with archiving to LTO tapes is that it often requires special software and special hardware to retrieve it. If the software you bought is no longer compatible with your OS 10 years from now, you're SOL. And if your tape drive doesn't have modern drivers or breaks you're also SOL.

I have 7 year old backups to DVD that I can't access now because Retrospect only works with specific DVD Recorders and I no longer have the recorder I originally used. This is what happens when you go with proprietary format.

Archives should be in a universal, easily accessed format - Magnetic Data Tapes are neither.

Marcello Mazzilli · May 19th, 2009, 04:57 AM

To Brian... ah ah.. you have HDV... I too had HDV until lats month.. the problem is now with th HM700 and totally tapeless footage... We'll see what happens. I'll make Blu-Rays and keep external disks for some time.. hoping BR will be reliable

Brian Standing · May 19th, 2009, 06:12 AM

Quote:

Originally Posted by Brett Sherman

If the software you bought is no longer compatible with your OS 10 years from now, you're SOL. And if your tape drive doesn't have modern drivers or breaks you're also SOL.

Archives should be in a universal, easily accessed format - Magnetic Data Tapes are neither.

There's a good reason why film is still considered the ideal archival medium. A caveman could probably figure out what to do with a reel of 35mm film.

I agree that a good, open-source backup software standard is sorely needed. Until then, I'm using the Microsoft NT Backup utility, because I figure it has the largest user base, it's free, and is the most likely to be supported years from now. I've read of some problems restoring data with NTBackup, but most of them have to do with restoring boot drive images. Since I'm just backing up data, with no compression, I'm keeping my fingers crossed there won't be any issues.

Part of the LTO standard requires backward compatibility for at least two generations. So my LTO-2 tapes should be readable by LTO-4 machines. I'm also banking on the very widespread use of LTO tapes in the corporate world, so that if my machine ever dies, there will be plenty of replacements or parts. If worst came to worst, I could always have a separate boot partition or a separate computer running whatever OS I need to access the equipment.

Part of any long-term archive strategy should probably involve transferring data onto new media periodically, so I could upgrade to LTO-4, recopy the tapes onto LTO-4 media, and keep myself going for another 2 generations.

Marcello, yes, I am still using HDV tape. But, I'm trying to set myself up for a successful backup strategy when I do make the switch to primarily tapeless. If I want to keep triple redundancy, I can just make two LTO tape backups when I no longer have HDV tapes to save.

It's all working so far. Ask me again in 50 years.

Robert M Wright · May 19th, 2009, 09:03 AM

A little while ago, I was experimenting and found I could compress AVCHD m2ts files (exported from Edius, not directly from a camera) significantly, with 7-ZIP (lossless, like WINZIP). I started a thread on it here (but nobody replied):

http://www.dvinfo.net/conf/panasonic...2ts-files.html

Whatever the archiving media is, if that kind of compression can be consistently achieved, it certainly would help cut the cost of archiving footage.

Brian Standing · May 19th, 2009, 01:21 PM

That's interesting, Robert. I would have expected that AVCHD files would be so heavily compressed that there wouldn't be much more room you could squeeze out of them. How long did compression with 7-Zip take on your machine?

I may try something similar with HDV m2t and Cineform files...

Is there any risk of errors in decompression, I wonder?

Jeff Kellam · May 19th, 2009, 03:34 PM

If you start looking at the cost per GB on BD and hard drives (or DV tape for DV shooters) you can see that hard drives are about half the cost per GB than tape or BD.

Hard drives are $11.53 per 100GB

Mini DV tape is $40 to $80 per 100GB (13GB per tape)

Blu-Ray discs are $20 & up per 100GB (25GB disc @ $5.00)

Im running a 1TB data drive in my main machine which I back up weekly to an unattached 1TB drive kept in a safe. When they both fill up, I start a new set. Im currently archiving everything, the clips, renders, VOBs, VEGs, deshaker DIR, everything. If I ever need to revisit a project, it's totally intact.

Sure, sometimes I skip a week between backups, but the bulk of my data is saved in duplicate.

At some point in the near future, I will use the newer 2TB drives (as the price comes down) and make another level of consolidated backup.

I think tapeless cameras BU is pretty easy and the files are more accessable. It also forces you to backup your digital photos regularly. I can easily view all of my tapeless projects I have ever done. I can't do anything with my old XH-A1 projects.

Robert M Wright · May 20th, 2009, 06:18 AM

Compression of the AVCHD m2ts files (from Edius) was pretty fast. I experimented with compressing HDV m2t files (captured from an HV20) also. The best I could do was 89% of original file size with the m2t files, using "PPMd" compression, which was awfully slow.

The lowest cost (per unit of storage) media I know of, is single layer DVD, at about $5 for 100GB.

Jeff Kellam · May 20th, 2009, 07:26 AM

Quote:

Originally Posted by Robert M Wright

The lowest cost (per unit of storage) media I know of, is single layer DVD, at about $5 for 100GB.

24 cents each for 21 discs would be about $5.00

It would take 213 discs to store 1TB.

May 17th, 2009, 12:33 PM	#1
Marcello Mazzilli Major Player Join Date: Apr 2009 Location: ITALIA Posts: 416	Tapless camera archiving solutions I am wondering what are other people's workflows in handling tapless cameras. I shoot on an HM700 now. I have jobs that keep me out for 3-4 days so I bought 6 SDHC 16Gb cards (and I have 2 8 GB cards I got with the camera). I prefer not to fiddle with disks, portable PCs, etc.. in the middle of the job so I just go from card 1 to 2, from 2 to 3 etc... In this months I never had to download something half way through a job. When I get home I decided to archive everything on Blueray. 25 Gb are goot for almost 1h30' and having found Verbatim disks at around 5,56 € (+ VAT) I can say that I can archive an hour at 3,50 € almost what I was doing with tapes before. My problem. Reliability. How will these first generation Bluray disks last? Can I trust them? Some people have other choices? 1) External HD? (just little less than BD, faster, but maybe less practical) 2) Buy new cards every job (cost like a DigiBeta or so) 3) Copy every card to cheaper Class4 ones just for archiving 4) Archive only good (actually used in project) footage through "project compacting" features of the editing software ? 5) Don't archive footage.. just rendered master Regarding solutions 4 and 5 I must tell you that in my life I never (never!) used twice a tape and kept all DVs (or HDVs) of my jobs and really I recommend taht to everybody. Many times clients called after years asking for a re-editing of a job or a new editing on old footage

May 17th, 2009, 08:06 PM	#3
Brian Standing Major Player Join Date: Dec 2003 Location: Madison, Wisconsin Posts: 621	If you want archival-quality data storage with a proven track record of reliability for years while sitting on the shelf, there's really only one answer: magnetic data tape. It may seem crazy to go from tape to tapeless, just to archive back to tape, but it really makes a lot of sense. LTO tape drives are fast, the media is reasonably priced and hold up to 800GB per cartridge. Plus the LTO format is open-source, so you don't have to worry about the company disappearing and taking its proprietary format with it. I picked up a reconditioned LTO-2 tape drive for $500 and use it to archive all my Cineform intermediates. Great investment and I breathe easier. __________________ http://www.prolefeedstudios.com/blog/ Documentary for the masses!

May 17th, 2009, 08:34 PM	#4
Doug Tessler Major Player Join Date: Nov 2006 Location: BOCa raton fl Posts: 342	get a cheap external hardrive This is the only way to go Get a cheap external hardrive. Tape drives are slow and expensive and you will be dealing with terabytes . You ca pick up a 1 terybyte drive for a couple of hundred bucks Doug

May 19th, 2009, 01:21 PM	#11
Brian Standing Major Player Join Date: Dec 2003 Location: Madison, Wisconsin Posts: 621	That's interesting, Robert. I would have expected that AVCHD files would be so heavily compressed that there wouldn't be much more room you could squeeze out of them. How long did compression with 7-Zip take on your machine? I may try something similar with HDV m2t and Cineform files... Is there any risk of errors in decompression, I wonder? __________________ http://www.prolefeedstudios.com/blog/ Documentary for the masses!

May 19th, 2009, 03:34 PM	#12
Jeff Kellam Major Player Join Date: Sep 2006 Location: Virginia Beach, VA Posts: 656	If you start looking at the cost per GB on BD and hard drives (or DV tape for DV shooters) you can see that hard drives are about half the cost per GB than tape or BD. Hard drives are $11.53 per 100GB Mini DV tape is $40 to $80 per 100GB (13GB per tape) Blu-Ray discs are $20 & up per 100GB (25GB disc @ $5.00) Im running a 1TB data drive in my main machine which I back up weekly to an unattached 1TB drive kept in a safe. When they both fill up, I start a new set. Im currently archiving everything, the clips, renders, VOBs, VEGs, deshaker DIR, everything. If I ever need to revisit a project, it's totally intact. Sure, sometimes I skip a week between backups, but the bulk of my data is saved in duplicate. At some point in the near future, I will use the newer 2TB drives (as the price comes down) and make another level of consolidated backup. I think tapeless cameras BU is pretty easy and the files are more accessable. It also forces you to backup your digital photos regularly. I can easily view all of my tapeless projects I have ever done. I can't do anything with my old XH-A1 projects. __________________ Panasonic HMC150/Canon A1/JVC HD1/Sony Vegas 8.0c

May 17th, 2009, 03:41 PM	#2
Brett Sherman Major Player Join Date: Mar 2006 Location: Mount Rainier, MD Posts: 428	It's a brave new world. I haven't switched yet to tapeless, but for all practical purposes now tapes are a one time capture for me. I use hard drives with a non-realtime mirror. Meaning I have an active hard drive and a backup hard drive. The backup hard drive is synced periodically, daily if I'm currently working on a project. This is both my footage and project media. I find this method of backup protects against the most types of failure - media failure (covered as long as both drives don't fail at the same time), file system failure (this is actually quite common and RAIDs don't protect against this at all), accidental deletion, and once the projects on the drive are retired, I can store the mirror in a separate location protecting against fire. Unfortunately hard drives don't last long, so you have to replace them every 5 years. An additional backup of the compressed project on Blu-ray would be a good idea in addition, I just don't have a Blu-ray burner yet. I don't have any idea about the reliability of BD-Rs, but at least there are no moving parts. I also don't have any idea on the long term viability of SD cards. My suspicion is that anything optical beats anything electronic, but it would be nice to have some data on real-world reliability.

May 18th, 2009, 05:47 AM	#5
Marcello Mazzilli Major Player Join Date: Apr 2009 Location: ITALIA Posts: 416	LTO ? Oh.. no..not again tapes! Also... GBytes are almost the same than on external HD (Now you can find 1,5 Tb for 100 €). If you use it just for backup I think it can las "for-ever". I agree with Doug.. probably at present external HD are better than Blu-rays.. but I like the practical issue that I can separate single footage from different jobs, but it in a box a bring it to the client. Still.. probably, at least until BR have proven some reliabilty, I should make an exernal HD copy.... Mmmm....Maybe tapes were not so bad after all ! :)

May 18th, 2009, 07:16 PM	#7
Brett Sherman Major Player Join Date: Mar 2006 Location: Mount Rainier, MD Posts: 428	Tape drives work for active backups, but aren't good for archiving IMHO. The problem with archiving to LTO tapes is that it often requires special software and special hardware to retrieve it. If the software you bought is no longer compatible with your OS 10 years from now, you're SOL. And if your tape drive doesn't have modern drivers or breaks you're also SOL. I have 7 year old backups to DVD that I can't access now because Retrospect only works with specific DVD Recorders and I no longer have the recorder I originally used. This is what happens when you go with proprietary format. Archives should be in a universal, easily accessed format - Magnetic Data Tapes are neither.

May 19th, 2009, 04:57 AM	#8
Marcello Mazzilli Major Player Join Date: Apr 2009 Location: ITALIA Posts: 416	To Brian... ah ah.. you have HDV... I too had HDV until lats month.. the problem is now with th HM700 and totally tapeless footage... We'll see what happens. I'll make Blu-Rays and keep external disks for some time.. hoping BR will be reliable

May 19th, 2009, 09:03 AM	#10
Robert M Wright Inner Circle Join Date: Jan 2006 Location: Minnesota (USA) Posts: 2,171	A little while ago, I was experimenting and found I could compress AVCHD m2ts files (exported from Edius, not directly from a camera) significantly, with 7-ZIP (lossless, like WINZIP). I started a thread on it here (but nobody replied): http://www.dvinfo.net/conf/panasonic...2ts-files.html Whatever the archiving media is, if that kind of compression can be consistently achieved, it certainly would help cut the cost of archiving footage.

May 20th, 2009, 06:18 AM	#13
Robert M Wright Inner Circle Join Date: Jan 2006 Location: Minnesota (USA) Posts: 2,171	Compression of the AVCHD m2ts files (from Edius) was pretty fast. I experimented with compressing HDV m2t files (captured from an HV20) also. The best I could do was 89% of original file size with the m2t files, using "PPMd" compression, which was awfully slow. The lowest cost (per unit of storage) media I know of, is single layer DVD, at about $5 for 100GB.