Archiving P2 Footage? at DVinfo.net

Calvin Dean · September 18th, 2007, 05:45 PM

After filling a P2 card with footage and dragging these clips to the hard drive, how do you archive this footage in case of future changes to the presentation? Obviously, you'll reuse the P2 card on the next project.

I'm considering buying a 200 because I like the workflow -- but I'm used to batch capturing from tapes. Plus, tapes are easy to store. What are your archival methods?

David Saraceno · September 18th, 2007, 06:10 PM

I use DVD9 for 8 GB cards

I also have three hard drives for redundant backup

Kaku Ito · September 18th, 2007, 09:59 PM

I tried many ways, blueray, dvdr, single drives, REV, G-raid and so on.
I'm probably settling for single hard drives (two copies), since any of the RAID without redundancy failed (like G-RAID at RAID level 0).
and maybe really important ones on blueray.

REV wasn't too bad only if the driver software does not conflict with your system (mine does, makes the finder go crash and comeback again and again).
I wish they make firewire version for 70GB drive.

Also, providing clips to, say like DVinfo, saved me from loosing everything.
My G-RAID crashed and ATTO R380 made the RAID volume getting broken, I lost data on RAIDs many times, then I tested RAID6, too but it gets really slow at the end of the capacity that won't even play back DVCPRO HD, I'm going to use RAID5 for the work volume, and a couple of single drives (same copies) to keep the original clips.

TingSern Wong · September 19th, 2007, 09:49 AM

Actually, the best archive media is still tape. Not those HD tapes, but, computer tapes. DLT, AIT-5, LTO, etc ... The best technology so far is LTO (Ultrium) tape. Today - you can get 800GB per tape cartridge at transfer speeds of up to 240MB/sec. How many DVDs can you store into one 800GB tape whose dimensions are 4" x 4" x 0.8"?

Francesco Dal Bosco · September 19th, 2007, 10:24 AM

It's my intention to buy a Blu Ray burner in order to store my P2 files.
Now i'm hearing about problems using this method. I really would like to know something more about it (I never suspected I could have troubles to
tranfer and archive P2 data on Blu ray or DVD disks).

David Saraceno · September 19th, 2007, 10:40 AM

What problems have you heard about blu ray archiving?

Francesco Dal Bosco · September 19th, 2007, 10:51 AM

David, thank you for your reply.
Maybe I've misunderstood the Kaku Ito post. I'm relatively new to
P2 workflow (my work is still for the most part on DV) and I'm
trying to learn everything I can on this forum.

Steve Rosen · September 19th, 2007, 11:53 AM

It is amazing to me that after 2 years there is not a tried and proven way to do this... I'm on the road a lot, and I shoot documentaries, 720/24pN, so I have a lot of material. My system is simple but time-consuming and intentionally over-redundant.

When I get back to the hotel room I have a Powerbook set up there with two good off-the-shelf 250g FW400 hard drives - they're cheap - about $120@. I offload the cards to one drive, then back up the folders to the other.. With the card removed from the computer I use P2 LogPro to check the files in both hard drives be be sure they aren't corrupted - then and only then do I re-format the cards in the camera. By the way, I do this myself rather than deligate it to an AC because I'm paranoid...

When they're full, I ship one drive via FedEx back to my office, and keep the other in my travel case... I have a number of drives - like four to eight - with me, as well as FedEx labels, so I can repeat this process.

When I get home I transfer the files from the drives to a FW800 500g drive in a Granite tray. I back up this drive to another 500g. I ingest the material I need into FCP using Log and Tranfer - Then and only then do I erase the files on the portable drives so I can reuse them - I reuse them about five or six times.

I keep both of the 500g drives on a shelf dedicated to storage.. If both drives die, which is not inconceivable, I'm f___ed...

Jon Fairhurst · September 19th, 2007, 12:36 PM

Quote:

Originally Posted by TingSern Wong

Today - you can get 800GB per tape cartridge at transfer speeds of up to 240MB/sec.

That's fast enough to record 1080i/60 or 720p/60 at 4:2:2 uncompressed.

Long, long ago I used tape for general computer backups at home. It wasn't a good solution, because it was slow and was of no use for finding single, small files. The only thing it was really good for was full disc restores, which I thankfully never needed to do. The other problem with this method is that it was only as good as your most recent backup. Let's say you did a backup a week ago, but need only a single file from last month. The choice was to rewind your life by a week or lose the file.

Hopefully, the latest software is smarter and more flexible. Also, with large video files, tape backup makes a lot more sense than storing general computer files.

LTO Ultrium would be a good match for a four drive RAID 0. Both have roughly the same transfer rates. The tape gives security, while the RAID 0 is good for working in the moment. The only mismatch is this: A good 4-drive RAID will have 3TB capacity, while the tape is 1/4 the size. That means either getting a four tape drive ($$$) or baby sitting the thing for a full transfer.

Mark Donnell · September 19th, 2007, 01:55 PM

I use external 500 GB hard drives for back-up. They are cheap ($ 120-140 on sale) and with two or three copies of a project it is highly unlikely that you will ever lose anything. For important stuff you can easily put one of these drives off-site in a safety deposit box or other location. For low-level raw material, I keep some on DVD.

TingSern Wong · September 19th, 2007, 07:55 PM

We are limited by the technology that is available at our time. Until we see higher densities random access h/w being made available (I read a holographic cube read/write by lasers that goes into the Petabytes range - that's 1,000 GB) that is affordable and not just labs stuff, the best stuff I see for archiving large amount of data is still tape. RAID 0 disks are good for working datasets. But after you are done with a current project, what do we use to move huge amount of data to slower, but, still viable media?

With 32GB P2 cards coming around the corner and 64GB sometime next year, even BlueRay Disks are not enough to archive one P2 card. Although the drive is expensive (one off cost) - individual tape cartridges are pretty cheap and if you calculate cost per MB, is pretty competitive.

LTO Ultrium is not the same technology as the older tape technologies - which are very slow. At 240MB/sec maximum speed so far, even normal hard-disks will have trouble coping with the transfer speeds.

Better tape archiving software now means you don't need to download the entire tape image to locate one file. But backing video files as opposed to computer data files is different - most P2 data files are huge - 4GB per file maximum - I have some right now ... so, the tape drive will spend less time locating the right file and uploading them to hard-disks. And you don't just restore ONE P2 data file - you have to restore an entire logical group - which will reach at least a couple of GB easily.

As an example of LTO-4 Ultrium drive, see this link -
http://h71016.www7.hp.com/dstore/ctoBases.asp?oi=E9CED&BEID=19701&SBLID=&ProductLineId=450&FamilyId=1249&LowBaseId=10372&LowPrice=$4. 00&familyviewgroup=923&viewtype=Matrix

Steev Dinkins · September 21st, 2007, 05:17 PM

I keep everything on hard drives - 500GB and 750GB drives. Every drive has a duplicate that is kept off site.

Jon Fairhurst · September 21st, 2007, 06:07 PM

Quote:

Originally Posted by Steev Dinkins

I keep everything on hard drives - 500GB and 750GB drives. Every drive has a duplicate that is kept off site.

This is a valid way to work. From what I've read, you should make new copies from the drives every 3 to 5 years. Hard drives don't last forever. Fortunately, they do get bigger, so the cost per GB continues to drop.

Andrew Kimery · September 21st, 2007, 07:23 PM

Quote:

Originally Posted by Jon Fairhurst

This is a valid way to work. From what I've read, you should make new copies from the drives every 3 to 5 years. Hard drives don't last forever. Fortunately, they do get bigger, so the cost per GB continues to drop.

I was just going to mention this. HDDs need to be spun up periodically otherwise they'll fail. So I'd say spin up the drives at least a few times a year just to keep them in proper working order. And for the really paranoid, don't have both your archive copies on the same brand of HDD. If defects roll off the assembly line (IBM DeathStars anyone) then having two drives from the same defective batch isn't a good thing.

For archive storage (not back-up) I think data tape is still the best way to go. As others have said the technology has improved and its been around long enough to be about as trustworthy as you can get.

-A

Mark Donnell · September 23rd, 2007, 01:53 PM

This month's DV Magazine has a nice introduction to various large-file storage methods, as well as an article on the future of tape as a recording medium. Tape will not die for a while to come if the technological improvements that they discuss actually come to market, although after using P2 I'm through with tape for now.

September 18th, 2007, 05:45 PM	#1
Calvin Dean New Boot Join Date: Aug 2007 Location: Olive Branch, MS Posts: 22	Archiving P2 Footage? After filling a P2 card with footage and dragging these clips to the hard drive, how do you archive this footage in case of future changes to the presentation? Obviously, you'll reuse the P2 card on the next project. I'm considering buying a 200 because I like the workflow -- but I'm used to batch capturing from tapes. Plus, tapes are easy to store. What are your archival methods?

September 19th, 2007, 10:24 AM	#5
Francesco Dal Bosco Regular Crew Join Date: Jan 2007 Location: Italy Posts: 31	Blu Ray problems It's my intention to buy a Blu Ray burner in order to store my P2 files. Now i'm hearing about problems using this method. I really would like to know something more about it (I never suspected I could have troubles to tranfer and archive P2 data on Blu ray or DVD disks).

September 19th, 2007, 07:55 PM	#11
TingSern Wong Major Player Join Date: Jan 2004 Location: Singapore, Rep of SINGAPORE Posts: 749	We are limited by the technology that is available at our time. Until we see higher densities random access h/w being made available (I read a holographic cube read/write by lasers that goes into the Petabytes range - that's 1,000 GB) that is affordable and not just labs stuff, the best stuff I see for archiving large amount of data is still tape. RAID 0 disks are good for working datasets. But after you are done with a current project, what do we use to move huge amount of data to slower, but, still viable media? With 32GB P2 cards coming around the corner and 64GB sometime next year, even BlueRay Disks are not enough to archive one P2 card. Although the drive is expensive (one off cost) - individual tape cartridges are pretty cheap and if you calculate cost per MB, is pretty competitive. LTO Ultrium is not the same technology as the older tape technologies - which are very slow. At 240MB/sec maximum speed so far, even normal hard-disks will have trouble coping with the transfer speeds. Better tape archiving software now means you don't need to download the entire tape image to locate one file. But backing video files as opposed to computer data files is different - most P2 data files are huge - 4GB per file maximum - I have some right now ... so, the tape drive will spend less time locating the right file and uploading them to hard-disks. And you don't just restore ONE P2 data file - you have to restore an entire logical group - which will reach at least a couple of GB easily. As an example of LTO-4 Ultrium drive, see this link - http://h71016.www7.hp.com/dstore/ctoBases.asp?oi=E9CED&BEID=19701&SBLID=&ProductLineId=450&FamilyId=1249&LowBaseId=10372&LowPrice=$4. 00&familyviewgroup=923&viewtype=Matrix Last edited by TingSern Wong; September 20th, 2007 at 09:53 AM.

September 21st, 2007, 05:17 PM	#12
Steev Dinkins Major Player Join Date: Nov 2004 Location: San Francisco, CA Posts: 613	I keep everything on hard drives - 500GB and 750GB drives. Every drive has a duplicate that is kept off site. __________________ www.holyzoo.com

September 18th, 2007, 06:10 PM	#2
David Saraceno Major Player Join Date: Jan 2006 Location: Inland Northwest Posts: 490	I use DVD9 for 8 GB cards I also have three hard drives for redundant backup

September 18th, 2007, 09:59 PM	#3
Kaku Ito Trustee Join Date: Jun 2004 Location: Chiyoda-ku, Tokyo Posts: 1,382	I tried many ways, blueray, dvdr, single drives, REV, G-raid and so on. I'm probably settling for single hard drives (two copies), since any of the RAID without redundancy failed (like G-RAID at RAID level 0). and maybe really important ones on blueray. REV wasn't too bad only if the driver software does not conflict with your system (mine does, makes the finder go crash and comeback again and again). I wish they make firewire version for 70GB drive. Also, providing clips to, say like DVinfo, saved me from loosing everything. My G-RAID crashed and ATTO R380 made the RAID volume getting broken, I lost data on RAIDs many times, then I tested RAID6, too but it gets really slow at the end of the capacity that won't even play back DVCPRO HD, I'm going to use RAID5 for the work volume, and a couple of single drives (same copies) to keep the original clips.

September 19th, 2007, 09:49 AM	#4
TingSern Wong Major Player Join Date: Jan 2004 Location: Singapore, Rep of SINGAPORE Posts: 749	Actually, the best archive media is still tape. Not those HD tapes, but, computer tapes. DLT, AIT-5, LTO, etc ... The best technology so far is LTO (Ultrium) tape. Today - you can get 800GB per tape cartridge at transfer speeds of up to 240MB/sec. How many DVDs can you store into one 800GB tape whose dimensions are 4" x 4" x 0.8"?

September 19th, 2007, 10:40 AM	#6
David Saraceno Major Player Join Date: Jan 2006 Location: Inland Northwest Posts: 490	What problems have you heard about blu ray archiving?

September 19th, 2007, 10:51 AM	#7
Francesco Dal Bosco Regular Crew Join Date: Jan 2007 Location: Italy Posts: 31	David, thank you for your reply. Maybe I've misunderstood the Kaku Ito post. I'm relatively new to P2 workflow (my work is still for the most part on DV) and I'm trying to learn everything I can on this forum.

September 19th, 2007, 11:53 AM	#8
Steve Rosen Major Player Join Date: Dec 2005 Location: Monterey, California Posts: 895	It is amazing to me that after 2 years there is not a tried and proven way to do this... I'm on the road a lot, and I shoot documentaries, 720/24pN, so I have a lot of material. My system is simple but time-consuming and intentionally over-redundant. When I get back to the hotel room I have a Powerbook set up there with two good off-the-shelf 250g FW400 hard drives - they're cheap - about $120@. I offload the cards to one drive, then back up the folders to the other.. With the card removed from the computer I use P2 LogPro to check the files in both hard drives be be sure they aren't corrupted - then and only then do I re-format the cards in the camera. By the way, I do this myself rather than deligate it to an AC because I'm paranoid... When they're full, I ship one drive via FedEx back to my office, and keep the other in my travel case... I have a number of drives - like four to eight - with me, as well as FedEx labels, so I can repeat this process. When I get home I transfer the files from the drives to a FW800 500g drive in a Granite tray. I back up this drive to another 500g. I ingest the material I need into FCP using Log and Tranfer - Then and only then do I erase the files on the portable drives so I can reuse them - I reuse them about five or six times. I keep both of the 500g drives on a shelf dedicated to storage.. If both drives die, which is not inconceivable, I'm f___ed...

September 19th, 2007, 01:55 PM	#10
Mark Donnell Major Player Join Date: Nov 2005 Location: Silver City, NM Posts: 385	I use external 500 GB hard drives for back-up. They are cheap ($ 120-140 on sale) and with two or three copies of a project it is highly unlikely that you will ever lose anything. For important stuff you can easily put one of these drives off-site in a safety deposit box or other location. For low-level raw material, I keep some on DVD.

September 23rd, 2007, 01:53 PM	#15
Mark Donnell Major Player Join Date: Nov 2005 Location: Silver City, NM Posts: 385	This month's DV Magazine has a nice introduction to various large-file storage methods, as well as an article on the future of tape as a recording medium. Tape will not die for a while to come if the technological improvements that they discuss actually come to market, although after using P2 I'm through with tape for now.