After filling a P2 card with footage and dragging these clips to the hard drive, how do you archive this footage in case of future changes to the presentation? Obviously, you'll reuse the P2 card on the next project.

I'm considering buying a 200 because I like the workflow -- but I'm used to batch capturing from tapes. Plus, tapes are easy to store. What are your archival methods?

I use DVD9 for 8 GB cards

I also have three hard drives for redundant backup

I tried many ways, blueray, dvdr, single drives, REV, G-raid and so on.
I'm probably settling for single hard drives (two copies), since any of the RAID without redundancy failed (like G-RAID at RAID level 0).
and maybe really important ones on blueray.

REV wasn't too bad only if the driver software does not conflict with your system (mine does, makes the finder go crash and comeback again and again).
I wish they make firewire version for 70GB drive.

Also, providing clips to, say like DVinfo, saved me from loosing everything.
My G-RAID crashed and ATTO R380 made the RAID volume getting broken, I lost data on RAIDs many times, then I tested RAID6, too but it gets really slow at the end of the capacity that won't even play back DVCPRO HD, I'm going to use RAID5 for the work volume, and a couple of single drives (same copies) to keep the original clips.

Actually, the best archive media is still tape. Not those HD tapes, but, computer tapes. DLT, AIT-5, LTO, etc ... The best technology so far is LTO (Ultrium) tape. Today - you can get 800GB per tape cartridge at transfer speeds of up to 240MB/sec. How many DVDs can you store into one 800GB tape whose dimensions are 4" x 4" x 0.8"?

It's my intention to buy a Blu Ray burner in order to store my P2 files.
Now i'm hearing about problems using this method. I really would like to know something more about it (I never suspected I could have troubles to
tranfer and archive P2 data on Blu ray or DVD disks).

What problems have you heard about blu ray archiving?

David, thank you for your reply.
Maybe I've misunderstood the Kaku Ito post. I'm relatively new to
P2 workflow (my work is still for the most part on DV) and I'm
trying to learn everything I can on this forum.

It is amazing to me that after 2 years there is not a tried and proven way to do this... I'm on the road a lot, and I shoot documentaries, 720/24pN, so I have a lot of material. My system is simple but time-consuming and intentionally over-redundant.

When I get back to the hotel room I have a Powerbook set up there with two good off-the-shelf 250g FW400 hard drives - they're cheap - about $120@. I offload the cards to one drive, then back up the folders to the other.. With the card removed from the computer I use P2 LogPro to check the files in both hard drives be be sure they aren't corrupted - then and only then do I re-format the cards in the camera. By the way, I do this myself rather than deligate it to an AC because I'm paranoid...

When they're full, I ship one drive via FedEx back to my office, and keep the other in my travel case... I have a number of drives - like four to eight - with me, as well as FedEx labels, so I can repeat this process.

When I get home I transfer the files from the drives to a FW800 500g drive in a Granite tray. I back up this drive to another 500g. I ingest the material I need into FCP using Log and Tranfer - Then and only then do I erase the files on the portable drives so I can reuse them - I reuse them about five or six times.

I keep both of the 500g drives on a shelf dedicated to storage.. If both drives die, which is not inconceivable, I'm f___ed...

Today - you can get 800GB per tape cartridge at transfer speeds of up to 240MB/sec. That's fast enough to record 1080i/60 or 720p/60 at 4:2:2 uncompressed.

Long, long ago I used tape for general computer backups at home. It wasn't a good solution, because it was slow and was of no use for finding single, small files. The only thing it was really good for was full disc restores, which I thankfully never needed to do. The other problem with this method is that it was only as good as your most recent backup. Let's say you did a backup a week ago, but need only a single file from last month. The choice was to rewind your life by a week or lose the file.

Hopefully, the latest software is smarter and more flexible. Also, with large video files, tape backup makes a lot more sense than storing general computer files.

LTO Ultrium would be a good match for a four drive RAID 0. Both have roughly the same transfer rates. The tape gives security, while the RAID 0 is good for working in the moment. The only mismatch is this: A good 4-drive RAID will have 3TB capacity, while the tape is 1/4 the size. That means either getting a four tape drive ($$$) or baby sitting the thing for a full transfer.

I use external 500 GB hard drives for back-up. They are cheap ($ 120-140 on sale) and with two or three copies of a project it is highly unlikely that you will ever lose anything. For important stuff you can easily put one of these drives off-site in a safety deposit box or other location. For low-level raw material, I keep some on DVD.

We are limited by the technology that is available at our time. Until we see higher densities random access h/w being made available (I read a holographic cube read/write by lasers that goes into the Petabytes range - that's 1,000 GB) that is affordable and not just labs stuff, the best stuff I see for archiving large amount of data is still tape. RAID 0 disks are good for working datasets. But after you are done with a current project, what do we use to move huge amount of data to slower, but, still viable media?

With 32GB P2 cards coming around the corner and 64GB sometime next year, even BlueRay Disks are not enough to archive one P2 card. Although the drive is expensive (one off cost) - individual tape cartridges are pretty cheap and if you calculate cost per MB, is pretty competitive.

LTO Ultrium is not the same technology as the older tape technologies - which are very slow. At 240MB/sec maximum speed so far, even normal hard-disks will have trouble coping with the transfer speeds.

Better tape archiving software now means you don't need to download the entire tape image to locate one file. But backing video files as opposed to computer data files is different - most P2 data files are huge - 4GB per file maximum - I have some right now ... so, the tape drive will spend less time locating the right file and uploading them to hard-disks. And you don't just restore ONE P2 data file - you have to restore an entire logical group - which will reach at least a couple of GB easily.

As an example of LTO-4 Ultrium drive, see this link -
http://h71016.www7.hp.com/dstore/ctoBases.asp?oi=E9CED&BEID=19701&SBLID=&ProductLineId=450&FamilyId=1249&LowBaseId=10372&LowPrice=$4.00&familyviewgroup=923&viewtype=Matrix

I keep everything on hard drives - 500GB and 750GB drives. Every drive has a duplicate that is kept off site.

I keep everything on hard drives - 500GB and 750GB drives. Every drive has a duplicate that is kept off site.This is a valid way to work. From what I've read, you should make new copies from the drives every 3 to 5 years. Hard drives don't last forever. Fortunately, they do get bigger, so the cost per GB continues to drop.

This is a valid way to work. From what I've read, you should make new copies from the drives every 3 to 5 years. Hard drives don't last forever. Fortunately, they do get bigger, so the cost per GB continues to drop.
I was just going to mention this. HDDs need to be spun up periodically otherwise they'll fail. So I'd say spin up the drives at least a few times a year just to keep them in proper working order. And for the really paranoid, don't have both your archive copies on the same brand of HDD. If defects roll off the assembly line (IBM DeathStars anyone) then having two drives from the same defective batch isn't a good thing.

For archive storage (not back-up) I think data tape is still the best way to go. As others have said the technology has improved and its been around long enough to be about as trustworthy as you can get.


-A

This month's DV Magazine has a nice introduction to various large-file storage methods, as well as an article on the future of tape as a recording medium. Tape will not die for a while to come if the technological improvements that they discuss actually come to market, although after using P2 I'm through with tape for now.

I think drives are the best too - they're cheap enough and small enough and fast enough that you can double back-up everything, even on the road - and you can buy them anywhere.

When I ingest footage into FCP I back up the Capture Scratch files to another drive as well.. It may be overly redundant, but I sleep at night.

Large enterprises - managing data centres, all backup their data on tapes. So do large video production shops. Nobody uses disks for that purpose. Disks are fine for working data. Once a project is finished - it gets archived ... and tapes are used for that purpose. The danger of removable disks for archival purpose is after 5 years of storage, the disk might not start up - motor problem.

As I mentioned previously, until the promised land of petabytes of petabytes of holographic memory storage becomes commercially available, tapes are still the best medium for long term storage of digital data.

There are affordable tape drives featuring automatic cartridge loaders as well. You can stack 4 to 8 blank tapes on the loader, and if you are doing a huge offload to tape, say, 4 disk RAID-0 drive (3TB) to 800GB tapes, the tape backup software and hardware will write on one tape, finishes, drops the first tape, automatically loads the second, write on the second tape, etc ... until the jobs finishes or it runs out of blank tapes (waits for you to load more tapes).

Well how about Raid 5? If a disk fails, no problem, just insert another one.

RAID5, RAID6, etc ... are not archiving solutions. They are meant to prevent a system collapse if you loose a disk. More important, you can't transport a RAID5 array to another place - unless that place has identical h/w setup for you to plug the RAID5 array in.

I've used tape back-up in the past. It was slow and inefficient. In order to grab one scene/clip, you had to grap an entire list. But that was 10 years ago. Perhaps it has improved. If today's tape back-up systems are fast and provide better managment, this would be a good option. Can anyone address these concerns?

Quantum makes DLT and LTO tape drives that address your concerns, yes. The drives are MXF-aware and have been specifically engineered to address the needs of the video community. The drives can seek out and restore one particular MXF file, or they can even extract just a sub-portion of a file and wrap that sub-portion into its own MXF wrapper.

The DLT version is capable of data transfer somewhere on the order of 40 megabytes per second; the LTO3 version does it in as fast as 80 megabytes per second. The tapes are huge (200 to 400 gigabytes, I think) and comparatively cheap (I think they're around $100).

Thanks, Barry. I was just about to mention Quantum SDLT tape drives. For more details - see here.

http://www.quantum.com/Products/TapeDrives/DLT/SDLT600A/Index.aspx

I used DLT IV's ten years ago. As I mentioned in a prior post, they were slow. Head clogs presented other problems. Have newer DLT's improved on this?

Glad to know they're faster now and that a single file can be restored.

I understand that the Quantum is using SDLT now, not DLT. In my workplace, they are backing up midrange servers using DLT tape drives as well. Haven't heard any complaints about head clogs or DLT drives being slow.

I have no experience so far about this new Quantum SDLT-600a drive.

Generally, if you present a tape drive with lots of small amount of data to write (like, text files, small exe files, etc) - the tape drive will be slow. This is because the drive needs to stop, rewind the tape a bit, read the tape until last block, then starts writing the next block. However, with higher density tape drives, small blocks means there will be more gaps than data.

Dumping P2 or HD data to high density tape drives won't present that kind of problem - as the video data tends to be in large blocks anyway.

The last time that I was on the road (actually Europe), I resorted to using my laptop and writing the P2 card files onto DL discs. When I got back home, I thought about moving the files to a hard drive but they can fail but a disc doesn't. In a way, it's the closest thing to a P2 card---no moving parts.

I think we are confusing the issue here. Archiving data means offloading the data from an active storage media to something totally offline and kept offline for a long period of time.

Moving data from P2 card to a portable hard-disk, to a RAID array, etc ... to my definition, is not archiving. Moving the data to tape, DVDs, CDs, etc - this is archiving.

The medium of storage should preserve the data for at least 5 years - 10 years ... so I would rule out using hard-disks for that long term storage.

Most magnetic tapes for computer data can be read back even after 10 years of proper storage.

5...maybe 10 years...it's not a huge amount of time. We can see today
great copies of 80, 90 years old films from the silent era. Even very old 8mm home movies, from 1940's or '50s, if stored in a dry place, can show their shining grays or their wonderful colors today. It's seems strange to me that we can't have yet a solid, efficient, convenient data storage method in this super-technologic era. As a filmaker and HVX200 user I really would like to know more about this crucial subject matter. From some posts here I assume that hard disks are not a secure storage option. I'm also hearing about tape based storage methods but I would prefer to not come back to tape anymore. Assuming you can leave your data (your films) in a DVD or BD for,say, five years what should you do after this period? And what about the distributed DVD copies? How long they can survive?

Hi all:

Interesting discussion here. Has anyone heard of disc rot? I already have CDs from as few as five years ago, both home recorded and commercial that have oxidized and turned to rust. Unreadable. Anyone who is archiving P2 cards to DVDs and DL DVDs is in for a nasty surprise someday.

Hard drives seem even more fragile. Unless you have a long term cloning plan and actually clone them perhaps every year and keep a master and safety clone, you will lose data. Even if you do clone them every year to new, fresh drives, you still might lose the data. And of course, if an earthquake strikes and all of your drives are dashed into the ground from falling off of their shelves, ya got nothin'

The production company I was at for three years started archiving all P2 material to DVCProHD tape. Not easy, not cheap, but reliable as heck and compatible, for the time being, with lots of decks although as we all know, in five years, we will all quaintly look back on tape decks and say, "remember when we used TAPE? Weren't those the good old days?" ;-)

S-DLT seems to be the best option today but it too is a clunky and expensive one. Someone will come out with something soon (holographic RAM?) that will eventually be cheap, huge and reliable for a long term.

Dan

Hi All,

Have You experience with Iomega Rev70 for P2 footage ( in future EX1 also) archiving? It is not cheaper than HDD per gb, but tape gives some advantage.

http://www.iomega-europe.com/home?p=4740

Transfer rate seems quite good, max 30MB/s

Maybe REV70 can be alternative until DVD DL and Blue-Ray/HDDVD do improvements in technical and in price side ?

Sulev

Hi All,

Have You experience with Iomega Rev70 for P2 footage ( in future EX1 also) archiving? It is not cheaper than HDD per gb, but tape gives some advantage.

http://www.iomega-europe.com/home?p=4740

Transfer rate seems quite good, max 30MB/s

Maybe REV70 can be alternative until DVD DL and Blue-Ray/HDDVD do improvements in technical and in price side ?

Sulev

I tested REV35 with firewire connection but the actual transferring rate wasn't that high. Plus with my Mac Pro, when I installed the iomega driver then the finder keeps crashing.

Also, answering to the other post, I didn't say bluray doesn't work but I said 50GB back up took more than 12 hours to make the backup. And it takes some time to get them back to your hard disk again.

This is a most interesting thread. I haven't been around for a while, but I have an HD gig coming up in November - my first. We will most probably be using P2, and suddenly, how to hand the raw material to the client, is a huge concern. Since the camera and P2 cards are not mine, it is imperative I transfer them to a hard drive right in the studio, and make a backup on our own hard drive for safety.

I have been around since 2" quad tapes and seen all types of tapes eventually begin to shed, clog heads, and, as the dropouts become excessive, the priceless material in those 2", 1", 3/4", 1/2" tapes is gradually rendered useless. This is true of analog as well as digital recordings.

Now we are faced with a really neat recording medium that MUST be dumped off immediately - but to what? As I mentioned, I am very weary of tape; I have also had nightmares with drives that crash, bad blocks, missing sectors, corrupt FAT, will not boot, whatever. Even removable media - CDs, DVDs, Blue Ray, HD-DVD, can meet up with disaster - but so far I am a tad more comfortable with some removable media than I am with drives and tapes. The problem m, as I see it, is that there are no really efficient and practical removable media available that I know of. If there are, please point me in the right direction.

Most of the work I did in the early 1970s has long been transferred quite a number of times with the resulting picture degradation, or destroyed…forever. Many a Bert and Ernie routine (I began at Sesame Street) have been lost, or at best, they are no longer of broadcast quality.

I wondered why everybody here is so weary of tapes? We have been backing up data from enterprises onto tapes for years now. If tapes are so bad, every Fortune 500 companies data will be gone now.

We should be looking at using computer grade data tapes and mechanisms to backup P2 data (remember, with P2 data - we are now talking about bits and bytes - not analog video or audio anymore).

We have tapes (kept in conditioned environments - optimal temperature and humidity) that are already more than 10 years old - and data can still be reliably retrieved from them.

I just wish they come out with denser tape technology that allow faster transfer rate, you can back up and retrieve the footage faster on a tape.

I heard about Japanese company developed a lot denser tape technology, so I'm hoping they will make AIT version using that technology. I remember it is multitimes denser.

There are tape drives and media available today that already exceeds the capability of the PC or Mac bus capacity.

SDLT-4 is 800GB per cartridge (native) and has a transfer speed of 300MB/sec. Fast enough, right?

Although the version for video is different. SDLT-600A is specifically meant for video (MXF aware).

320GB per cartridge - and interfaces via 1GBit Ethernet port.

There are tape drives and media available today that already exceeds the capability of the PC or Mac bus capacity.

SDLT-4 is 800GB per cartridge (native) and has a transfer speed of 300MB/sec. Fast enough, right?

Although the version for video is different. SDLT-600A is specifically meant for video (MXF aware).

320GB per cartridge - and interfaces via 1GBit Ethernet port.

That sounds pretty good. I find it I usually use up about 300GB using two cams for concerts.

Kaku,

The sad part about this - SDLT-600A is only available in the US. It is not available in Europe or Asia right now. I have no idea when it will be made available to us. I checked with Quantum dealers - and every single one all throw up their hands ....

I will have my HPX500 in two weeks and I am bit worry about the storage best way.
But my question is once you have finished the documentary, for example, why you dont transfer to a DVCPROHD tapes the finnal product like always? May be the rest of the material can be storaged in Hard discs, or whatever, and the risk to loss that is not so important.
Probably it is something I lost in the way to understand wverything.
Thanks
Pablo

DVCPRO HD tape drives cost US$25,000. Way too expensive. Besides, why record on DVCPRO HD tapes - which is analog (somebody please correct me if I am wrong) ...

Easier to backup the digital data on computer grade media (like SDLT tapes) or even removable Hard-disks (like some people have suggested).

The Quantum SDLT-600A will be the ideal backup mechanism for MXF format - which is what P2 recording is based on.

I think everything depends of which kind of job you do. You are doing 4 or 5 documentaries a year or you are doing ENG production to TV and recording every day. I am from the first group so for me the SDLT 600 A way is not so cheap (more than 5000 $) and the tapes it needs Super DLTtape II (124$) for 300G are not either cheap.
It is true that a DVCPRO HD record machine is expensive, but if you are not doing many many productions a year, you dont need to buy it, just go and rent.

On the other hand DVCPRO HD tapes are not analogical are digital

My question is Can you transfer your finnal product from the NLE to DVCPRO HD tape? Can you transfer to another tapes and formats like HDCAM?

Thanks

Transferring from NLE to DVCPRO HD tape deck? Of course you can. You can output as Firewire or SDI - that will be best option. Pure digital to record on digital. Or you can output via component output and let the deck digitise the output into DVCPROHD tape.

If you can rent a DVCPRO HD tape deck, why not rent a Quantum SDLT-600A tape drive then? I am sure the rental will be cheaper than a DVCPRO HD tape deck.

I am doing P2 with MXF - and the Quantum is the best option. Everything is preserved in the 'Computer' domain. I can recall any clip down to the frame level without having to do a capture (which I will have to if I output it on DVCPRO HD tapes). That's the beauty of using the SDLT-600A ... provided Quantum makes it available for Europe and Asia.

Thanks Tingsern. I don´t know if it will be so easy to find a Quantum SDLT-600A for rent, at least in Spain but I will have in mind

One thing to keep in mind when backing up on DVCPRO tape is that the native resolution of the P2 card won't work. 1080p is not possible, the best you can do is 720p. I understand you refer, not to backing up as you record, but transferring later. My experience has been only with feeding via
FireWire a DVCPRO-HD tape deck as we recorded on the P2. I don't know if the same problem pops up when dubbing to tape after post.

Pablo,

I will be pleasantly surprised if you can find that SDLT-600A drive in Europe. I checked with Quantum - they told me that only USA has it (officially).

Ozzie,

Will that problem still be there if I feed it HD-SDI or component analog input?

Our feed was HD-SDI and not analog.

in this subject, you should follow the goings on over here:

http://www.amianet.org/

DVCPRO HD tape drives cost US$25,000. Way too expensive. Besides, why record on DVCPRO HD tapes - which is analog (somebody please correct me if I am wrong) ...
Consider yourself corrected. ;)

DVCPRO-HD tape is digital tape, and the footage is bit-for-bit identical to what would be recorded on the P2 cards. It's a direct digital copy.

One thing to keep in mind when backing up on DVCPRO tape is that the native resolution of the P2 card won't work. 1080p is not possible, the best you can do is 720p.
Not true. DVCPRO-HD tape has supported 1080 from the first day it was introduced; 720p was actually a later development. DVCPRO-HD tape records 1080p the exact same way the P2 card records it: as either 2:3, 2:2, or 2:3:3:2 pulldown.

In any case, it'll be recorded identically to how the P2 card records it.

Now, with all that said, I strongly agree with Tingsern:
Easier to backup the digital data on computer grade media (like SDLT tapes) or even removable Hard-disks (like some people have suggested).

Archiving to DVCPRO-HD tape is slow, exorbitantly expensive, inconvenient, and a huge step backwards in the workflow. Why go with a $25,000 deck, and $1 per gigabyte tapes, to convert all your lovely metadata-enhanced clip-based files into an old tedious linear tape-based, timecode-based workflow?

Why not, instead, get a $3,000 LTO3 deck, archive at 10 cents per gigabyte, and do so at 2x or 4x or 8x faster than realtime, and get to archive all your metadata and keep the clip-based nature of the files?

If you want to archive on tape, go with data tape, a much more efficient, modern, affordable, faster, and useful archive system. The only reason I can think of to involve DVCPRO-HD tape at all would be if your delivery requirements absolutely specified a tape master (such as for Discovery HD, who won't accept digital masters yet). Other than that, ditch the old videotape workflow and get the full benefit of a tapeless, data-based workflow.

The LTO3 tape archiving solution seems very interesting. I'm wondering how I should connect the drive to my computer. I suppose it's not possible to use firewire. I'm working with three 750g internal discs of my Mac Pro. Should I have SCSI discs instead?

Your internal HDDs can stay at SATA ... no problems. No need to change that to SCSI. However nearly all tape drives today (external version) are still SCSI. Haven't seen any LTO3 drives that are Firewire.

You can buy a SCSI host adapter and plug the card into your Mac Pro ....

If you need Firewire or USB2 - go for VXA-320 ... cheaper than LTO3 - but the speed is slower ....

Thank you very much TingSern. Maybe the firewire soultion is the right one for me. I will check the specs of vxa-320 and compatible tapes. As Barry said LTO 3 tapes have a really affordable price (about 10 cents per Giga).

LTO3 (and now, LTO4) is a high end solution. Very fast tape drive. You will need SCSI 160 minimum for the host adapter. Traditionally meant for enterprise use. Tape drive is expensive (US$5K list).

http://www.exabyte.com/products/products/get_products.cfm?prod_id=601#prices

VXA-320 is a "cheaper" option ... (US$1.5K list). Some manufacturers offer it as Firewire or USB2 - but, the real performance comes when you plug it into SCSI bus ...

http://www.exabyte.com/products/products/get_products.cfm?prod_id=641#prices