Archiving video. at DVinfo.net

Eric Peltier · November 28th, 2006, 01:21 AM

hello all,
I have a question about archiving our masters and MXF files from all the projects we shots using the P2 workflow.
I'm convinced DLT is the way to go, instead of buying more hard drive that take space and are more incline to failure.
Our IT guy, is convinced that hard drive are better because it's a cheaper solution.
he also think that we should compress the data ( what I mean by compressing is using ZIP or RAR or something like that)
Now, I've always heard this is a big no no in a post environment.
heard of stories where the data ended up corrupted when de-compressed.

My understanding of these compression format is very limited, this why I turn to all of you,
Am I crazy to think this is not a good idea?
loved to hear some real life experiences.

Sergio Perez · November 28th, 2006, 02:14 AM

DLT is great. Just copy to it and use it when you need. Hard drives are okayish,but be sure to have two copies in two different HDD's. use them in regular intervals (once every month, do a virus scan on them or something that makes the drive mechanism spin), in order to prevent the drive motor to die. Store both mediums (DLT or HDD's) in a cool, dry and safe place (Humidity boxes).

With DLT, you'll feel a lot more safe than with HDD's, but its a lot more expensive than the HDD's. DLT is the industry standard, and used by the big corporations for their files backups.

Have you considered getting an Iomega REV drives? Those sound right. the company I work for has just invested on them. Haven't arrived yet, but sounds good on paper.

And don't forget Blu-ray. It should be a nice alternative for storage too.

Don't compress anything. Leave everything has they are. When you compress, you are changing something. If anything goes wrong in the compression and decompression process, its game over for your footage.

Eric Peltier · November 28th, 2006, 03:31 AM

Quote:

Originally Posted by Sergio Perez

DLT is great. Just copy to it and use it when you need. Hard drives are okayish,but be sure to have two copies in two different HDD's. use them in regular intervals (once every month, do a virus scan on them or something that makes the drive mechanism spin), in order to prevent the drive motor to die. Store both mediums (DLT or HDD's) in a cool, dry and safe place (Humidity boxes).

With DLT, you'll feel a lot more safe than with HDD's, but its a lot more expensive than the HDD's. DLT is the industry standard, and used by the big corporations for their files backups.

Have you considered getting an Iomega REV drives? Those sound right. the company I work for has just invested on them. Haven't arrived yet, but sounds good on paper.

Thanks sergio
That confirms what I thought about compressing the data.
I did take a look at Rev pro media, sounds interesting but unfortunately the media can only hold 35gig max (as of today) and cost around $60 per disk.
DLT tape is 300gig and cost around $90.
I just did an interesting calculation to compare prices between hard drive and DLT backup.
Based on $1000 per TB which is what hard drive cost now.
and based on the price of the new Quantum drive $7000 + $90 per DLT tape
here's what I came up to.
Capacity(TB)---HD costs------DLT Cost
---1------------ $1,000----------$7,270
---2-------------$2,000----------$7,540
---3-------------$3,000----------$7,810
---4-------------$4,000----------$8,080
---5-------------$5,000----------$8,350
---6-------------$6,000----------$8,620
---7-------------$7,000----------$8,890
---8-------------$8,000----------$9,160
---9-------------$9,000----------$9,430
---10-----------$10,000----------$9,700
---11-----------$11,000----------$9,970
---13-----------$13,000---------$10,510
---14-----------$14,000---------$10,780
---15-----------$15,000---------$11,050
---16-----------$16,000---------$11,320
---17-----------$17,000---------$11,590
---18-----------$18,000---------$11,860
---19-----------$19,000---------$12,130
---20-----------$20,000---------$12,400
past 10TB DLT becomes much cheaper, for 20TB you pay $20k in hard drive vs $12,400 for DLT.
+ I'm not counting the time and energy (AC Power) it takes to run those hard drive.
We have at our facility around 7TB that needs to be backed up cause our Raids are getting full, I think DLT is definitely the winner.
thanks for your feddback

Dean Sensui · November 28th, 2006, 03:32 AM

I transfer my P2 footage to a mirrored RAID. It's based on Firmtek hardware. The drives are IBM/Hitachi SATAs on sleds which allow them to be hot-swappable. Ultimate capacity is unlimited except for one's budget.

In the event a drive dies, the second one continues to operate undisturbed. I actually did a test where I pulled a drive to simulate a drive failure while data was being written and the "good" drive continued to operate normally.

Plugging in the "dead" drive and reformatting it, then mirroring it, restored things back to normal in this test.

Writing data to a mirrored RAID is, essentially, making an instant backup. Storage cost is about $1.15 per minute or so. Including the cost of the drives, sleds and shipping.

Fast, reliable and economical.

Martin Iverson · November 28th, 2006, 09:05 PM

Those hard drive prices are a little high. A 300GB Maxtor DiamondMax w/16MB buffer goes for $119. That makes it aprox $440 per usable TB. So 20Tb is only $8,800.

Eric Peltier · November 28th, 2006, 09:19 PM

Those price are based on Raid NOT just the hard drive.
Apple Raid are $12,999 for 7TB
LaCie Raid $1299 for 1 TB
Granite Digital $5955 for 6 TB

If you want secure backup Raid are the only way.
so no, those prices are correct.
e.

Martin Iverson · November 29th, 2006, 11:36 AM

RAID is inherently insecure and must itself have redundancy either built into the RAID protocol (ie RAID 50) or with a separate RAID system (ie. RAID 1). It seems illogical to use RAID for secure backup of master files.

It seems to me that the logical use of a hard drive for backup is to plug in a new unit, fill it with data (from a RAID), disconnect it, label it and put it in a secure storage space, the same as you would a master tape. Should that one brand new drive fail the data is much more likely to be recoverable than if it is spread across a RAID system.

This is just my opinion of corse Eric. I've been following and respecting your advise on this forum for quite a while and I offer this opinion in the spirit of contributing, not confrontation.

Andy Nickless · November 29th, 2006, 11:53 AM

Quote:

Originally Posted by Dean Sensui

In the event a drive dies, the second one continues to operate undisturbed.

I really don't see this . . .

The whole idea of a raid is for speed - that's why they write to both disks at once. So how can this be viable - I'd love to know because raids don't write one file to one disk and another to the other disk - they spread the data between both at the same time - as I said, for speed.

And I agree with Martin - raids are for Editing rather than storage.

If I'm wrong, please educate me!

Andy

Gunleik Groven · November 29th, 2006, 12:29 PM

You're wrong! -;)

There are many levels and functions for RAID. What you describe is RAID 0, Then speed is the consideration. Than you have a total redundant RAID where data is written to two (sets of RAID 0 - sometimes) disks to have total backup. Then you sorta have everything inbetween. Some optimized for speed, others for security, some for both (Like RAID 5)

Gunleik

Dean Sensui · November 29th, 2006, 01:19 PM

RAID 0 for speed.
RAID 1 for reliability.

The RAID I'm using for storage of master files is a level 1 RAID.

It writes identical data to two disks simultaneously.

This type of RAID is used for systems where reliability is a must. Such as banking data, air traffic control, etc. It's been around for quite sometime actually.

As mentioned before, I did a test where I pulled one of the drives while data was being written (to simulate a failed drive) and the data remained intact on the drive that was still running.

After the pulled drive was reformatted (to simulate installing a new drive), the data was re-mirrored and the system was back to normal.

With a RAID 1 you don't have to waste time writing to one set of drives, then to a second set because you're always writing to two drives simultaneously.

Andy Nickless · November 29th, 2006, 01:49 PM

Quote:

Originally Posted by Dean Sensui

RAID 0 for speed.
RAID 1 for reliability. . .

. . . With a RAID 1 you don't have to waste time writing to one set of drives, then to a second set because you're always writing to two drives simultaneously.

Wow!
You learn something every day!

Thanks for that. I'll keep it in mind next time I need a new External HD.

Andy

Eric Peltier · November 29th, 2006, 03:44 PM

Quote:

Originally Posted by Martin Iverson

RAID is inherently insecure and must itself have redundancy either built into the RAID protocol (ie RAID 50) or with a separate RAID system (ie. RAID 0). It seems illogical to use RAID for secure backup of master files.

It seems to me that the logical use of a hard drive for backup is to plug in a new unit, fill it with data (from a RAID), disconnect it, label it and put it in a secure storage space, the same as you would a master tape. Should that one brand new drive fail the data is much more likely to be recoverable than if it is spread across a RAID system.

This is just my opinion of corse Eric. I've been following and respecting your advise on this forum for quite a while and I offer this opinion in the spirit of contributing, not confrontation.

hey, no worries, this is exactly what this forum is about, I think something, you think different, sometimes I'm wrong, sometimes you are, in the end we both learn something.
e.

Martin Iverson · November 29th, 2006, 06:09 PM

You're right Eric and this time I have to correct myself. I edited my posting from so that it correctly states RAID 1 instead of RAID 0 for having built in redundancy. I knew that, just typed it wrong.

For those following along generally a RAID 1 will slow down your system as opposed to RAID 0 which will increase processing speed. RAID 1 is not recommended for video editing where processing speed is important. I run RAID 0 with 3 300GB hard drives and use a program called Carbon Clone Copy to copy the data on the RAID onto a single 750GB hard drive. The RAID gives me speed and the single hard drive gives me reliable back up.

Dean Sensui · November 29th, 2006, 06:57 PM

Here's the cost of a Firmtek solution:

Firmtek SeriTek/2eVEN4 External Bundle (External 4-Bay Enclosure & 4-Port PCI Host Adapter): $579

4 500-GB Hitachi/IBM SATA drives: $185 each x 4 = $740

Total startup cost for 2 terabytes of storage: $1,319.

Each additional terabyte requires a pair of Hitachi SATA drives and two Seritek "trays".
Drives: $185 x 2 = $370
Trays: $23 x 2 = $46.
Total : $416.

To get a total of 20 terabytes:
First two TB = $1,319
18 additional TB = $7,488
Total = $8,807

These drives can be configured in a variety of ways. Single drives, striped pair (RAID 0) or mirrored pair (RAID 1). And they can be set up in combinations as well. You can have one pair set up as RAID 0 while the other pair is RAID 1 in the same enclosure. Or you can go nuts and stripe all four drives as a single high-speed RAID.

Keep in mind that this cost analysis is for 20 TB of straight storage, not duplicated RAID 1 (mirrored drives). If you mirror the drives, the cost per usable terabyte doubles.

What a hard-drive system gives you, that DLTs won't, is random access. Locate that clip with a catalog system, plug in the drive pair, and grab it. No need to fast forward or shuttle a tape.

With any archive system, what's vitally important are backups. Even tapes fail. So those things have to be taken into consideration when designing a system for the long term.

And then there's upgrades. When something else takes the place of SATA or any other storage medium, how long will it take to migrate that data to a newer medium? For that matter, how many of us have migrated older tape formats to newer ones?

It's all scary stuff when you sit and really think about it.

Sergio Perez · November 29th, 2006, 09:14 PM

Really scary, Eric. Bottom line, what we all really needed is a DVCPRO HD deck, but that's just too expensive. How about renting a deck, buying some tapes and copy your cards back to p2, then use the camera's FW port to copy to the deck? I really don't know how much a dvcpro HD tape goes for, but, concerning safety, having the footage on a playable tape were you can visually see the footage on a deck, always seemed the most reliable...

My ideal solution was to have a MXF backup and and a dvcpro HD TAPE backup- but that's insane and overbudget...

November 28th, 2006, 01:21 AM	#1
Eric Peltier Regular Crew Join Date: Apr 2006 Location: San Francisco/Paris Posts: 121	Archiving video. hello all, I have a question about archiving our masters and MXF files from all the projects we shots using the P2 workflow. I'm convinced DLT is the way to go, instead of buying more hard drive that take space and are more incline to failure. Our IT guy, is convinced that hard drive are better because it's a cheaper solution. he also think that we should compress the data ( what I mean by compressing is using ZIP or RAR or something like that) Now, I've always heard this is a big no no in a post environment. heard of stories where the data ended up corrupted when de-compressed. My understanding of these compression format is very limited, this why I turn to all of you, Am I crazy to think this is not a good idea? loved to hear some real life experiences. __________________ Sights Sounds Stories www.tinboxtwo.com www.reardenstudios.com

November 28th, 2006, 02:14 AM	#2
Sergio Perez Major Player Join Date: Feb 2005 Location: Macau Posts: 331	DLT is great. Just copy to it and use it when you need. Hard drives are okayish,but be sure to have two copies in two different HDD's. use them in regular intervals (once every month, do a virus scan on them or something that makes the drive mechanism spin), in order to prevent the drive motor to die. Store both mediums (DLT or HDD's) in a cool, dry and safe place (Humidity boxes). With DLT, you'll feel a lot more safe than with HDD's, but its a lot more expensive than the HDD's. DLT is the industry standard, and used by the big corporations for their files backups. Have you considered getting an Iomega REV drives? Those sound right. the company I work for has just invested on them. Haven't arrived yet, but sounds good on paper. And don't forget Blu-ray. It should be a nice alternative for storage too. Don't compress anything. Leave everything has they are. When you compress, you are changing something. If anything goes wrong in the compression and decompression process, its game over for your footage. __________________ If you don't believe in your film, no one else will.

November 28th, 2006, 03:32 AM	#4
Dean Sensui Inner Circle Join Date: Oct 2001 Location: Honolulu, HI Posts: 2,054	I transfer my P2 footage to a mirrored RAID. It's based on Firmtek hardware. The drives are IBM/Hitachi SATAs on sleds which allow them to be hot-swappable. Ultimate capacity is unlimited except for one's budget. In the event a drive dies, the second one continues to operate undisturbed. I actually did a test where I pulled a drive to simulate a drive failure while data was being written and the "good" drive continued to operate normally. Plugging in the "dead" drive and reformatting it, then mirroring it, restored things back to normal in this test. Writing data to a mirrored RAID is, essentially, making an instant backup. Storage cost is about $1.15 per minute or so. Including the cost of the drives, sleds and shipping. Fast, reliable and economical. __________________ Dean Sensui Exec Producer, Hawaii Goes Fishing

November 28th, 2006, 09:19 PM	#6
Eric Peltier Regular Crew Join Date: Apr 2006 Location: San Francisco/Paris Posts: 121	Those price are based on Raid NOT just the hard drive. Apple Raid are $12,999 for 7TB LaCie Raid $1299 for 1 TB Granite Digital $5955 for 6 TB If you want secure backup Raid are the only way. so no, those prices are correct. e. __________________ Sights Sounds Stories www.tinboxtwo.com www.reardenstudios.com

November 29th, 2006, 11:36 AM	#7
Martin Iverson Regular Crew Join Date: May 2006 Location: Anacortes, WA Posts: 78	RAID is inherently insecure and must itself have redundancy either built into the RAID protocol (ie RAID 50) or with a separate RAID system (ie. RAID 1). It seems illogical to use RAID for secure backup of master files. It seems to me that the logical use of a hard drive for backup is to plug in a new unit, fill it with data (from a RAID), disconnect it, label it and put it in a secure storage space, the same as you would a master tape. Should that one brand new drive fail the data is much more likely to be recoverable than if it is spread across a RAID system. This is just my opinion of corse Eric. I've been following and respecting your advise on this forum for quite a while and I offer this opinion in the spirit of contributing, not confrontation. Last edited by Martin Iverson; November 29th, 2006 at 05:45 PM.

November 28th, 2006, 09:05 PM	#5
Martin Iverson Regular Crew Join Date: May 2006 Location: Anacortes, WA Posts: 78	Those hard drive prices are a little high. A 300GB Maxtor DiamondMax w/16MB buffer goes for $119. That makes it aprox $440 per usable TB. So 20Tb is only $8,800.

November 29th, 2006, 12:29 PM	#9
Gunleik Groven Regular Crew Join Date: Sep 2006 Location: Espa - Norway Posts: 151	You're wrong! -;) There are many levels and functions for RAID. What you describe is RAID 0, Then speed is the consideration. Than you have a total redundant RAID where data is written to two (sets of RAID 0 - sometimes) disks to have total backup. Then you sorta have everything inbetween. Some optimized for speed, others for security, some for both (Like RAID 5) Gunleik __________________ Red? Sure!

November 29th, 2006, 01:19 PM	#10
Dean Sensui Inner Circle Join Date: Oct 2001 Location: Honolulu, HI Posts: 2,054	RAID 0 for speed. RAID 1 for reliability. The RAID I'm using for storage of master files is a level 1 RAID. It writes identical data to two disks simultaneously. This type of RAID is used for systems where reliability is a must. Such as banking data, air traffic control, etc. It's been around for quite sometime actually. As mentioned before, I did a test where I pulled one of the drives while data was being written (to simulate a failed drive) and the data remained intact on the drive that was still running. After the pulled drive was reformatted (to simulate installing a new drive), the data was re-mirrored and the system was back to normal. With a RAID 1 you don't have to waste time writing to one set of drives, then to a second set because you're always writing to two drives simultaneously. __________________ Dean Sensui Exec Producer, Hawaii Goes Fishing

November 29th, 2006, 06:09 PM	#13
Martin Iverson Regular Crew Join Date: May 2006 Location: Anacortes, WA Posts: 78	You're right Eric and this time I have to correct myself. I edited my posting from so that it correctly states RAID 1 instead of RAID 0 for having built in redundancy. I knew that, just typed it wrong. For those following along generally a RAID 1 will slow down your system as opposed to RAID 0 which will increase processing speed. RAID 1 is not recommended for video editing where processing speed is important. I run RAID 0 with 3 300GB hard drives and use a program called Carbon Clone Copy to copy the data on the RAID onto a single 750GB hard drive. The RAID gives me speed and the single hard drive gives me reliable back up.

November 29th, 2006, 06:57 PM	#14
Dean Sensui Inner Circle Join Date: Oct 2001 Location: Honolulu, HI Posts: 2,054	Here's the cost of a Firmtek solution: Firmtek SeriTek/2eVEN4 External Bundle (External 4-Bay Enclosure & 4-Port PCI Host Adapter): $579 4 500-GB Hitachi/IBM SATA drives: $185 each x 4 = $740 Total startup cost for 2 terabytes of storage: $1,319. Each additional terabyte requires a pair of Hitachi SATA drives and two Seritek "trays". Drives: $185 x 2 = $370 Trays: $23 x 2 = $46. Total : $416. To get a total of 20 terabytes: First two TB = $1,319 18 additional TB = $7,488 Total = $8,807 These drives can be configured in a variety of ways. Single drives, striped pair (RAID 0) or mirrored pair (RAID 1). And they can be set up in combinations as well. You can have one pair set up as RAID 0 while the other pair is RAID 1 in the same enclosure. Or you can go nuts and stripe all four drives as a single high-speed RAID. Keep in mind that this cost analysis is for 20 TB of straight storage, not duplicated RAID 1 (mirrored drives). If you mirror the drives, the cost per usable terabyte doubles. What a hard-drive system gives you, that DLTs won't, is random access. Locate that clip with a catalog system, plug in the drive pair, and grab it. No need to fast forward or shuttle a tape. With any archive system, what's vitally important are backups. Even tapes fail. So those things have to be taken into consideration when designing a system for the long term. And then there's upgrades. When something else takes the place of SATA or any other storage medium, how long will it take to migrate that data to a newer medium? For that matter, how many of us have migrated older tape formats to newer ones? It's all scary stuff when you sit and really think about it. __________________ Dean Sensui Exec Producer, Hawaii Goes Fishing

November 29th, 2006, 09:14 PM	#15
Sergio Perez Major Player Join Date: Feb 2005 Location: Macau Posts: 331	Really scary, Eric. Bottom line, what we all really needed is a DVCPRO HD deck, but that's just too expensive. How about renting a deck, buying some tapes and copy your cards back to p2, then use the camera's FW port to copy to the deck? I really don't know how much a dvcpro HD tape goes for, but, concerning safety, having the footage on a playable tape were you can visually see the footage on a deck, always seemed the most reliable... My ideal solution was to have a MXF backup and and a dvcpro HD TAPE backup- but that's insane and overbudget... __________________ If you don't believe in your film, no one else will.