What do people use to archive video?
 

Hi all

Just wondering what type of storage options are out there for putting away stock footage and completed footage where it can be retrieved at some date in the future?

I would like to transfer my stock footage to some "external entity" but I'm wondering what's the most cost effective solution. I am looking at a RAID array as I want the system to be able to handle a disk failure but RAID gets very expensive very rapidly.

Are there other options? I'm thinking of having around 500Gigs of storage initially but with the option of going higher.

Just wondering if I'm mad and should look at some other way of doing this. Things that go through my head are firewire disks in mirrored pairs? Enough to hold a year's worth of footage and just put it away and label it?

Any of you got any home grown solutions for this?
I'd appreciate some input.
Thanks
Donie

Most people simply use DV tapes for that. Some use recordable
DVD's (not handy for lots of data).

Hi Rob

Yes, I can understand most people use DV tapes but they are not the best when you want to gain access to a clip easily as you first have to find which tape it's on and then log it.

I am looking for a more flexible solution to gettting at the old stuff I've put away...

Putting it on DVD means encodign to mpeg2 and that usually cannot be easily re-used in editing without some performance penalties.

Storing raw footage on DVD+R directly would require a lot of disks. They are still too expensive to do this and it would take forever to archive existing stuff I have.

Thanks
Donie

I meant RAW DV on DVD, not encoding to MPEG2. That's why
I said this wasn't feasible for much data indeed. Tape has the
problem of not being quickly accessed, but you didn't list that
as an requirement for your solution.

The only option then is to use harddisks indeed. 200 GB isn't
that uncommon anymore. With a simple RAID controller you could
easily create a RAID 10 system with 4 harddisks that would do
striping + mirroring. Promise makes such cards for not too much
money. You then would have 400 GB of storage space available
that is mirrored for safety. In the future you could perhaps
upgrade to more drives.

I think external solutions will be quite expensive. If you just
put this on a "server" PC and share the partitions on all your
comptures where you need it through the same drive letters
(M for media, O for output for exampl) on a fast network (100
mbit might not be fast enough, so gigabit is probably best which
is falling down fast in price for entry products -> over copper,
not fibre!!).

Hi Rob

I sort of had come to the same conclusion myself but thanks for your input. I will look at building a RAID solution from a base PC setup. It would be nice to access it like you say and I also had Gigabit ethernet in my head as well.

Do you have any idea about a good RAID card. I'm thinking that 4 drives would not be enough but how easy is it to increase the drives in the future? Let's say 500Gig drive become commonplace in the next two years how could I introduce these types of disks into the RAID without losing the existing data?

These are the type of problems that I think I might run into. I basically want a storage solution that will grow with my needs over the years and I never want to lose the archive.

Mybe in the future I will transfer the terabyte(s) to a "box of matches" type storage but for now I just need to guarantee the safety of the stuff.

Thanks for your help and a great site.
Donie

I met a guy last year at NAB that mentioned something he did that maked a lot of sense and I've been doing it ever since.

He said he just buys a new external HD for each project.

Genius.

That way, when the project's done I simply put the drive up on the shelf. When/If I need it again I just plug in and go.

It's been great.

Is there a hot swap enclosure, external, that is firewire?

Yep, there are quite a few.

I was thinking of using USB2 external hard drive, but the one I got has a 4-gig limit for file size. Is that old stuff? Are the "current" USB2 and 1394 drives w/o the 4-gig limit?

<<<-- Originally posted by Law Tyler : I was thinking of using USB2 external hard drive, but the one I got has a 4-gig limit for file size. Is that old stuff? Are the "current" USB2 and 1394 drives w/o the 4-gig limit? -->>

If you are running Windows, then that means the drive is formatted w/ the FAT32 file system. If you have Windows 9x or ME, then there's nothing you can do about it. If you are running 2000 or XP then you can reformat the drive into the NTFS file system. This is what I had to do w/ my Iomega external USB2.0 drive. Doing so will also probably "void" your warranty -- which pretty much only means that if you ever have a problem with the drive and call customer service, they won't help you if you mention that you reformatted it to NTFS.

I typically RAR up DV and burn the RAR files to DVD. This makes it easy to archive video segments that would eitherwise not fit on a single DVD.

An 80 gigabyte external hard drive is $140. That's $1.75/GB.
I just bought 100 DVD+R discs for $110.72 (shipping included). That's 23 cents/GB. That means I could afford to make an archive plus six backup copies to store in safety deposit boxes across the globe, if I felt so inclined, for the same cost as buying a new external drive for each project, and DVDs are considerably less prone to shock trauma than storage devices that use lots of moving parts like HDDs--so you can just stuff them in a DV binder holder and not worry if they fall off the shelf in a freak earthquake.

I still have my old iomega jaz drive from 1997. $100/GB! Yee-ah!

Let's get some facts straight here. A harddisk will not dictate how
large a file can be stored on it (unless the file is larger then the
space available on the drive, ofcourse)! The file system dictates
this. As John correctly describes, only the NTFS file system can
handle files larger than 4 GB on the Windows platform. The only
OSes supporting this for a desktop system are Windows 2000
and Windows XP.

You can upgrade a drive in XP from FAT32 to NTFS, see the
Windows help file for instructions on this. It should not delete
your data, but it's a good thing to have backups in place before
attempting any of this.

If you decide to reformat your drive you will LOOSE ALL DATA
on it!!! Just so you know.

I've never ever heard of warranty being lost when you reformat
your drive. This should be idiotic as well since it will do no damage
to your harddisk at all and you have the right to store on the
drive what you want (a filesystem is just another sets of bits
and bytes on your harddisk, nothing like black magic or something).

In the old days you had a low-level format. Now I can understand
that this might void warranty. But a high-level format like Windows
does by laying down a clean file system has nothing to do with
your hardware (the drive in this case), it's the same thing as
writing a file to your drive (basically).

Also personally I would not want to store projects on a drive
on a shelf. Why? Because if you pull the drive from storage 5
years later the drive might not want to function anymore (due
to a number of possible reasons). It's way better to keep the
drive online (so you can spot potential problems), especially
in a mirrored position or use a non moving non mechincal backup
solution like DVD.

The problem with RAID is that it gets expensive FAST if you want
advanced stuff like external devices (which usually required SCSI
and a SCSI raid controlelr) or more than 4 drives.

I think I would personally do this if I had the money now but didn't
want to spend an arm end leg.

1) get a dedicated (!) computer/server for my file storage needs with perhaps another backup medium attached like tape or DVD burner to quickly move relative large amounts of data if needed. Make sure it is a big tower with plenty of watts in the power unit (or perhaps get a dual power unit)

2) run a server OS of any kind (microsoft probably be the easiest in this case where my personal preference would be windows 2000 or 2003) or a stable client OS (microsoft professional range where my personal preference for such a system would go to windows 2000)

3) use one harddisk on the internal IDE connector on the motherboard where the OS is installed on. Format it in NTFS and install the Promise driver before installing the card

4) install gigabit ethernet using copper

5) install a promise RAID ATA-133 or S-ATA raid controller

6) get 4 identical harddisks which run at 7200 RPM with 8 MB cache. You might need to get extra cooling for this as well.

7) setup a RAID 10 system with these 4 harddisks in the promise controller during boot up

8) format the 400 gb drive in windows with NTFS and I would partition this space as well to more managable sizes

If this is viable for you depends on a couple of things. One being
budget obviously. The more money you can spend, the better
a system with better safety you can get. Also your knowledge
or perhaps the knowledge of someone who builds your systems
comes into play as well as how fast you would probably fill up
this space before you would need to scale up (this will determine
how much space you will need initially).

You can use more than one promise controller in your system,
so in theory you can have more arrays then one. With an upgrade
you would basically install your second RAID system in the old one
(or hook up a new server to the network) and simply transfer
the files. Or you might be able to hookup the old drives in the
new array and have the array replicate the information accross
the whole array (that depends on which products you get).

Promise also sells external RAID solutions, see this page for more info.
But that will probably be expensive. I've never seen prices of
those, so who knows. Try to find a reseller of their products
and see what they charge for such solutions. It's still all ATA
or S-ATA drives, so that should help keeping the costs down.

Otherwise you might get a system from DELL for example with
a professional RAID controller + SCSI drives.

There are also new 1TB drives out which are basically multiple
drives with RAID striping in one package, see this thread.
However this is without failover though.

Rob, thanks so much for all that information. Thanks for taking the time to create such a detailed post. I will try to go that route soon so that information is invaluable to me. When I get it finished, a few months time, I will let you know how it went.

Thanks
Donie

Just to complete the Rob's post, to convert a logical drive under Xp or 2000:

\> Convert drive_letter: /FS:NTFS

As Rob said, it is a good idea to have a backup before trying this, but you should not have any lost of data.

<<<-- Originally posted by Rob Lohman :

I've never ever heard of warranty being lost when you reformat
your drive. This should be idiotic as well since it will do no damage
to your harddisk at all and you have the right to store on the
drive what you want (a filesystem is just another sets of bits
and bytes on your harddisk, nothing like black magic or something).
. -->>>


OK, so "void your warranty" might have been a little melodramatic.

This is how Iomega states it: "Important: Iomega does not support nor endorse formatting your Peerless drive using NTFS." (Their external usb2.0 drive comes formatted in FAT32)

That's from their online FAQ. Wen I called Iomega and asked what "does not support" means, I was basically told -- as I posted before -- that it means that customer service won't help you if you have a problem with the drive and tell them that it's formated to NTFS.

Silly? Of course. Especially when you see that the Iomega website disclaimer is posted along with instructions for reformatting the drive to NTFS. But I don't makes the rules, I just talk about 'em.

Thank you for telling me this. It is the first time I've heard of that.
From now on I will advice to anybody who wants to listen to stay
away from those Iomega harddisks. It's just absurd. Especially
since almost everybody is using Windows 2000 / XP now-a-days
(in the PC world), especially with video and photo stuff for which
these drives are mostly used. And guess what kind of file system
we would like for that. Indeed, NTFS. Much better if you want
your data to survive a power loss for example.

Thanks for clarifying it John!

Rob

Without getting too meta here, for the sake of fairness I do want to add that I've owned the Iomega 80GB USB2.0 HDD for just over one year and that I have had (knock on wood) no problems so far. I have it formatted to NTFS and I've even used it successfully as a capture drive. So far I've been happy with it.

If I boycotted every company that acted absurdly, I'd have to return everything from my Iomega hard drive to my Sony DVD burner (don't even get me started on Sony customer service) to everything I've ever bought at Best Buy! :D

I can understand your point of view, John. I'm not saying their
product is bad (some of you might even know me from working
with Iomega on another product) or that it doesn't do NTFS.
But how can I recommend a product for such a serious business
as video work if they won't support it if we put NTFS on it? I just
can't.

There are a lot more excellent external harddisk products out
there that don't have any problems with you using NTFS. If
someone asks me for a recommendation I can assure you I
will not recommend the Iomega anymore due to this. That's all
I'm saying.

well i do both, i have a raid5 of (4) wd sata 250 gig drives using the adaptec 2410 raid card (4 channel max) so i have a usable amount of like 700gig

then you get a single ide drive and put it in a usb 2.0 enclosure, you copy the entire project (raw files, project file and finished file etc) to the external (both as occasional backup during the project and then for final archival) then take the drive out of the ext. enclosure and put it in a box on a shelf labeled.

i JUST got a 40 gig ipod for a valentines day/bday present and i'm gonna play around with using that to move projects from home to work, it will be nice to play with files at work and be able to take them home easily.

i need to look into the options for going from the 4 channel adaptec sata raid card to the 8 channel card, i'm not out of space but i'm kind of wondering what happens if you need to grow it, if you have to start all over again?!?

matthew

I think most people who are going beyond 4 discs will move to
an external enclosure of some sort. I linked to a Promise one.
Don't know if Adaptec has those. Ofcourse it's no problem getting
such a system from the big boys for big cash though.

Raid 5 is another option indeed which will give you more space,
but less security with the more drives you use.

yeah adaptec has an enclosure device.

i have had mixed luck with promise in the past and i just can't trust them for anything really important.

your mileage may vary.

other than raid 5 what would you use? raid 10??

mirroring is a bit too wasteful and stripping drives has failed me in the past.

my preference is a raid5 with a hot spare.

if i had a card with more than 4 ports i would definitely add a hot spare.

i'm using another drive off of the raid to do occasionally backups of the project to.

matthew

Mirroring with striping. But only if you want and can afford it.
Raid 5 definitely is a very good choice, security with only one
drive "lost". But it is less secure than mirroring. Raid 0 is just
striping, Raid 1 is just mirroring, Raid 10 is mirroring + striping
(need 4 harddisks for that) and Raid 5 is striping + parity (minimum
of 3 harddisks)

If you keep the drive online you'll need a UPS (or surge protector) [which I suppose you should have anyways], keep your computer virus free, and the drives will break down faster.

DVDs don't last forever. You need to store them well, and even then they'll degrade.

DV tape doesn't last forever. But based on a similar computer backup tape, it should last at least 15 years.

A hard drive you store on your shelf would be nice if it doesn't break down. Do they break down when you don't use them?

i guess i just assume everyone here knows to run a GOOD ups on their editing system?

the people i know who have mysterious computer problems are almost always running with no ups protection.

and no a $20 power strip with a surge protector from best buy doesn't count.

i own a fairly good sized isp and EVERYTHING runs on APC ups's

i have a few that came on pallets they are so big. i have one that is a raid of batteries where if one module fails it keeps running and notifies you. the modules weight something like 400+ pounds.

there is a new cd-rom laser format coming out, i forget the name but it will hold a ton more on a disc than the current stuff

matthew

Hi Matthew

I didn't realise this was going to get into a "mine is bigger than your" type thread but I get your point ;)

I also know people who have mysterious computer problems but it's usually not the hardware or the software ;)

Donie

Blueray. I think you can already buy it from Sony? Probably not
as a datastorage system, though.

yeah i think that is it, its like $3k but with the size of hard drives and the size of files that people manipulate these days it will probably become mainstream faster than you would think.

matthew

There's a link to a Promise controller with 4 channels that supports
8 harddisks. See the thread here.

There are also two more RAID threads going on:

http://www.dvinfo.net/conf/showthrea...threadid=21152
http://www.dvinfo.net/conf/showthrea...threadid=20783

"DVDs don't last forever. You need to store them well, and even then they'll degrade."

No, but in 15 years DVDs will be an obsolete storage format anyway.

15 years ago the computer archive storage format of preference was 5.25" floppy disks that held 360 kB, or 0.0000748 DVDs.

If you haven't transferred your 5.25" floppy archives to some different medium by now, it's probably because the data isn't worth archiving.

When we speak of "archiving" in computer terms, we're talking 5-10 years. Anything greater than ten years is an epoch.

The only people who archive UNIVAC reels onto present-day media are computer historians.