What do people use to archive video?

[Donie Kelly]

Hi all

Just wondering what type of storage options are out there for putting away stock footage and completed footage where it can be retrieved at some date in the future?

I would like to transfer my stock footage to some "external entity" but I'm wondering what's the most cost effective solution. I am looking at a RAID array as I want the system to be able to handle a disk failure but RAID gets very expensive very rapidly.

Are there other options? I'm thinking of having around 500Gigs of storage initially but with the option of going higher.

Just wondering if I'm mad and should look at some other way of doing this. Things that go through my head are firewire disks in mirrored pairs? Enough to hold a year's worth of footage and just put it away and label it?

Any of you got any home grown solutions for this?
I'd appreciate some input.
Thanks
Donie

[Rob Lohman]

Most people simply use DV tapes for that. Some use recordable
DVD's (not handy for lots of data).

[Donie Kelly]

Hi Rob

Yes, I can understand most people use DV tapes but they are not the best when you want to gain access to a clip easily as you first have to find which tape it's on and then log it.

I am looking for a more flexible solution to gettting at the old stuff I've put away...

Putting it on DVD means encodign to mpeg2 and that usually cannot be easily re-used in editing without some performance penalties.

Storing raw footage on DVD+R directly would require a lot of disks. They are still too expensive to do this and it would take forever to archive existing stuff I have.

Thanks
Donie

[Rob Lohman]

I meant RAW DV on DVD, not encoding to MPEG2. That's why
I said this wasn't feasible for much data indeed. Tape has the
problem of not being quickly accessed, but you didn't list that
as an requirement for your solution.

The only option then is to use harddisks indeed. 200 GB isn't
that uncommon anymore. With a simple RAID controller you could
easily create a RAID 10 system with 4 harddisks that would do
striping + mirroring. Promise makes such cards for not too much
money. You then would have 400 GB of storage space available
that is mirrored for safety. In the future you could perhaps
upgrade to more drives.

I think external solutions will be quite expensive. If you just
put this on a "server" PC and share the partitions on all your
comptures where you need it through the same drive letters
(M for media, O for output for exampl) on a fast network (100
mbit might not be fast enough, so gigabit is probably best which
is falling down fast in price for entry products -> over copper,
not fibre!!).

[Donie Kelly]

Hi Rob

I sort of had come to the same conclusion myself but thanks for your input. I will look at building a RAID solution from a base PC setup. It would be nice to access it like you say and I also had Gigabit ethernet in my head as well.

Do you have any idea about a good RAID card. I'm thinking that 4 drives would not be enough but how easy is it to increase the drives in the future? Let's say 500Gig drive become commonplace in the next two years how could I introduce these types of disks into the RAID without losing the existing data?

These are the type of problems that I think I might run into. I basically want a storage solution that will grow with my needs over the years and I never want to lose the archive.

Mybe in the future I will transfer the terabyte(s) to a "box of matches" type storage but for now I just need to guarantee the safety of the stuff.

Thanks for your help and a great site.
Donie

[Bryan McCullough]

I met a guy last year at NAB that mentioned something he did that maked a lot of sense and I've been doing it ever since.

He said he just buys a new external HD for each project.

Genius.

That way, when the project's done I simply put the drive up on the shelf. When/If I need it again I just plug in and go.

It's been great.

[Harry Settle]

Is there a hot swap enclosure, external, that is firewire?

[Bryan McCullough]

Yep, there are quite a few.

[Law Tyler]

I was thinking of using USB2 external hard drive, but the one I got has a 4-gig limit for file size. Is that old stuff? Are the "current" USB2 and 1394 drives w/o the 4-gig limit?

[John Britt]

<<<-- Originally posted by Law Tyler : I was thinking of using USB2 external hard drive, but the one I got has a 4-gig limit for file size. Is that old stuff? Are the "current" USB2 and 1394 drives w/o the 4-gig limit? -->>

If you are running Windows, then that means the drive is formatted w/ the FAT32 file system. If you have Windows 9x or ME, then there's nothing you can do about it. If you are running 2000 or XP then you can reformat the drive into the NTFS file system. This is what I had to do w/ my Iomega external USB2.0 drive. Doing so will also probably "void" your warranty -- which pretty much only means that if you ever have a problem with the drive and call customer service, they won't help you if you mention that you reformatted it to NTFS.

[Robert Knecht Schmidt]

I typically RAR up DV and burn the RAR files to DVD. This makes it easy to archive video segments that would eitherwise not fit on a single DVD.

An 80 gigabyte external hard drive is $140. That's $1.75/GB.
I just bought 100 DVD+R discs for $110.72 (shipping included). That's 23 cents/GB. That means I could afford to make an archive plus six backup copies to store in safety deposit boxes across the globe, if I felt so inclined, for the same cost as buying a new external drive for each project, and DVDs are considerably less prone to shock trauma than storage devices that use lots of moving parts like HDDs--so you can just stuff them in a DV binder holder and not worry if they fall off the shelf in a freak earthquake.

I still have my old iomega jaz drive from 1997. $100/GB! Yee-ah!

[Rob Lohman]

Let's get some facts straight here. A harddisk will not dictate how
large a file can be stored on it (unless the file is larger then the
space available on the drive, ofcourse)! The file system dictates
this. As John correctly describes, only the NTFS file system can
handle files larger than 4 GB on the Windows platform. The only
OSes supporting this for a desktop system are Windows 2000
and Windows XP.

You can upgrade a drive in XP from FAT32 to NTFS, see the
Windows help file for instructions on this. It should not delete
your data, but it's a good thing to have backups in place before
attempting any of this.

If you decide to reformat your drive you will LOOSE ALL DATA
on it!!! Just so you know.

I've never ever heard of warranty being lost when you reformat
your drive. This should be idiotic as well since it will do no damage
to your harddisk at all and you have the right to store on the
drive what you want (a filesystem is just another sets of bits
and bytes on your harddisk, nothing like black magic or something).

In the old days you had a low-level format. Now I can understand
that this might void warranty. But a high-level format like Windows
does by laying down a clean file system has nothing to do with
your hardware (the drive in this case), it's the same thing as
writing a file to your drive (basically).

Also personally I would not want to store projects on a drive
on a shelf. Why? Because if you pull the drive from storage 5
years later the drive might not want to function anymore (due
to a number of possible reasons). It's way better to keep the
drive online (so you can spot potential problems), especially
in a mirrored position or use a non moving non mechincal backup
solution like DVD.

The problem with RAID is that it gets expensive FAST if you want
advanced stuff like external devices (which usually required SCSI
and a SCSI raid controlelr) or more than 4 drives.

I think I would personally do this if I had the money now but didn't
want to spend an arm end leg.

1) get a dedicated (!) computer/server for my file storage needs with perhaps another backup medium attached like tape or DVD burner to quickly move relative large amounts of data if needed. Make sure it is a big tower with plenty of watts in the power unit (or perhaps get a dual power unit)

2) run a server OS of any kind (microsoft probably be the easiest in this case where my personal preference would be windows 2000 or 2003) or a stable client OS (microsoft professional range where my personal preference for such a system would go to windows 2000)

3) use one harddisk on the internal IDE connector on the motherboard where the OS is installed on. Format it in NTFS and install the Promise driver before installing the card

4) install gigabit ethernet using copper

5) install a promise RAID ATA-133 or S-ATA raid controller

6) get 4 identical harddisks which run at 7200 RPM with 8 MB cache. You might need to get extra cooling for this as well.

7) setup a RAID 10 system with these 4 harddisks in the promise controller during boot up

8) format the 400 gb drive in windows with NTFS and I would partition this space as well to more managable sizes

If this is viable for you depends on a couple of things. One being
budget obviously. The more money you can spend, the better
a system with better safety you can get. Also your knowledge
or perhaps the knowledge of someone who builds your systems
comes into play as well as how fast you would probably fill up
this space before you would need to scale up (this will determine
how much space you will need initially).

You can use more than one promise controller in your system,
so in theory you can have more arrays then one. With an upgrade
you would basically install your second RAID system in the old one
(or hook up a new server to the network) and simply transfer
the files. Or you might be able to hookup the old drives in the
new array and have the array replicate the information accross
the whole array (that depends on which products you get).

Promise also sells external RAID solutions, see this page for more info.
But that will probably be expensive. I've never seen prices of
those, so who knows. Try to find a reseller of their products
and see what they charge for such solutions. It's still all ATA
or S-ATA drives, so that should help keeping the costs down.

Otherwise you might get a system from DELL for example with
a professional RAID controller + SCSI drives.

There are also new 1TB drives out which are basically multiple
drives with RAID striping in one package, see this thread.
However this is without failover though.

[Donie Kelly]

Rob, thanks so much for all that information. Thanks for taking the time to create such a detailed post. I will try to go that route soon so that information is invaluable to me. When I get it finished, a few months time, I will let you know how it went.

Thanks
Donie

[Jean-Philippe Archibald]

Just to complete the Rob's post, to convert a logical drive under Xp or 2000:

\> Convert drive_letter: /FS:NTFS

As Rob said, it is a good idea to have a backup before trying this, but you should not have any lost of data.

[John Britt]

<<<-- Originally posted by Rob Lohman :

I've never ever heard of warranty being lost when you reformat
your drive. This should be idiotic as well since it will do no damage
to your harddisk at all and you have the right to store on the
drive what you want (a filesystem is just another sets of bits
and bytes on your harddisk, nothing like black magic or something).
. -->>>


OK, so "void your warranty" might have been a little melodramatic.

This is how Iomega states it: "Important: Iomega does not support nor endorse formatting your Peerless drive using NTFS." (Their external usb2.0 drive comes formatted in FAT32)

That's from their online FAQ. Wen I called Iomega and asked what "does not support" means, I was basically told -- as I posted before -- that it means that customer service won't help you if you have a problem with the drive and tell them that it's formated to NTFS.

Silly? Of course. Especially when you see that the Iomega website disclaimer is posted along with instructions for reformatting the drive to NTFS. But I don't makes the rules, I just talk about 'em.