Archiving EX files

Bill Ravens · October 13th, 2008, 12:45 PM

SDHC can be stored in a fireproof vault. Commercially available vaults are available in a number of sources. The cards, themselves are tagged, manually, with a numbering system. The numbering system is managed, again manually, via a spreadsheet file. Not the best of systems, but, it works for a small business. Most of my customers keep their own archives. I don't maintain a library for them. So, my filing system is small enough for me that I can manage it, well enough.

In all fairness, I must say that finished edits don't go onto SDHC cards. Generally, the finished edit goes onto a DVD. However, the beauty of Avid is that the project files are small and the original media files are what's kept. Intermediate files are generally discarded because they can be re-imported from the originals at any time. The only exception are for FX renders, which have to be wrangled seperately.

Craig Seeman · October 13th, 2008, 01:04 PM

But the SDHC cards are so small. What do you use to keep them from winding up in a big heap. There are file systems and cabinets for DVD, DV, DVCAM, many tape formats of course, hard drives can at least be stacked.

I don't know what's to prevent those cards from winding up in a big pile. Also, personally, I like being able to see client name and shoot date and reel (or disc) number on the media itself. I'd probably go nuts looking at 74869 (and what tiny numbers that would be!) and wondering what was on it and dashing off to the database to see. I guess my fantasy would be a tiny labeling printer and a storage unit that would fit the cards in slots.

Bill Ravens · October 13th, 2008, 01:07 PM

I don't do this, but, why can't an SDHC project be stored in those small 3x5 manila envelopes. You can write whatever you wish on the outside of the envelope and store the envelope in a drawer rack built for such things.
Another possibility is to store multiple SDHC cards in those special glassine notebook pages made to store 35mm slides. Problems like inadvertently turning the notebook upside down and dumping out all the cards makes me shiver with fear, tho'.

Ted OMalley · October 13th, 2008, 01:26 PM

Quote:

Originally Posted by Craig Seeman

But the SDHC cards are so small. What do you use to keep them from winding up in a big heap. There are file systems and cabinets for DVD, DV, DVCAM, many tape formats of course, hard drives can at least be stacked.

I don't know what's to prevent those cards from winding up in a big pile. Also, personally, I like being able to see client name and shoot date and reel (or disc) number on the media itself. I'd probably go nuts looking at 74869 (and what tiny numbers that would be!) and wondering what was on it and dashing off to the database to see. I guess my fantasy would be a tiny labeling printer and a storage unit that would fit the cards in slots.

Perhaps something like this would help.

(I'm sorry, I know I'm not being very helpful today, but I couldn't restrain myself.)

Jay Gladwell · October 13th, 2008, 02:16 PM

Ted, I got just the thing to help restrain you. It's a beautifully tailored white jacket. It comes with extra long sleeves and it ties in the back.

That should do it!

Jay Gladwell · October 13th, 2008, 03:05 PM

Quote:

Originally Posted by Craig Seeman

Again cost of SDHC is probably in line with using HDCAM but some clients aren't going to want to pay an additional $40/hr of media used. They SHOULD but many won't. Of course if they will then certainly that's a reasonable way to go.

Craig, I would simply add that expense to the overall cost of the job. Play it up as an asset, not a liability! A measly $40/hr of media is nothing considering the client pays for the media anyway--tape for shooting and/or storage.

Worst case, shoot on your original SxS cards, copy those files to the SDHC cards for archiving--keep them for the client (charging an archiving fee) or hand them over to the client just like we did with tapes.

I honestly can't see it being done any simpler or cheaper.

George Kroonder · October 13th, 2008, 04:34 PM

Quote:

Originally Posted by Bill Ravens

As far as your conservative approach, George, I wish you the best. My customers and I are quite happy with this solution.
...
Maybe you should try super 16 film. You don't need storage media with it.

ROFL... Thanks for the tip Bill.

Quote:

Originally Posted by Bill Ravens

Personally, I despise tape. It's unreliable, it exhibits random dropouts, its bulky, heat sensitive, and sllllooooowwwww. And talk about sensitivity to magnetic fields...yikes!

Modern (data) tape systems are anything but slow, LTO4 goes up to 240MB/s 2:1 compressed (that is 840GB an hour) and is very reliable. They certainly are heat sensitive, but I doubt an SDHC card would hold up long on a hot plate. LTO uses such a very strong magnetic field so they are much pretty not sensitive to lower strength fields like those from monitors, magnets, security devices or any other "environmental" field.

LTO verifies the data off of the tape using a read head directly after the recording head; the recorded data is verified as it is written (and rewritten if verification fails). HP archival tape is warranted 30 years. A 800GB native (up to 1.6TB compressed, not video obviously) archival tape is under $80.

That is 25x 32GB cards worth of data (with a little compression and increased efficiency, probably 30 cards). Or $1750 worth of $35, 16GB cards for just $80. Also many projects would not fit on a single 16GB card, or even a 32GB.

I still use the Gen3 LTO with half the capacity at half the cost. A dedicated backup workstation with a mirrored disk staging area to put completed projects on before they go to tape archive. And I use the same system for daily backup and can easily take the data offsite for disaster recovery.

Restoring a 100GB project takes about 45 minutes, including retrieving the tape. I know you can't copy 7 16GB SHDC cards to disk that fast (or 4 BDR's or 12 DVD-DL). The newer tapedrives would be even faster.

I can start a backup/archiving action or a restore in 5 minutes and walk away. Even if it would take all day to finish, which it doesn't, it doesn't tie me down more than that. I'll take that over sitting there watching files copy over from a SD card for 15 minutes per card (at 20MB/s). Eject > Copy > Repeat (x7) - no thanks!

Quote:

Originally Posted by Craig Seeman

But the SDHC cards are so small. What do you use to keep them from winding up in a big heap.

The book on this has already been written. And here's some pocket book/binder leaves for organizing SD cards.

George/

P.S. I have nothing against SDHC cards except that they are slow and unproven.

Bill Ravens · October 13th, 2008, 05:32 PM

George...

your tape solution sounds pretty professional, aka hi-end. It also sounds pretty nice. Keep in mind that I haven't been in this business 35 years like a lot of you more seasoned guys. I have a very small operation, my overhead is zero, or I couldn't make this fly. I have fun, or I wouldn't do it. Wish I had started this business when I was much younger, but, oh well. Wishes don't count for much in this world. My training is in engineering(as are my degrees) I do know the technology, and perhaps that's a strength in that I know what new technology to put my faith in. I just don't have the history, know what I mean? And I envy you guys with history, you know how things got to where they are. Just don'tt be afraid of new stuff...some of it is pretty reliable...and pretty slick.

Ted OMalley · October 13th, 2008, 09:33 PM

George,

Great link on the binder pages - thanks!

George Kroonder · October 14th, 2008, 06:33 AM

Quote:

Originally Posted by Bill Ravens

your tape solution sounds pretty professional, aka hi-end.
[...]
Just don't be afraid of new stuff...some of it is pretty reliable...and pretty slick.

You may call it high-end, but really in the end it is just economical; remember, I am Dutch ;-) so if there is a better, "cheaper" way to do things I'm all for that.

I have an IT background, so the need for "backup" and archiving has been permanently imprinted on my brain. That's not always the case with you "video guys" :-)

You know the backup system here probably costs as much as an EX1 and I'm looking to upgrade to the LTO4 system early next year, which costs are probably closer to that of an EX3 (US pricing). Ouch. So if there's something that beats it, bring it on.

But projects are getting bigger with every camera upgrade, more powerful systems and more data is continually kept on online storage. A good backup and archival regimen is a necessity for me.

I can't afford a freak cascading fault to lead to a probable catastrophic business failure. And those are the most common; when you "forget" to do something (like checking backup logs) or do something that you really shouldn't - like putting projects on scratch disks that aren't backed up as you don't have room elsewhere - and then something else fails (a disk, human error, etc.) and "y're screwed".

So even though we've only got a handful of systems, this was necessary, not luxury. But I know this won't be for everyone. It just depends on the type of work you do and if you're a freelance shooter or mainly do events, this probably will not suit you at all.

And as long as you are (relatively) happy with your chosen solution and don't feel like it isn't up to the job or wasting your time, you're good.

George/

P.S. Thanks Ted. Thought maybe number the cards and put a paper card in the pouch with that number + an index or such.

Jay Gladwell · October 14th, 2008, 01:07 PM

Here's an interesting article that carries with it some promising news!

http://www.sandisk.com/Corporate/Pre...e.aspx?ID=4353

George Kroonder · October 14th, 2008, 01:46 PM

Hi Jay, I saw that around when it came out in July. Just read this: "SanDisk SD WORM cards are available now worldwide in 128-megabyte capacity and are expected to be available in higher capacities later in the year. Pricing is available on request." (emphasis added).

That's not going to solve our archiving issues just yet. From what I recall it was intended for medical info or such (maybe legal?) and had been announced years earlier. Aparently there are some technological issues there...

George/

Jay Gladwell · October 14th, 2008, 02:12 PM

Quote:

Originally Posted by George Kroonder

Hi Jay, I saw that around when it came out in July. Just read this: "SanDisk SD WORM cards are available now worldwide in 128-megabyte capacity and are expected to be available in higher capacities later in the year. Pricing is available on request." (emphasis added).

That's not going to solve our archiving issues just yet. From what I recall it was intended for medical info or such (maybe legal?) and had been announced years earlier. Aparently there are some technological issues there...

George/

Re-read my post, George. I said "promising news," as in the near future!

Chuck Spaulding · October 14th, 2008, 02:17 PM

Quote:

Originally Posted by Bill Ravens

And for anyone who thinks that a RAID drive supported by the cost of NAS storage is a cheap solution, guess I'm wasting my breath. The non-recurring costs, alone, are a back breaker on a deal like that. The only time that's cost effective is for a business with an overhead that's probably bigger than all the capital in my business.

Sorry, I didn't mean to waste your breath.

I'm not sure what RAID drives and NAS have to do with tape back up?

I have five FCP edit bays and need to back up about 4TB's per month. WHILE Shooting to tape the original tape is an integral part of our current archival strategy. Now that we are switching to the EX3 I am looking for more efficient, secure and easy to integrate into our post production environment archival strategy. Also in all likely hood we will add two or three more edit bays resulting in an additional 3TB's per month.

It seems to me, unless I'm missing something, that the cost per gig for DLT is much lower than solid state memory. Don't get me wrong, if what your suggesting worked better and was cheaper I'd be very glad to use it.

Bill Ravens · October 14th, 2008, 02:34 PM

Chuck...
I'm sorry if I sounded rude. It's just that backup and archiving processes are pretty different for a business that has massive amounts of data to backup and retrieve, than for a small, home owned business like mine. NAS is something that requires an investment in hardware/software infrastructure that can't be justified for a small business. It's just not cost effective. The cost of installing such a system for my business would never pay back the investment.

One of my complaints with this otherwise excellent forum is that there are many different types of users who read and post here. Clearly, I think, not all architectures scale with the size of the business. I would never presume to advise someone on your scale of business with my processes. I would think the reverse would likewise be true. I think it's obvious that manual storage and retrieval of even 32Gb flash cards would be cumbersome and inefficient for your needs.

I have observed that people jump into discussions with advice, criticisms, judgments, arguments when their input is not necessarily relevant to the discussion. Such is the case, IMHO for NAS RAID backup systems and onesy/twosy backup systems.

October 13th, 2008, 09:33 PM	#24
Ted OMalley Major Player Join Date: May 2006 Location: Tampa, FL Posts: 596	George, Great link on the binder pages - thanks! __________________ Sony EX3, Vegas 9.0 64bit, Windows 7(64), Core i7, 12GB, RAID1 & 0, HotSwap SATA, 30" LCD(2560x1600)-GTX285 & 24" LCD(1360x768)-7800GT

October 13th, 2008, 12:45 PM	#16
Bill Ravens Join Date: Nov 2001 Location: santa fe, nm Posts: 3,264 Images: 10	SDHC can be stored in a fireproof vault. Commercially available vaults are available in a number of sources. The cards, themselves are tagged, manually, with a numbering system. The numbering system is managed, again manually, via a spreadsheet file. Not the best of systems, but, it works for a small business. Most of my customers keep their own archives. I don't maintain a library for them. So, my filing system is small enough for me that I can manage it, well enough. In all fairness, I must say that finished edits don't go onto SDHC cards. Generally, the finished edit goes onto a DVD. However, the beauty of Avid is that the project files are small and the original media files are what's kept. Intermediate files are generally discarded because they can be re-imported from the originals at any time. The only exception are for FX renders, which have to be wrangled seperately.

October 13th, 2008, 01:04 PM	#17
Craig Seeman Inner Circle Join Date: Feb 2004 Location: Brooklyn, NY, USA Posts: 3,841	But the SDHC cards are so small. What do you use to keep them from winding up in a big heap. There are file systems and cabinets for DVD, DV, DVCAM, many tape formats of course, hard drives can at least be stacked. I don't know what's to prevent those cards from winding up in a big pile. Also, personally, I like being able to see client name and shoot date and reel (or disc) number on the media itself. I'd probably go nuts looking at 74869 (and what tiny numbers that would be!) and wondering what was on it and dashing off to the database to see. I guess my fantasy would be a tiny labeling printer and a storage unit that would fit the cards in slots.

October 13th, 2008, 01:07 PM	#18
Bill Ravens Join Date: Nov 2001 Location: santa fe, nm Posts: 3,264 Images: 10	I don't do this, but, why can't an SDHC project be stored in those small 3x5 manila envelopes. You can write whatever you wish on the outside of the envelope and store the envelope in a drawer rack built for such things. Another possibility is to store multiple SDHC cards in those special glassine notebook pages made to store 35mm slides. Problems like inadvertently turning the notebook upside down and dumping out all the cards makes me shiver with fear, tho'.

October 13th, 2008, 02:16 PM	#20
Jay Gladwell Join Date: Jul 2002 Posts: 1,244	Ted, I got just the thing to help restrain you. It's a beautifully tailored white jacket. It comes with extra long sleeves and it ties in the back. That should do it!

October 13th, 2008, 05:32 PM	#23
Bill Ravens Join Date: Nov 2001 Location: santa fe, nm Posts: 3,264 Images: 10	George... your tape solution sounds pretty professional, aka hi-end. It also sounds pretty nice. Keep in mind that I haven't been in this business 35 years like a lot of you more seasoned guys. I have a very small operation, my overhead is zero, or I couldn't make this fly. I have fun, or I wouldn't do it. Wish I had started this business when I was much younger, but, oh well. Wishes don't count for much in this world. My training is in engineering(as are my degrees) I do know the technology, and perhaps that's a strength in that I know what new technology to put my faith in. I just don't have the history, know what I mean? And I envy you guys with history, you know how things got to where they are. Just don'tt be afraid of new stuff...some of it is pretty reliable...and pretty slick.

October 14th, 2008, 01:07 PM	#26
Jay Gladwell Join Date: Jul 2002 Posts: 1,244	Here's an interesting article that carries with it some promising news! http://www.sandisk.com/Corporate/Pre...e.aspx?ID=4353

October 14th, 2008, 01:46 PM	#27
George Kroonder Major Player Join Date: Jan 2008 Location: Utrecht, NL \| Europe 3rd Rock from the Sun Posts: 612	Hi Jay, I saw that around when it came out in July. Just read this: "SanDisk SD WORM cards are available now worldwide in 128-megabyte capacity and are expected to be available in higher capacities later in the year. Pricing is available on request." (emphasis added). That's not going to solve our archiving issues just yet. From what I recall it was intended for medical info or such (maybe legal?) and had been announced years earlier. Aparently there are some technological issues there... George/

October 14th, 2008, 02:34 PM	#30
Bill Ravens Join Date: Nov 2001 Location: santa fe, nm Posts: 3,264 Images: 10	Chuck... I'm sorry if I sounded rude. It's just that backup and archiving processes are pretty different for a business that has massive amounts of data to backup and retrieve, than for a small, home owned business like mine. NAS is something that requires an investment in hardware/software infrastructure that can't be justified for a small business. It's just not cost effective. The cost of installing such a system for my business would never pay back the investment. One of my complaints with this otherwise excellent forum is that there are many different types of users who read and post here. Clearly, I think, not all architectures scale with the size of the business. I would never presume to advise someone on your scale of business with my processes. I would think the reverse would likewise be true. I think it's obvious that manual storage and retrieval of even 32Gb flash cards would be cumbersome and inefficient for your needs. I have observed that people jump into discussions with advice, criticisms, judgments, arguments when their input is not necessarily relevant to the discussion. Such is the case, IMHO for NAS RAID backup systems and onesy/twosy backup systems.