A Very Sad Story

[Mark Job]

Hi Friends:
I was debating wether or not to re-title this account "A Very Stupid Story," instead of what I wrote, but overall I'm sure you will see why both titles can apply.

I, like many shooters and editors on this forum, have hopped on the clip based recording band wagon in a big way. I am even involved in building my own solid state digital recorder based on SD card technology, but I digress. What I'm about to relate only serves to utterly *prove* why there is still room in the professional production and post production world for piles and piles of cassette tapes !

For the past couple of years I have been working on a personal production, which has been a labor of love for me, now I fear over 1/3rd of the production is now gone forever ! When I'm shooting for a client on a professional production, I always record to tape in the camera and a Flash XDR solid state recorder simultaneously. Lately, I have been slowly lulled into what I will refer to as a "false security" in my clip based technology. The only production I have ever worked on, where tape was not used from beginning to end of principal photography, was this one and it bit me big time folks !

In my case, I was sabotaged in post production by three very tragic events:

1. The accidental formatting of a media drive just before I had a chance to backup to data tape the video on it, and...

2. The failure of a secondary hard drive required to make up a Raid Array I was in the process of implementing to doubly secure my edited footage !

3. The tragic fact I do *NOT* have all of my original footage on camera recorded cassettes !

Yup, I feel like an idiot ! I shot this production about 70 % on tape and about 30 % on a FireStore drive. There most certainly is room for tape. This experience teaches a hard lesson and makes an irresistible argument for always shooting with a tape backup, because you just don't know what can happen in principal photography or in post !

...Now this leads me to my second related topic:

Hard Drive Recovery*

I'm in the process of running recovery software after recovery software on my accidentally formatted external media drive in order to try and retrieve what has been lost. (A total nightmare ! Avoid at all costs !) Many of them simply do not work !* The most I have been able to get back so far are there in correct bit count, but are retrieved corrupted and unreadable ! many of these online Internet boutique software companies are making claims about their products which are nothing more than salesman's talk ! I mean some of these software programs don't work at all ! I'll let you know which one does work when I find it.

I make this post as a warning to the wise. Shoot with your clip recorder, but don't forget to also roll tape in your camera !

[Perrone Ford]

Mark, I am sorry to read your story.

Away from my video editing duties, I am the caretaker of numerous terabytes of corporate data. The safeguards we have put in place to protect that data are numerous. But back to this issue at hand.

I went TOTALLY tapeless workflow in 2005 after an 8 week trial period of shooting both to Firestore and tape. However, there are some safeguards that I always follow.

1. Always maintain 3 copies of the material while the project is underway, and at least one of those has to be off-site.

2. Upon final master, reduce 3 copies to two until client buys off.

3. When client buys off maintain the source and the master on optical or other format. Two copies for things that might be needed later. One on optical, one on tape for SD material.

In our recent, totally tapeless production for our short film, we removed the CF card out of the camera (Canon 5D) copied the file to my laptop, and made an immediate second copy to an external drive that went home with the director. He made nightly backups of his source, and I made backups at home and at the office. We had 5 live copies of the masters at any time during production. We are just coming to the end of post, and all 5 copies are still live.

Going tapeless is nothing to be afraid of. But if you read my clamorings on the Convergent Design's threads, the MAJOR thing I've asked for is redundant recording. Writing to 2 cards at once. This will give me two high bit rate copies in the unit, and a third low-res copy in the camera.

I do not feel more secure with tape-based recording. It is subject to far more types of damage than tapeless technology. And even the process of verifying it's viability in archive causes degradation of the medium.

So to me, the take away lesson here is to properly safeguard your media no matter whether you are originating on tape or on tapeless.

[Mark Job]

Hi Perrone:
You are absolutely right ! Don't go on the power of your clip based capture alone, and normally, I don't, but you should have at least three copies of what you're doing when in post. Ironically, I was in the process of making further backups of my original media when catastrophe struck hard ! I still believe in tape though. I don't see brand new-freshly recorded in camera cassette tapes as inferior media, however. I will gladly take my XL H1 HDV tapes in lieu of nothing at all ! I have shot several excellent productions on HDV cassette and I still dig the format big time, but yes indeed, the Flash XDR Long GOP 180 Mbps files are *much* better ;-) I don't think I'm in a position to go 100 % non-tape - Not after what I'm going through now ! I will use both- Even if I was shooting HDCAM SR ! I would pluggin my XDR and shoot tape, tape, tape !

EDIT: Your suggestion for dual record on CF card media from a Flash XDR or a Nano Flash unit is a really *Good* idea !

[Marty Welk]

Perrone, is Mirror raid really a wise way to do a backup, when a Device or USER error is still based on that SAME device?

i been thinking how i am going to backup the chip stuff myself. and if (say) the Camera itself had the ability to record to Both SxS at the same time i would not use it.
Or if some external recording device had a way to record to 2 items at the same time i would not use it either.
What i percieve to be the most effective backup when working with chip stuff, is 2 seperate recording devices, like say the In Cam, and the unit riding off the outside of the cam. Then with these cameras going all computer, actually needing one more CAM (running) itself as a backup

I TOO fear chip recording, i can cut my tape, pull 4 feet off, paste it back together with a bit of an angle, and i still would have 90% of my data. take 12 Bits off the front of a file, and you got nothing :-)
When the tape clogs (which mine never did once while shooting) the old DV devices would have a fit, if a tape was being chewed up and spit out (still never had this happen live) a person would know quite quickly. when a chip is going corrupt , the only way i am going to know its failing is if that causes it to stop writing.

Just like tape, somehow ya gotta have Enough Chips to KEEP the media , wheras i have read many people doing a DUMP and re-use IN THE FIELD oh my. what a bad time to have to motivate and check data. On the other hand, having enough chips for (say) 3 days of shooting is insane costs.

I (also) never understtod AUTO mirroring data on a Raid mirror system on the computer, if your system got a virus yesterday today BOTH do :-) manuel mirroring always seemed smarter.

[Perrone Ford]

I think you have to look at what you are protecting against. That is where people go wrong.

Are you protecting against corrupt data? Drive Failure, catastrophic failure in a geographic area, fire, what? In my day job, I have to protect against all of the above. So we have backups running live, everywhere. We have 4 layers of backup and at least 1 is geographically dispersed, and two of the others are on separate physical media.

I welcome having a device write simultaneously to two pieces of media. Because frankly, that is where things tend to go wrong. Human error. Formatting something accidentally. Dropping something in water. Leaving it in your pants. Loosing it. I dare say 90% of media problems are human error. It's the RARE occasion that it's something else. And usually a copy in two physical places is all that is required to avert disaster.

[Marty Welk]

the one time i lost big lots of data on my computer, was when i had it in 2 places , i even CRCed it all to make sure. I was cleaning stuff up, and was going to format one of the drives, then copy back. things didnt go as planned , undoubtedly a user error, and I not only formated the drive (which can be unformatted) but had written over it, with the wrong drive. that hurt, had to feed the tapes back in again.

with a Raid 0 system, one time i thought i had lost everything, as SPLIT raid 0 is nothing but gobeltygook, but managed to recover everything by reassembling the raid (something that says it would destroy all data) exactally how it was, and re-partitioning exactly the same way it was (good thing i knew its partition specs). miraculously everything was right were it was again. whew. With Split Raid0 the only data you can recover (with recovery software) without getting both drives back operational, is stuff that is under the stripe cluster size, like little 16k and less sized files.

Restoration - A freeware program that works bestes with NTFS, is an item i always keep on my drive , in case i ever need it. it can Yaaawwn recover a NTFS systems data without an overwrite (by copying to other media) IT was recommended long ago at the XPtweaks forum, and seemed to do the best for recovering NTFS, even though it was free, and other SoldWare junk wouldnt do what it would.

what are we usually protecting against , Humans, computers, and humans using computers :-) i am still not sure about turning my Camera into a Computer.

[Mark Job]

Hey Marty:
I think you and Perrone exposed the two main causes of data loss.....

1. Human Error.
2. Hardware failure.

Human error is essentially what we must protect against the most. In other words, I have to protect my computers from myself !

I have found the new high performance 7200 RPM Sata II hard drives are failing at a much greater rate than I ever remember seeing IDE hard drives fail at. I had one drive I managed to format (Twice !), and one hard drive that failed. In fact, When the drive manufacturers say 3 to 5 years max, then they mean look out after the 5 year mark ! SCSI drives run flat out all the time they are turned on (unless you checked disconnect on the SCSI HBA), but they seem to last much longer. I still have fully functional Seagate Cheetah 10 K RPM Ultra 2 and Ultra 3 SCSI drives which work perfectly after more than 10 years.

Marty, you wrote that you had success using restoration. Restoration was the first thing I tried and it couldn't see any of my erased files on my formatted drive. Is there some special way you have to use this program ?

[Marty Welk]

scan all clusters, in the pull down menu? wait 100 years.
when desperate i just select different things on anything, and hope.

[Bill Koehler]

Hi Mark,

For what little it's worth, I've used a program called Zero Assumption Recovery and it worked for me. It has a trial/scan mode where it will scan the drive and show you what it found. If you like what you see, you buy it @ $50 USD, pop in the serial #, and it does the actual recovery. I will say that if the drive is large, it does take a while to run, because it does indeed scan the whole thing.

One thing I like about it is that it does not modify the goofed up drive, which could potentially make the problems worse. It restores to a second drive that you specify.

As I say, it worked for me, YMMV.

ZAR data recovery software - do it yourself data recovery and digital image recovery.

[Noel Lising]

In my experience, you will be better off going to a company who does data recovery. The company I use to work for, has time and again deleted video files, we goto this company who does reverse engineering and they are able to recover data 99% of the time. It cost more than a thousand bucks to have it done, maybe you can google companies that does that in your area.

Good luck.

[Giroud Francois]

unless you have an hardware problem, recovery company are just a scam.
they are so many programs (even for free) that can recover a formatted, erased , corrupted disk.
And you have to know what you are looking for, because a raw disk recovery could be useless while still expensive if the guys do not know (not what they are doing, but why they are doing it).
if you have an hardware problem you have to find if it is electronic or mechanical.
sometime a night deep frozen in the fridge allows to read the disk for a while.
and the first thing to do if the disk is ok but unreadable, is to make a raw copy of it.
sometime trying to recover data destroy the last hope of using another method, so if you need to try several time, you better to do it on a copy, on a healthy disk.

[Dean Sensui]

Here's what I do:

Copy the SxS cards to a mirrored RAID. Then make another copy onto a single hard drive.

The mirrored RAID protects me against a hardware failure.

The single additional copy protects me against myself.

So there are always three copies. If any one drive goes bad, there is always one drive somewhere with a copy of the original footage.

I edit with a striped RAID for the sake of system speed. But at the end of the day the entire show gets backed up to a mirrored RAID. I always assume the striped RAID will fail, although it never has after all of these years.

I use only IBM/Hitachi SATA hard drives. They're not expensive but they've proven very reliable. I have more than two dozen of them sitting on the shelf and to date (more than five years) none have gone bad.

I did scare myself badly when I thought I had deleted an entire drive, but soon discovered to my great relief that it was just a mis-labeled and unused drive.

[Mark Job]

Originally Posted by Bill Koehler

Hi Mark,

For what little it's worth, I've used a program called Zero Assumption Recovery and it worked for me. It has a trial/scan mode where it will scan the drive and show you what it found. If you like what you see, you buy it @ $50 USD, pop in the serial #, and it does the actual recovery. I will say that if the drive is large, it does take a while to run, because it does indeed scan the whole thing.

One thing I like about it is that it does not modify the goofed up drive, which could potentially make the problems worse. It restores to a second drive that you specify.

As I say, it worked for me, YMMV.

ZAR data recovery software - do it yourself data recovery and digital image recovery.

...Hi Bill:
OK. I will go check it out and we will see.

Thanks,

[Mark Job]

Originally Posted by Bill Koehler

Hi Mark,

For what little it's worth, I've used a program called Zero Assumption Recovery and it worked for me. It has a trial/scan mode where it will scan the drive and show you what it found. If you like what you see, you buy it @ $50 USD, pop in the serial #, and it does the actual recovery. I will say that if the drive is large, it does take a while to run, because it does indeed scan the whole thing.

One thing I like about it is that it does not modify the goofed up drive, which could potentially make the problems worse. It restores to a second drive that you specify.

As I say, it worked for me, YMMV.

ZAR data recovery software - do it yourself data recovery and digital image recovery.

Hey Bill:
Thank you so much for the heads up on this software ! After trying nearly 20 other disk recovery softwares, ZAR 8.4 is *The Only One Which Actually Worked !* The trial version allows you to recover 4 folders at a time free of charge. My missing video files were in only three folders so the trialware version allowed me to recover them fully without any corruption :-) I am one happy camper tonight ! The whole process took around six hours but it was all worth the wait. The trialware instructions state if you want to recover anymore folders, then you have to exit the program and begin a new scan all over again, and you cannot save a scan to file for later quick reloading unless you purchase the full version. I will gladly hand over the $50.00 charge to these folks ! I consider this program well worth the cash !

[D.J. Ammons]

Mark,

This is great news! Congratulations on the recovery of your footage.