need storage advice for DV feature

[John Michael Harper]

Hey there, first time posting. I've been able to find lots of information so far, but time is of the essence so here goes.....

Our production company is in need of video storage.

Our crew has finished shooting a miniDV feature length comedy and we have about 25 hours of footage. We had originally planned on purchasing two 250 GB external firewire drives to edit the movie.

Having purchased the first drive in August to complete a corporate video, this 250 GB Maxtor external Firewire/2.0 drive has corrupted 2x in the past 3 months, 1st when I was working on the corporate video (this was explained to me by Maxtor support as a formatting problem), and this week when I began to capture tape for the feature. From reading threads on this site, we believe that while the Maxtor may be good for storage or smaller dv projects, it's not for feature editing (I/O problems).

I've checked out the threads on the benefits of RAID storage, but there didn't seem to be many endorsements as far as products available in the market or a spec on how to build a RAID storage system from scratch. We're no-budget, so we're looking to spend <$500 on a solution, whether that means buy or build. Keep in mind, we're frugal - not cheap.

Could you folks please help us in taking the best next steps in post? Are SCSI drives recommended? Is there anyway we can still make use of the Maxtor drive in the edit? Is it feasible to rent RAID storage? While we have a dv deck, a good computer, and Premiere Pro, none of us have previously edited a feature. Maintaining a steady workflow is very important to us. Thanks in advance for all your suggestions.

JMH

p.s. Could it be that the Maxtor's just a lemon?
p.p.s. I do have 2 open drive bays in my Gateway Tower if that helps.

[Lars Siden]

Hi,

I've had some bad luck with HDD:s - mostly IBM ones - so I guess some drives are "lemons" :-)

Lately I've read up on hardrives - one very important factor seems to be HEAT. HDD doesn't like to be over 45+ centigrades for a longer period of time. They like to be kept cold. Now I have 3x 200gb drives mounted in my case with a big 120mm blower fan mounted on them - after that I haven't had a single problem.

I belive Seagate will release a new series of drives just about now - big size ones - like 400gb+, they are said to be very reliable due to fewer moving parts.

To sum it up:

1. Internal or external doesn't really matter as far as it is chilled properly
2. External may be more practical - but beware of handling the drive
3. I'd go for the Seagate Barracuda series of drives ( if possible )

Remember - if you'd like an external drive, you don't have to buy the pre-built ones like Lacie / Maxtor - you can buy an empty external HDD box and mount whatever drive you like!

Best regards,

Lazze Z

Sweden

[John Michael Harper]

Lars,

Do you have your 3 x 200GB hard drives configured in a RAID array? Do you edit feature length projects on your workstation? If the drives aren't configured for mirroring, have you had any problems with data loss or corrupted file systems?

Thanks.

[Dan Euritt]

i've worked with thousands of hard drives over the years, and keeping 'em cool is exactly right.

just yesterday, tho, i had a fairly new seagate 200 gb in a cheapo cendyne external firewire case(oxford chip set & fan), and it started getting i/o errors on large data transfers, going from hard drive to hard drive... i thought the drive was toast... i even tried swapping firewire cables.

i put it on an easygo usb adaptor, tho, and it straightened itself right out.

john, the dv datarate isn't significant enuf to require raid arrays, no matter how many hours of footage you have... you just need to find out what's wrong with your current setup.

i would suggest mounting the troublesome maxtor drive inside the pc, as a troubleshooting step... unless you can return it for another one... it could be something besides the drive itself.

[Kevin Shaw]

I use mostly Western Digital EIDE drives and they've worked fine for me for several years for projects involving many hours of raw footage. Cooling is definitely important, so if you're having touble with that one solution might be a better-designed case. Lately I'm running several of my hard drives in external enclosures and have found that it definitely helps to get an enclosure with a built-in fan, which some of them don't have.

SCSI drives are theoretically better designed and more durable, but since you're on a low budget that may not be a practical alternative.

A typical "striped" RAID setup will increase reliability issues, so that's not the way to go if dependability is your main concern.

[Lars Siden]

<<<-- Originally posted by John Michael Harper : Lars,

Do you have your 3 x 200GB hard drives configured in a RAID array? Do you edit feature length projects on your workstation? If the drives aren't configured for mirroring, have you had any problems with data loss or corrupted file systems?

Thanks. -->>>

I'm just an amateur videographer - my real profession is as a System Architect and software developer. So I only do short films and home DVD:s - but still - I have about 10.000 photos that I care about and about 25 hrs of video.

On my editing workstation I don't have raid - the drives are solitary - one drive per function ( System, Capture/Edit, render ). Then I transfer the finished stuff ( programs, video, photo etc ) to may Server. On the server I have a H/W (SATA, Promise, 256mb onboard) Raid 5 for security.

Good luck!

// Lazze \\

[John Michael Harper]

Thanks everyone for your suggestions. You guys were really quick to respond and I very much appreciate it.

I'm going to get back on the line with Maxtor support and see what they have to say (it's still under warranty). I'm also going to cost research maxing out the drive bays in my tower with fast, dependable storage and a cooling solution. The need for external storage doesn't seem to make much sense now with those slots open. If I can salvage my current external drive, then I'll use it as an archive for files that we're not using.

This is the first of hopefully not too many troubleshooting issues. However, it's nice to know that the DV Info Community is there as a resource to help moviemakers realize their goals.

Thanks again,

John

[John Michael Harper]

I spoke with Jeff at Maxtor Support and he made the case for the idea that it is Windows XP, not the drive that may be at fault. According to him, the true test as to whether the drive is faulty or not is to delete the partition and reformat it.

I suspect that when I tried to delete a file from the drive while it was accessing it, Windows screwed up the file system.

I told Jeff that it was funny that I first looked to blame the hardware, rather than the Microsoft Gods.

So I'm going to keep my fingers crossed and follow his instructions. He did however, say it could be the drive, which he said that if I wanted, Maxtor would replace at no charge.

[Dan Euritt]

you can get fans that fasten to the front of the drive bays of a pc, but they are usually noisy as heck, but it provides the best cooling... i have mounted fans inside of pc's, suspended via tie wraps, which is a lot quieter... anything to get air going across the drive.

[Lars Siden]

<<<-- Originally posted by John Michael Harper : I spoke with Jeff
I suspect that when I tried to delete a file from the drive while it was accessing it, Windows screwed up the file system.
-->>>

If you use NTFS as filesystem - then it is _almost_ impossible to destroy the filesystem "by mistake". NTFS is a journal keeping FS with backups of the MFT and other important parts.

You could run the HDD with a faulty windows driver and there by generate errors.

Beware - that if you delete the partition and re-format the drive - then if the drive DO have bad sectors - they'll be re-mapped - so it will look like the drive is 100% error free and working - BUT it MAY be the "quiet moment before the storm".

// Lazze \\

[Rob Lohman]

I think some myth busting is in order. I highly HIGHLY doubt this
is a Microsoft "thing". Both FAT and NTFS are highly tested and
used filesystems, and if it where that easy to destroy things we
would not be using Windows. I've been using Windows for over
10 years both private and professionally (including professional
hosting) and never ever had such filesystem troubles.

The only case where I lost data was:

1. failing harddisks

2. bad IDE controllor / motherboard combination (back in the days)

3. virusses

4. once with a power failure, but that was on a FAT file system (ie, should NOT happen on a NTFS system)

Since you are running a 250 GB drive, 99% chance you are running
with an NTFS filesystem and as Lars says, it is nearly impossible
to destroy data by accident on that unless something is seriously
wrong with your hardware (harddisk, ide controller) or have other
problems like a virus.

Also:


> video, this 250 GB Maxtor external Firewire/2.0 drive has
> corrupted 2x in the past 3 months

corrupted in what way? Was the drive replaced?

> and this week when I began to capture tape for the feature.

again, corrupted in what way?

> Maxtor may be good for storage or smaller dv projects,
> it's not for feature editing (I/O problems).

a harddisk should be good enough to do anything you want
with it, including feature editing as long as it is cool enough
and treated properly

> I've checked out the threads on the benefits of RAID storage,

RAID is interesting in two cases: more speed (not really needed
for DV) or more safety (which might be interesting for you).

Cheap and good simple RAID controllers are made by Promise
an Adaptec for example.


> We're no-budget, so we're looking to spend <$500 on a

The problem in your case is the amount of information you've
got. You guys have had a high shooting ratio: even if you where
to produce a 2 hour movie, 25 hours of material gives you a
12,5:1 shooting ratio.

25 hours of RAW DV is 325 GB of material. If you want to transfer
ALL of that to a system you will need 2 (!) drives at least (since
the largest drive currently available is 250 GB).

Basically if you want to use RAID for safety you can do the
following configuration:

RAID 1 (mirroring): need 4 drives for 325 GB of material
RAID 3/5 (parity): need 3 drives for 325 GB of material

However, although RAID 1 (together with RAID 0 for striping)
is pretty cheap to come by, RAID 5 is not.

Let's make a small list for you:

RAID 0+1 Promise Fasttrack TX2000 controller: $90
4x 200 GB IDE drive: 4x $125 = $500
total: $590

Then you have 400 GB of space which is completely mirrored onto
two other drives (providing for harddisk failure). You might even
be able to get it cheaper.

ofcourse this leaves you with "just" 75 GB to work with, which
isn't much. Unless you use your existing 250 GB drive for
export etc.

If you where to go to 250 GB drives to reach a total of 500 GB
of space you would end up at $770. But prices may be different
in your part of the world, ofcourse!!

However, if you would want to go with RAID 5 you would need
to get a Promise SuperTrak SX6000 controller for $300. You
can now use 3 drives at 200 GB to get:

3x 200 GB IDE drive: 3x $125 =$ 375
total: $675

or $810 for 3 250 GB drives. You keep the same amount of
available space, ie 400 GB or 500 GB.

The first option is almost in your financial reach, but might
produce lots of heat in your case (4 EXTRA drives in there!) and
might even require an extra or a replacement power unit.

It would be far better to decide which footage of those 25 hours
you are going to capture (I would advise to do get multiple takes
of the same material, but you don't need 25 takes of one scene)
to get say around max 200 GB of footage (ie around 15 hours of
RAW DV footage instead of 25 hours) so you can get a Promise
controller that does RAID 1 (like the first one I mentioned) and
only need two 200 or 250 GB harddisks.

This would set you back $340 for a 200 GB solution or $430 for
a 250 GB solution, but only with two drives and you have the
safety of an extra mirror drive.

Ofcourse everything I've talked about is completely INTERNAL
in your case. External RAID is for more costly!

> post? Are SCSI drives recommended? Is there anyway we can

I don't recommend SCSI these days. It still is much more expensive
than IDE/SATA, both in controllers and in drives and might even
produce more heat. IDE/SATA drives should be fine.

> Maintaining a steady workflow is very important to us.

I suggest you do some tests to see how the workflow will work.
Working with a good folder structure and notes will be critical.

> p.s. Could it be that the Maxtor's just a lemon?

yes, that is very possible.

I think Maxtor is just pointing their finger at another company to
keep the problem away from themselves. The problem might not
have to be with the harddisk, but it is the most likely candidate
depending on the kind of "failures" you have had.

How where you connecting your current drive?

[John Michael Harper]

To answer your questions, Rob:

> video, this 250 GB Maxtor external Firewire/2.0 drive has
> corrupted 2x in the past 3 months

corrupted in what way? Was the drive replaced?

Incident 1 (Firewire): Probably during a system restart, the Maxtor's file system was damaged and Windows could not recognize the named drive, but rather recognized the drive as a generic, Local Disk. Windows Explorer could not access any files on the drive. The drive was not replaced. I had initially formatted the drive using Partition Magic 8.0, and received word from support that using that app was the incorrect way to the drive. Maxtor ships it's drives FAT32 and have to be reformatted NTFS. Maxtor support stated that Windows XP was sometimes finicky about this. After reformatting the drive NTFS using the Disk Management tool, everything seemed to work ok.

I wonder though if I deleted the original paritition.....

Incident #2 (USB 2.0): I was using Vegas Video to capture, and I was using Windows Explorer to preview video with ZoomPlayer. All of the video clips that I had captured 100 GB with Vegas, were currently in the Media Bin. In Explorer, I deleted a video clip I didn't plan on using and I got an error message saying that the recycle bin of the drive had corrupted! After the next system restart, the Local Disk phenomenon occurred yet again.

Do you have any thoughts based on the above?

Thanks for your suggestions about RAID storage, I will definitely consider these spec'd out configurations. And I was definitely considering using the Maxtor external for renders, etc.

The challenge of the shooting ratio will in part be helped by batch capturing. This should also protect us somewhat from future hard disk trouble so in case a drive fails, then we can more quickly recapture the source.

Finally, I did come down on Jeff at Maxtor for not seeming to entertain the idea that it was the drive that had failed. He pointed to some possible causes -

1. power failure
2. viruses
3. the firewire card
4. the cable
5. power surge
6. improper shutdown
7. improperly shuttling the drive between computers
8. and Windows

Everything but the drive, right? Anyway, Jeff's analogy that the hard disk was like a glass of water. If you add gasoline or poison to that glass of water it's undrinkable, but if you empty the glass, the glass should be ok. If the glass has got a leak, then that's a problem. He also did cop to the fact that the drive might be bad and offered to replace it no questions, it if I continued to have issues.

I may have had a problem when I reformatted the drive after the first failure. According to Jeff, if you reformat a drive without deleting the partition, then it creates a film over the bad partition. The errors, are in fact, still there.

Jeff said that if the partition is deleted, the drive reformatted using Disk Management, and it writes to zeroes successfully, then it is not the drive at fault.

I think it's a lot easier to understand hard drives than it is Windows XP, simply because Microsoft keeps the understandings of their OS to themselves - like why does my iPOD crash my computer (unrelated topic).

You wrote:"I think some myth busting is in order. I highly HIGHLY doubt this is a Microsoft "thing". Both FAT and NTFS are highly tested and used filesystems, and if it where that easy to destroy things we would not be using Windows. I've been using Windows for over 10 years both private and professionally (including professional hosting) and never ever had such filesystem troubles."

The last part of that statement sounds like an advertisement, although I understand it's an endorsement. Most of us have been using Windows for 10+ years, privately and professionally, just because that's the way it is. Just because you haven't had any file problems doesn't make it so and just because they're Microsoft doesn't make them infallible. I think you know this too, because you only, highly, HIGHLY doubt their culpability in this matter.

In fact, when I called Maxtor, I was incredulous to any suggestion that it wasn't the drive, but because a human being made his case for why it wasn't the drive, I considered another cause. Sure, that's probably his job, but I can get a new drive if I want at relatively little hassle. I could also go find another OS.....waitaminute....(Would I dare to switch to Apple? All my friends use Windows! Who is that Linux baby?)

I'm not looking to assign blame or bust myths, but answers - and while there seem to be a lot of opinions out there, I think I'm just going to have to wait and see about the drive.

Thanks again for your suggestions, especially as this all relates to a good workflow. You've also made me think that as soon as something fails, whether it be hardware or software, that I need to analyze the cause immediately and log it in a notebook (settings, what was I doing, etc.) That way when I'm asking for help and questions such as yours come up, I'm not saying hmmmm.....what was it again?

My lazy solution usually is just to send the error report to Microsoft and then move on. When their page pops up "This Error is Being Researched," it's just not enough.

So as we enter post, I'll do my best to keep a detailed catalog of errors so they are more easily digested by the highly knowledgeable folks of this good community.

Thanks,
JMH

[Rob Lohman]

I'll try to answer as best as I can.

> Incident 1 (Firewire): Probably during a system restart, the
> Maxtor's file system was damaged and Windows could not
> recognize the named drive, but rather recognized the drive as a > generic, Local Disk. Windows Explorer could not access any files

what filesystem was on the drive at that point in time? I assume
it was not NTFS? This should not happen with NTFS, even if it
was improperly paritioned.

> the drive using Partition Magic 8.0, and received word from
> support that using that app was the incorrect way to the drive.

Was there a reason for you to do it this way. I think PM8 just
calls the Windows format routines to format a partition, so I
doubt this could really be an issue. But I don't know if PM8
supports such large drives.

> Maxtor support stated that Windows XP was sometimes finicky > about this. After reformatting the drive NTFS using the Disk
> Management tool, everything seemed to work ok.

It sounds a bit strange that this would "suddenly" happen. I
highly doubt that was the issue.

> I wonder though if I deleted the original paritition.....

That's always a good idea. As it is to use the default tools that
come with Windows. At least that rules out other problems, so
in that case I agree with Maxtor, but it really should not be a
problem.

> currently in the Media Bin. In Explorer, I deleted a video clip I
> didn't plan on using and I got an error message saying that the
> recycle bin of the drive had corrupted! After the next system
> restart, the Local Disk phenomenon occurred yet again.

Since this was NTFS this sounds highly unlikely like a software
issue. I asusme you are virus free and such?

This definitely sounds like a hardware problem like:

1. bad firewire/USB2 port (although it's highly unlikely both would be bad!)

2. bad cable

3. bad controller inside the case

4. bad drive

I assume you DID follow the proper instructions to dismount the
drive in Windows? Using the "Safely Remove Hardware" system
or waiting till the system shuts completely down?

> 1. power failure

this should not be an issue with NTFS. In the worst case you have
one corrupt or deleted file.

> 2. viruses

quite possible, although given your error message not very likely
(usually a virus operates behind the scenes and tries to avoid
giving the user warnings and errors etc.)

> 3. the firewire card

possible, but since you also had a problem through USB2, again
quite unlikely

> 4. the cable

possible, but again also with USB2, so unlikely

> 5. power surge

that's one thing only Maxtor could know

> 6. improper shutdown
> 7. improperly shuttling the drive between computers

same as the power outage

> 8. and Windows

not possible. It might've been a rogue driver for example, but
even that is not very likely

> He also did cop to the fact that the drive might be bad and
> offered to replace it no questions, it if I continued to have issues.

that's good to know!

> first failure. According to Jeff, if you reformat a drive without
> deleting the partition, then it creates a film over the bad
> partition. The errors, are in fact, still there.

I don't see how a high-level soft format could introduce such
magnetic problems. All a format / partition does is write bytes
in a certain pattern to the disk. Now this pattern might not
completely agree with Windows, but it should be completely
overwritten if you re-format/partition. However, it is known that
magnetic signatures can stay behind, but if it did that to such
a degree that Windows cannot reliably read the drive then the
drive is very bad (ie, broken) and needs to be fixed / replaced.

> Jeff said that if the partition is deleted, the drive reformatted
> using Disk Management, and it writes to zeroes successfully,
> then it is not the drive at fault.

Well, this only indicates that the drive allows you to write and
read data to the drive. It does not garantuee that the data can
be stored and read correctly.

> I think it's a lot easier to understand hard drives than it is
> Windows XP, simply because Microsoft keeps the
> understandings of their OS to themselves - like why does my
> iPOD crash my computer (unrelated topic).

Well, that's not entirely true. I'd say we know a lot more about
Windows than the internal working of a harddisk. Yes, the basic
ways of an harddisk are known, but it is a VERY complex device
which is really a black box.

I have 10 years of Windows programming experience and know
how a PC and Windows work lowlevel (Microsoft provides a LOT
of information on how the Operating System works!) it really isn't
that much of a black box, otherwise no-one would make software
for this platform <g>

The problem (in regards to your ipod question) is often that all
problems seem to be coming from Windows although they usually
really aren't (sometimes they are though, not saying there are no
problems with Windows).

Keep in mind that all sort of drivers, hooks, services and applications
are installed into a Windows system and each could produce
results such as crashing the system (depending on privileges).

It also depends on which version you are running. Which version
of Windows are you using with what service pack level?

> " I've been using Windows for over 10 years both private and
> professionally (including professional hosting) and never ever
> had such filesystem troubles."
>
> The last part of that statement sounds like an advertisement,
> although I understand it's an endorsement. Most of us have
> been using Windows for 10+ years, privately and professionally,
> just because that's the way it is. Just because you haven't had
> any file problems doesn't make it so and just because they're
> Microsoft doesn't make them infallible. I think you know this

I can imagine your point of view. The idea I was trying to get
accross was that I am no regular user. I've done a lot of system
administrator and programming on a professional level. I have
maintained and built systems that care for hundred of thousands
of users a day that pound harddisks and the system far futher
than any consumer system.

If there where such basic flaws in the FAT / NTFS filesystem I and
the rest of the world would KNOW about this and it would NOT
be in use. Such levels of data loss is just not acceptable in any
professional manner.

Now I'm NOT saying Microsoft is infallible, heck, we all know that
until recently they didn't really look at security / bugs etc. and
system prior to windows 2000 / XP have been highly unstable.

However, I have trust that at least the NTFS filesystem is really
good and works as it should.

> too, because you only, highly, HIGHLY doubt their culpability in
> this matter.

As said, it is possible they are responsible. You would be the
first user I find that has harddisk problems due to Windows
(at this point in time with the current hardware / software).

> I'm not looking to assign blame or bust myths, but answers -
> and while there seem to be a lot of opinions out there, I think
> I'm just going to have to wait and see about the drive.

That's true. Why not get something like HDTach and test the
drive in a stress test?

> a good workflow. You've also made me think that as soon as
> something fails, whether it be hardware or software, that I
> need to analyze the cause immediately and log it in a notebook

that's always a good thing to do! The more information the better
in solving problems and mysteries, especially over such a medium
as the internet!

> So as we enter post, I'll do my best to keep a detailed catalog
> of errors so they are more easily digested by the highly
> knowledgeable folks of this good community.

good idea! <g>

As a last word (for now), I don't think the Maxtor guy is talking
nonsense, he certainly has some very good points. I just don't
agree with some of his thoughts and remarks. But it sounds like
he is more than willing to help you out which is a real good thing!

[John Michael Harper]

Hey Rob,

Thanks for the encouragement. I ran HDTach 2.70 in the read test, the sequential speed pauses and drops to 0 at 143.7GB. I ran the test 3x with the same result. I repartitioned and reformatted the drive (NTFS) yesterday so in theory it should be clean. Please advise.

John

[Rob Lohman]

Well, that isn't good. A drive should not drop to zero. I can't
remember, but did HDTach have write tests and/or write/verify
tests as well? If so run those (ALL DATA WILL BE DESTROYED!!!!)
and see what happens. It sounds like a faulty drive to me!