HVX-200 Archiving

[Barry Green]

And HDs are coming, with the FireStore and the CinePorter and the CitiDisk and others. There are ways around proprietary media. And it hasn't been all that proprietary anyway -- if Panasonic wanted to lock you into P2, they wouldn't have provided firewire streaming which lets you totally bypass P2.

Hard disks are what people are using. They're dirt cheap, reasonably fast (plenty fast in a RAID), easily accessible, and getting bigger, faster, and cheaper every week. Hard disks do have their drawbacks for acquisition, but for editing they're ideal.

[Chris Hurd]

Originally Posted by Steve Mullen

If folks want super fast random access -- the major virtue of P2 -- because they don't want to seach though video tapes, then they certainly don't want -- the moment they erase a P2 card -- to place it's contents on a many, many times bigger data tape that can't be visually searched.

You're confusing the immediate editing stage with the long-term archival stage. Super fast random access, the major virtue of P2, is already there at ingest where it's most needed. The data remains on the super fast random access hard drives of the NLE throughout the editing process. After the project is finished, then it becomes a question of how to transition all the data to long-term storage. That's where LTO is an attractive option, and of course, super fast random access just isn't needed for archiving purposes. We're putting this material away for awhile. We might, or might not, need it later. Most of the time we won't need it later, but if we do, then it's there. Within that context, how does having super fast random access benefit us? It doesn't. Simply knowing where the material is... properly logging it and creating a searchable index... is much more beneficial for an archive. Because most likely we're digging into the archive only to pull out one or two shots, not to edit the whole thing again.

And, it's crazy to copy once to a HD and then later copy again to data tape. Time is money.

Nonsense; that's an easily automated process. No big deal at all.

It seems very clear to me that only HDs offer super fast random access... Thus, I reject both data tape and optical disk.

You can reject it, but that doesn't mean anybody else has to reject it. The LTO3 option is best described as an archival medium, a searchable one at that, provided the appropriate P2 meta data was properly annotated in the first place. Super fast random access simply isn't a requirement for pulling material out of archival storage. All that's needed is an accurate, searchable index. Efficient archive retrieval isn't a question of random access speed; it's a question of effective data management techniques.

You might think it needs to be visually searchable, but that's only because you don't know any other way to find a clip. A visual search is an incredible time waster (hence your desire for fast random access of the archive). Obviously though, the better solution in terms of a more efficient use of time is to simply enter some search terms... whether they're TC numbers, dates, names, addresses, project codes, GPS coordinates, slate numbers, whatever... and go directly to the clip(s) you need to pull from the archive. Clearly that's a smarter way to work than visual scanning.

It's the same circumstances as well for using LTO as an immediate back-up to ingested material. The working copy is in the editing system, a bit-for-bit backup is nearby on LTO. If the NLE drives crash, then you'll need the entire contents of that tape anyway, so super fast random access is of no practical value there either.

[Steve Mullen]

Originally Posted by Chris Hurd

You're confusing the immediate editing stage with the long-term archival stage. Super fast random access, the major virtue of P2, is already there at ingest where it's most needed. The data remains on the super fast random access hard drives of the NLE throughout the editing process.

I'm not confusing anything -- as in my model the acquisition phase may be months long and carried out nowhere near an NLE. The NLE comes into play when the on-location work is complete. The editor may not even be hired yet.

So every few minutes, the P2 cards -- over a 10-12 hour day -- somewhere in the "African bush" -- have got to be copied and the P2 cards erased.

The HD(s) might, or might not, be used ON the NLE. Frankly, I think they go into a vault and the NLE uses a copy of the logged portions. Any time they are need they are remounted. There is no need, to copy the contents again to tape!


"All that's needed is an accurate, searchable index. Efficient archive retrieval isn't a question of random access speed; it's a question of effective data management techniques."

Meta-data, shmeta-data. We all know that we what we "should do" but we almost never do it. I'd much rather plug some drives into SATA connector than count on meta data.

"If the NLE drives crash, then you'll need the entire contents of that tape anyway, so super fast random access is of no practical value there either."

Again you are assuming a very different model where you edit the intermedite storage. I'm assuming a much more time efficient model where there are only P2 and HD. Both are fast and both are random access. In other words, I'm trying to keep the positive P2 attributes from beginning to end -- while overcoming the one P2 negative -- cost.

Believe it or not, I'm trying to see how to make P2 practical for long-form work, field work. Picture Panasonic's 60B drive field unit. But, picture it with a slot for a REV cartridge of say 300GB.

OK -- backup data tape may be cheaper, but if CHEAP was your primary goal you likely would not go down the P2 path to start with. P2 implies you value fast and random.

OK -- you might just want an HVX200 and really not care about P2. But, Panasonic is betting its future not on the folks who want an HVX, but on folks wanting P2. And, it's archiving -- not HVX200 image quality or features that is the focus of this thread.

[Chris Hurd]

Your model is *one* way to work. It's certainly not *the* way to work, and that's being generously diplomatic. The overriding point is that with this format it's all simply data in an IT stream from the moment of acquisition forward. And because of that, P2 adopters can choose from a variety of viable options for immediate backup and long-term storage purposes. To each his or her own.

I'm glad we can agree that archiving is the topic of this thread.

[Chris Hurd]

Originally Posted by Steve Mullen

Picture Panasonic's 60B drive field unit. But, picture it with a slot for a REV cartridge of say 300GB.

A year ago I posted the observation that Panasonic most likely will lean in that direction. The statement I made suggested a clue in the current product model number: AJ-PCS060. If there were no plans to ever upgrade that thing, then it would have been called the AJ-PCS60. The extra "0" should be our best indication that an AJ-PC240, AJ-PC360 or AJ-PCS500 might not be too far away. A removeable cart slot makes too much sense, so I wouldn't count on it.

[Jeff Kilgroe]

Originally Posted by Dan Euritt

that is not correct, but the fact that you are even preaching an ancient dead format like dlt, instead of lto, pretty much sez it all.

When I say "DLT" I'm using it as a blanket statement to refer to all Digital Linear Tape devices, including LTO, AIT, VXA, etc... As most manufacturers and industry professionals do. The only specific product that actually calls itself "DLT" is Quantum's offerings and their latest S-DLT4 products are hardly ancient and dead. In fact they currently offer the highest data capacities in the industry and some of the fastest transfer rates. But I'm also not preaching DLT... In fact, I have only mentioned a specific DLT product once in this thread (the 400GB S-DLT3) as a pricing example. The product I've discussed most and come closest to preaching about (in terms of tape) is the Exabyte VXA2 format as it would fit into budget for a lot of HVX200 users. And at a point of 220 hours of 100Mbps video storage, it would become the same price as shooting HDV to miniDV tape.

make no mistake about it, what we are talking about here is utilizing the best i.t. practices we can afford for backing up our data, and in the real world, some people feel that tape is dead:

I think I've already said that. I also said that tape is not for everyone and HDDs may be the best options for some. I often wonder if people in these forums actually read an entire post before jumping all over someone.

"But when it comes to backup and recovery of essential data, tape is fading from the lineup of usual technology suspects. It's glitch-prone, for one thing. A Yankee Group survey of IT executives last year found that 42 percent of respondents were unable to recover data from tape in the previous year as a result of tape unreliability."
http://www.businessinnovation.cmp.co...eature_4.jhtml

Blah, blah, blah.... And this means what? It's just IT industry FUD and pointless, foundless statistics. "It's glitch-prone" -- that's a classic. What isn't? If you can look at me with a straight face and tell me that HDD solutions or optical disc or any other common format also isn't glitch prone, then I'll nominate you for an Academy Award.

but why buy ancient technology like scsi, when you can get fiber channel? the best bang for the buck is sata, by far, and it's currently headed for 600MB/sec and beyond.

Why buy SCSI? Hmmm.... Perhaps because it's a continuously evolving standard that provides excellent ROI in the right circumstances. Many fiber channel implementations are in fact fiber SCSI and can allow for connection of SCSI devices within the system. Just like any other technology, SCSI is not ideal for every user or situation. SATA is great and I agree that a huge percentage of the industry is headed that way (wait a minute, I think I've already said that in a previous post). But anyone saying that it's the best option isn't considering all the possibilities, nor are they considering the potential needs of others. In some installations, SCSI may still be the best option... If this weren't true, SCSI would have vanished from the marketplace and would no longer continue to be developed. You do realize that the SCSI-5 standard has preliminary approval and should hit the market within the next year? 600MBs SATA-3 is puny compared to SCSI-5 at 4GBs per channel with up to 4 channels per interface.

wrong again, jeff:
"According to consulting firm TheInfoPro, 90 percent of companies plan to move from tape to Serial Advanced Technology Attachment (SATA) disk drives as part of their backup/recovery and data lifecycle management plans by 2006."
http://www.businessinnovation.cmp.co...eature_4.jhtml

More FUD. But so what... I'm sure many companies will continue to expand their datacenters with more drive storage and primary backup systems based on hard drives. Only makes sense... And yet companies like Quantum, StorageTek and Exabyte continue to develop new tape products and large scale tape-based archival systems capable of shuffling thousands of petabytes of data on a regular basis and their sales and profits continue to grow. Neat.

Just curious... How much real world experience do you have with this type of stuff? Or do you just take what's printed in these run-of-the-mill, IT publications as the gospel? If you want, I can reference articles that refute what you have quoted here. No point in it... I've been everything from a hardware designer, to a software developer, a CG artist and more over the last 20 years and have run two web companies and a large datacenter. The most important thing Iv'e learned is that there's no such thing as an "expert" in the IT business. And over the years, I've seen more of these "experts" and publications turn out to be wrong in their predictions or accounting than they are right.

so a tape drive just plugs into the computer to be accessed like a hard drive? and you can do incremental backups to it without a software program? that's certainly news to me :-)

You obviously haven't used many tape systems. No, they don't act like a hard drive, they act like a tape drive. As posted earlier, there are ANSI and IEEE standards for tape drive I/O. There are ISO standards for tape file systems as well. That's why the generic Windows backup program can talk to and utilize just about any decent tape drive on the market without using the "BACKUP" software that comes with the drive. Just as a CD drive works on a system. You plug it in and the system knows how to talk to it. If you use an ISO standard format, you can read the CD on a PC or Mac or SGI or any system that understands ISO formats. Most Linux distributions as well as most other unix variants like Irix, Solaris, BSD, etc.. also have simple backup utilities that can talk to industry standard tape systems.

Doing incremental or intelligently structured backups becomes a very laborous task without special software. But my point may have been lost in what I was saying. Essentially, I was getting at using a proper archiving solution rather than typical backup software. Most backup softwares package up your data into proprietary containers, requiring you to have that same company's software to restore the data. Bad idea for a long-term archival solution.

Once again, I'm not preaching DLT or any tape format. I'm pointing it out as a viable option. Too many people here have discounted tape as something of the past, yet many of those same people were the most vocal about the HVX not being HDV because they want to have their backups already there on a tape. Pretty ironic, really.... Some of the tape naysayers are also the guys talking about doing their backups on BluRay or HD-DVD. Pretty silly to me... They're going to say that VXA2 tape is too expensive because you have to buy a $1K tape drive and $22 per 160GB tape. And then they say they're going to buy a $1K BluRay drive and $25 per 25GB disc media. OK, fine, we'll all probably buy BluRay drives so we can author HD content. But as an archival solution does it make sense? It's not even an available product yet and nothing is proven about its reliability or viability as a long-term format...

I've never discounted HDD as a backup/archival solution. I would question some of the plans posted here for long-term archival on HDD, just from a logistical point of view of having to deal with hard drives and connectors/cabling, etc... if you ever need to pull significant amounts of data from your archive. At least the way most people here are talking about using them (disconnect once the drive is full and wrap it in a static bag and bubble wrap and shelve it). Hmmm... If I was going to archive on HDD, I'd set up SATA enclosures with a RAID-5 config. Unplug the RAID once it's full and shelve it with a dust cloth or bag over it and connect a new one. But then we have to refigure the cost... Now it's buying 3 to 5 drives at a time plus a decent enclosure for them. Just like anything else, you get what you pay for and it usually costs a bit more to do something right.

[Jeff Kilgroe]

Originally Posted by Chris Hurd

Your model is *one* way to work. It's certainly not *the* way to work, and that's being generously diplomatic. The overriding point is that with this format it's all simply data in an IT stream from the moment of acquisition forward. And because of that, P2 adopters can choose from a variety of viable options for immediate backup and long-term storage purposes. To each his or her own.

I think that's where so many people get all fired up over this tape vs. HD vs. optical disc debate... Everyone is considering their workflow or the way they want to work and not considering other possibilities. Just like Steve's example above, tape doesn't make sense. I think in that african bush scenario, I would only trust redundant RAID5.

[Steve Mullen]

Originally Posted by Jeff Kilgroe

I think that's where so many people get all fired up over this tape vs. HD vs. optical disc debate... Everyone is considering their workflow or the way they want to work and not considering other possibilities. Just like Steve's example above, tape doesn't make sense. I think in that african bush scenario, I would only trust redundant RAID5.

When I brought up the idea of creating several spreadsheets, I did so in hopes we could create CATEGORIES: ENG, Wedding, Long-form, etc.

--------

I once had a tape backup system in which the tape cassette was not much bigger than a large DVCPRO cassette. Now, if this is today's DLT that writes at 80MBs, and if a tape can be mounted as a logical volume on the desktop for near random access of files, then it could be used anywhere.

Each P2 card would incrementally be copied to tape. When one got ready to edit -- one would copy the files to disk. Then the low-cost tape goes on a shelf.

Perhaps, a portable tape drive with a P2 slot would work well in the bush.

[Dan Euritt]

Originally Posted by Steve Mullen

Perhaps, a portable tape drive with a P2 slot would work well in the bush.

tape is the worst format you can use in a dirty environment... a sealed hard drive would be the better choice.

the thing to look for with p2 hdd acquistion is the trend towards bigger, cheaper hard drives... so you can always swap in a bigger, cheaper hdd, or get bigger, cheaper p2 cards, but with tape, the storage capacity limitations will never improve.

[Steve Mullen]

What's missing is the integration you talk of that is needed.

Once a company puts the commodity drive in a package that includes a SATA interface, it won't be $40! And, if SATA isn't part of the package, then the disk package will too require the purchase of a "player" because the drivebay simply isn't part of the equation for many.

There is a difference between putting "media" on a shelf and putting a drive+SATA INTERFACE on a shelf.

[Steve Mullen]

Follow this link for info on a SATA HD RAID solution:

http://www.tomshardware.com/2006/04/...age/index.html

[Dean Sensui]

I was looking at the cost of DVCPro HD tape and SATA drives and came up with this comparison. Keep in mind that prices vary somewhat but this is a pretty good starting point:

DVCPro HD tapes
126 minutes: $80 = 64 cents/minute
64 minutes: $31 = 49 cents/minute
$25,000+ for deck!
Add backup costs.

250 GB drive: $120 = 48 cents/minute
No deck needed. Quick random access
Add backup costs.

If you get the right SATA hot swap system, sleds for the bare drives are about $20 each. Not bad.

[Scott Auerbach]

Yikes. You say tomayto and I say tomahto. <g>

That said, (both ways!)... I have to come down on the side of the Death-to-Tape fanatics. We used DLT for a short while, around 8 years ago, on a PBS educational series. The read/write speed was abysmal, even by the standards of the day. The one time we needed a recovery, the tape fell apart.

Your mileage may vary.

Overlooked in most of this holy war is this basic fact:
Archiving? Pshaw!

~~Anything~~ we archive on is going to be largely obsolete in 5 years. That's hardly an "archival" time frame. It's stopgap.

It seems to me that one of the most important considerations is going competely overlooked: the speed at which 5 years of archived material will be able to be cloned over to the current available media storage solution in 2011.

I know this from my painstaking experience of transferring a zillion 100MB Zip drives to DVD-R using a USB Zip drive. How many of us have put off dubbing old <insert here: Hi-8, Beta SP, 3/4", whatever> tapes because the real-time dubbing process was just too awful to consider? (Author's note: my hand is raised. I still have ~~standard 8mm movies~~ to transfer!)

Fast and high capacity, good. Slow and cumbersome, bad. SATA-II drives do it for me. They're as fast as anything out there, somewhat reasonably priced, and might even have a little extra lifespan eked out by putting them in a different, future enclosure when eSATA becomes obsolete. The drives' read/write speed might be slow by 2011's standards, but at least you'd still be able to hook them up to your computer with minimal investment. And I'll take a "slow in 2011" 3Gb/sec transfer speed over DLT any day. Especially since the DLT's oxide will be flaking like lead paint in a tenement by the time you get around to copying it to newer media.