![]() |
HVX-200 Archiving
So, the HVX-200 looks pretty nifty, but...
How are you going to archive your footage? Keeping it on a hard drive is not a long term storage option, drives die and are pretty unstable. Ultimately, you're still going to have to back up to tape. The Panasonic AJ-HD1200A is $21K and writes onto DVCPRO-HD, the AJ-SPD850 is $15K writes to a DVD. Prices go up from there for better decks. The only other option is to take your hard drive/Firestore/P2 Store to a post house that can make your ephemeral files into a hard copy for long term archiving. Clients have a hard enough time getting the correct tape stock, or remembering to call the sound guy, how are they going to be able to reliably back up and transfer footage? I'd like to hear how HVX users are archiving and backing up your footage. |
Storage
This has been discussed quite a bit and you can find some good info by using "HVX" and "Storage" in your search.
Personally I'm still looking for that long-term solution. Currently I'm burning selects to DVD and waiting for Blu-Ray. The rest is sitting (precariously) on a 1 TB drive that is about half-full. Good luck in your search! |
As Hans said, this has been well covered earlier however, I'll add these quick tips:
The notion that "hard drives die and are unstable" just isn't true with current drive technology. There are literally tens of thousands - if not millions - of HDD's each with thousands of hours of failure-free time living in servers around the world. 10 years ago, it was common to find drives that died or would lose stability within just a few hundred hours. The MTBF tables that manufacturers claim today are near-gospel truth. Sure, there's always going to be the one bad apple in the bunch just like anything else (how many times have you run across a CD/DVD that just wouldn't burn or read?) but 99.9% of the time you can rely on HDD tech to carry the load. Lastly, I have a 1TB external drive that I'm using strictly for archiving purposes. It only gets turned on when it's time to add more "raw" clips and, to save a current backup of the full project. Outside of that it stays in a powered-off state. With that little amount of usage the drive will become obsolete before it fails - probably within another 10 years - then it would be time to rotate the archives to the newer media type, whatever that becomes. I'd say a 10-year archive-migration schedule is not only prudent but provides the most robust, cost effective and safe storage method there is. |
I have to agree with everything Robert posted... Also take a look at some of the newest tape solutions, they're not all that bad either in terms of price per GB and are also super-reliable.
For overall ease of use and cost though, a hard drive archive solution isn't bad. I would say that with the current cost of storage, it would be wise to not place all your eggs in one basket... Double up on HDDs and keep duplicate storage volumes or keep a tape/DVD/BD/HDDVD/whatever archive of what's on your HDDs. And worst case scenario, if your HDD does go down just remember that because a drive won't work, doesn't mean your data is lost forever. Many HDD failures can be repaired by a skilled geek. |
Hard drives are very reliable. The is how we are archiving our footage...hard drives wrapped in plastic on a shelf next to the Varicam tapes.
DVDs are far more fragile, and prone to scratching and flaking. I'd use them as second redundant backups, but that is it. |
The problem with hard drives is that (a) they can and do fail unexpectedly, and (b) when they fail the data is unrecoverable for any reasonable fee. But if I was backing up data from an HVX200 I'd probably use redundant hard drives because that's the quickest and easiest thing to do, plus then you can edit straight from the drives.
Archiving to DVDs or HD DVDs sounds cost-effective until you consider the time required to do that, plus the effort required to keep track of all the discs. Tape backups are good because tapes can sometimes be salvaged for just a few bucks with minimal loss of data when they break, but then what's the point of having a non-tape-based acquisition format? Hopefully in a few years the cost of flash memory will drop enough that it will be cost-effective to just keep HVX200 data on the recording media and buy a big stack of that. |
Tape: LTO. About half the cost of a DVCPRO HD deck, and it preserves the P2 metadata.
|
Quote:
Compare storage options to archive 2TB of data: 4 x 500GB Hard Drive - $1300 (bare SATA drives) VXA-2 Tape system - $995 w/1 tape and software Additional 12 tapes (13 tapes total = 2080GB) $300 Cool, eh? So at the 2TB mark, HDD and VXA-2 tape are even in price. From there on out, VXA-2 is cheaper and it can hold 2.5 hours of DVCPROHD 100 on a single tape. At the 4TB mark, or roughly 50 tapes, the VXA-2 archival system has an operational cost of $0.40 per gigabyte. At 100 tapes, (yeah, after you shoot 250 hours of video), your archival media cost would be about $0.28 per gigabyte. 8TB of hard drive storage will cost you $0.60 per gigabyte if you buy a bulk carton of those 500GB SATA drives at about $299 each. |
Excellent. Very fine post there, Jeff, thank you.
|
People frequently bring up video tape (i.e., DV or even DVCPRO HD tape stock) as being more reliable than hard drives, but honestly, where's the data to back that up? (--rimshot--) I have had more tapes in various formats fail for me over the years than I'd care to remember. And in this HD world, DVCPRO HD tapes can cost nearly a buck a minute, which is a lot more than a hard drive at 1 gig a minute.
I still just see data when I see this kind of discussion, and as a tech worker for most of my life, it's not any different than any other data other than the volume. Too much fear and and a false sense that good old fashioned video tape was a great archive medium. |
Quote:
|
You cant edit with DLT tapes - you can with HDDs.
|
Not an issue. The DLT holds your original archived, protected material. You wouldn't want to be changing that. Simply ingest what you need into your editing system and do your editing on your NLE's drives.
|
Quote:
How many accumulated hours will it take to copy P2 cards every few minutes to hard disk (RAID 1) and then copy the contents of hard disk to tape? These "time costs" must be added to the media (tape) cost, and only then compared to self-archieving media like XDCAM HD and miniDV tape. On the other hand, with XDCAM HD you need to consider the Player's cost compared to the cost of P2 plus a P2 reader. A "wash?" However, if you are alone -- it's not simply $$$. You need to find the time to write AND verify the tape before you erase your hard drive files. Likewise, before you erase each P2 card, you need the time to copy AND verify to a hard disk. Assuming you can "find" the time --what would be neat is a Shuttle like PC with dual HD drives (RAID) plus a Exabyte VXA-2 drive. Or, a box with these components and a USB connection. |
Well said, Steve. It's this very issue that is the sole determination whether or not this camera is usable in the field. If you're on a distant location and shoot for ten hours (which is the equivalent of what? 2.5 cineporters?) and then, after a long day have to sit up and write the data to tape at best at 2X realtime... it just doesn't add up.
Tape/XDCAM media is here to stay until Flash/P2 is throwaway media under $50/hr. mg |
Quote:
But even at the prices I was figuring, hard drives aren't a bad deal at all and they have a lot of things going for them that tape doesn't. Tape has its advantages too... Mostly I was trying to point out that there are some pretty affordable tape options since most of the tape naysayers complain about how expensive it is compared to hard drive. Realistically, no media out there is perfect and anything can fail. The best is to have a good backup strategy that produces redundant copies of across 2 or 3 volumes in case a tape or hard drive fails. It's the old addage about not putting all your eggs in one basket. If you're smart about how you archive your data and you can make it work with hard drive or tape or even something like DVD-R or BluRay/HDDVD, then that's great. In the end, what works for one won't work for another Tape is a tough nut to swallow if you don't shoot tons of video and don't want all the extra cost up front -- that's where most of the cost of tape archiving is - in the drives. Media does cost, but tape media is probably still the cheapest media out there in terms of $/GB. You also have to figure that most all the moving parts in a tape backup workflow are in the tape drive itself. Now how many hours on that tape head do you think you'll get before you have to buy a new drive? 500 hours of video and 200 tapes down the road, you may be all happy about having a $0.18/GB archival system when suddenly you have to spend another $2K to replace your drive. Now that tape system is just as expensive (or more) than HDDs would have been. |
hard drive prices continue to drop like a rock, but dlt pricing has always been absurdly overpriced.
having spent 10 years as a pc network admin, i can tell you that tape drive interfaces and the operating systems that the tape drive software requires will always become obsolete... how well i remember the panic phone call from my old employer, as they tried to recover a dos-based tape backup that used a parallel port :-/ how many of you still have a parallel port on your computer, or a scsi interface, for that matter? buying into a home-built external raid with removable drive trays, and multiple interfaces like usb, firewire, sata, etc., will work in your favor as time goes on... you don't need software to control it, hdd pricing will continue to drop, and if you keep the drives in sets as they fill up, so you should be able to recover the data if one drive actually dies, which is doubtful. as pc operating systems go, the upcoming microsoft vista is a major upgrade, whereas winxp was actually considered to be a minor upgrade over win2k... and win2k was a major upgrade over win98. so if dlt is what you must have, make sure that the control software that it uses is future-proof... don't limit yourself to a winxp-only dlt solution. |
Quote:
It would be helpful if someone created an Excel spreadsheet that allowed one to enter the type (24p, 1080p, etc.) of video shot, number of minutes continuous shooting, number of minutes per day, number of days shooting in a row, etc. Then the sheet would determine the number and capacity of P2 cards needed. Plus the options for achieving and their cost in terms of money and time. Once a basic sheet was created, it could be modified into TYPES of shooting: ENG, DOC, etc. There's two ways such models could be used: one, for those who have already decided to buy an HVX200 -- what should they buy "in addition." For those who are evaluating tapeless vs tape shooting -- the sheet could be expanded to include XDCAM HD and HDV. There are only a few inherent reasons to exclude any of these alternatives (Adam's tests showed they all produce great HD video) so a cost analysis should, IMHO, be part of everyones buying equation. While "image" may be subjective, total cost of production is not. For geeks -- the sheets could be enhanced to consider falling P2 and optical/HD prices and increased optical/HD capacity and optical disc write speed. And, that would tell one WHEN P2 will be cost effective vs other options. This should be done by someone who favors P2 so there can be no claims of bias. |
But that kind of thinking just doesn't apply in the real world of shooting.
Right now I've only got two 4GB cards. It's all I need, and it works fine. I paid off my complete HVX package on the first job I landed with it. Free and clear. Anything from here on out is just gravy, and my situation is far from unique. It doesn't matter whether some other system offers cheaper cost per gigabyte, because I got this specific tool to do a job, it does it, and it pays rental fees to me every time I use it. I deliver the footage on hard disk to the client, and they love it. Owning the HVX has proven to be a positive-cash-flow generator, regardless of whether HDV tape is $4 each or blu-ray disks are $30 or whatever. The client pays me more than it cost me, so how can it possibly get more cost-efficient than that? It's positive cash flow. Everyone I know of who's using the HVX professionally will tell the same story. It's not a case of "how many tapes do I have to save on not buying before P2 costs break even"... it's a case of "oh, you can shoot high-def? So I don't have to rent a $1200/day camera? Shoot this project for me, I'll pay $600/day for the camera package, plus your normal rate." That's how it's working in the field. And yes, the clients do ask to see footage before voting for the HVX over the VariCam and CineAlta. And yes, every one of them so far has voted for the $600/day HVX vs. the $1200/day VariCam and $1400/day CineAlta. Not that there isn't a difference in the footage, of course there is -- but on a cost/benefit ratio, they've chosen the HVX every time. Peter Kagan'll tell you about a national commercial for Subway, starring Heisman trophy winner Reggie Bush, where he offered to shoot it on his own Aaton 35mm camera, and instead they opted for the lower cost of the HVX, and they were thrilled with what they got. He told us at the HD Bootcamp that he's now ready to sell his Aaton. I mean, I guess the cost-per-gig-storage thing is relevant to some people who are closed shops, one-man-band type of things, weddings and whatnot, but -- if you're working with clients and agencies, doing advertising, infomercials, or other types of paying work, it's just a total non-issue. It isn't even a factor. You add a $150 hard disk into the budget, rather than a box of HD tapes and $150/hr deck rental time, and you go from there. |
Quote:
Bottom-line, positive cash flow is not how a business evaluates itself. If it were, all companies that had a cash flow that was positive would have the same stock price. They don't, because that is not a useful measure of how succesful a business is. It's like saying you have "passing grades." That won't get you a great job. |
Quote:
Quote:
And parallel ports and SCSI ports are still widely used and are industry standard. Not to mention that all current SCSI systems are backward compatible to the old 25pin SCSI-I, provided the proper connectors/adapters are used and the devices are properly placed on the SCSI chain. Next to the latest fiber channel implementations, UWD SCSI is still the highest performance storage interface out there. Parallel ports are still standard fare on just about every PC system out there as so many parallel devices are still in use. USB to Parallel or Serial adapters are pretty commonplace too and cost about $35. There's a good bet that (S)ATA hard drive interfaces will be non-standard, obsolete crap within 10 years. Is this a big deal? No, because there will still be plenty of adapters and interface cards out there to help us out. Quote:
Quote:
Quote:
Same applies to HDD. Keep the data natively raw. If you must compress, use a standard method. |
Whatever, Steve.
NASA's buying the HVX, and I dare say they stringently evaluated its performance. The BBC is buying the HVX, and there's nobody who evaluates a camera more thoroughly. Independent producers are doing great work with it and making money with it. Lots of it. We'll continue to do so, whether you think a blu-ray disk is cheaper per gigabyte or not. For those of us who are actually in the business, it's kind of hard to beat having gear that pays you to use it, vs. using gear that you have to pay to use. |
After having compared the HVX to other cameras in its price range, I'd say the HVX feature list is what's selling the camera.
|
As a long time editor, I have used DLT frequently for backing up media from completed projects. The DLT's I have used were very slow, perhaps newer models are faster. However, I never had a DLT fail. The actual process didn't impact operator hours that badly because the accepted workflow was to set it up at the end of the day and let it backup overnight. I restored many projects that needed updates years after the original edit. For large projects we restored overnight. Saving hundreds of gigs of media via DVD or even Blue Ray is going to require alot more operator time than the various data tape systems. Down the road, new archival technologies like holographic storage will become available. In the meantime data tape formats are useful and cost effective. Hard drives are fine as well but if I was backing up media that there was no original tape source for such as P2, then I would double backup to two hard drives for safety. I suggest that those who put all their eggs into one hard drive have probably never faced the prospect of a hard drive that has died with valuable media that can't be replaced at any cost. These cost-per-gig comparisons do not factor in the value of the media itself if you had to reshoot media that was somehow lost.
|
Quote:
make no mistake about it, what we are talking about here is utilizing the best i.t. practices we can afford for backing up our data, and in the real world, some people feel that tape is dead: "But when it comes to backup and recovery of essential data, tape is fading from the lineup of usual technology suspects. It's glitch-prone, for one thing. A Yankee Group survey of IT executives last year found that 42 percent of respondents were unable to recover data from tape in the previous year as a result of tape unreliability." http://www.businessinnovation.cmp.co...eature_4.jhtml Quote:
Quote:
"According to consulting firm TheInfoPro, 90 percent of companies plan to move from tape to Serial Advanced Technology Attachment (SATA) disk drives as part of their backup/recovery and data lifecycle management plans by 2006." http://www.businessinnovation.cmp.co...eature_4.jhtml Quote:
|
Quote:
that was about 10 years ago, which gives you an idea of how old the dlt format is... we lost mission-critical data off of a file server because of it. i'd like to believe that modern dlt tape is more reliable. |
Quote:
We really want the P2 contents to go to another random access media that we can easily view the contents to find something we can only see or hear. Something we have a memory of. That means either optical or hard disk that can be mounted like a P2 card. Once you decide this: 1) You can debate optical vs HD cost-per-bit 2) You can debate optical vs HD reliability 3) You can debate optical vs HD storage life But, there is one thing beyond debate -- SPEED. Only a HD can move data at the rates possible with P2. And, while some who shoot only a couple of 4GB cards a day may not care, to replace tape or challange XDCAM HD, FASTER THAN REALTIME transfer is a must. IMHO this means a HD in a nice package. Which means a visit to the Grass Valley booth to see how they use the REV HD. Lastly, if you really, really want to insure data safety, HD can be run as a RAID. That's a unique option. |
Quote:
It also remains to be seen if Grass Valley will produce a 1/3" variant in the future. Such a beast would have the potential to give the HVX200 a real hiding, certainly from the archive/storage viewpoint. And yes, any discussion of archive costs must take into account time as well as media costs. |
Quote:
|
Quote:
If folks want super fast random access -- the major virtue of P2 -- because they don't want to seach though video tapes, then they certainly don't want -- the moment they erase a P2 card -- to place it's contents on a many, many times bigger data tape that can't be visually searched. And, it's crazy to copy once to a HD and then later copy again to data tape. Time is money. It seems very clear to me that only HDs offer super fast random access (as does P2), super fast copy to, plus current huge capacity, and rapidly decreasing cost-per-bit. Thus, I reject both data tape and optical disk. Which, of course, is why GV has gone with HD right from the get go. And, why so many folks want to attach a HD to their camcorders. This indicates to me that if Sony and Panasonic weren't so interested in selling propritary media -- they would have simply gone with HD. |
And HDs are coming, with the FireStore and the CinePorter and the CitiDisk and others. There are ways around proprietary media. And it hasn't been all that proprietary anyway -- if Panasonic wanted to lock you into P2, they wouldn't have provided firewire streaming which lets you totally bypass P2.
Hard disks are what people are using. They're dirt cheap, reasonably fast (plenty fast in a RAID), easily accessible, and getting bigger, faster, and cheaper every week. Hard disks do have their drawbacks for acquisition, but for editing they're ideal. |
Quote:
Quote:
Quote:
You might think it needs to be visually searchable, but that's only because you don't know any other way to find a clip. A visual search is an incredible time waster (hence your desire for fast random access of the archive). Obviously though, the better solution in terms of a more efficient use of time is to simply enter some search terms... whether they're TC numbers, dates, names, addresses, project codes, GPS coordinates, slate numbers, whatever... and go directly to the clip(s) you need to pull from the archive. Clearly that's a smarter way to work than visual scanning. It's the same circumstances as well for using LTO as an immediate back-up to ingested material. The working copy is in the editing system, a bit-for-bit backup is nearby on LTO. If the NLE drives crash, then you'll need the entire contents of that tape anyway, so super fast random access is of no practical value there either. |
Quote:
So every few minutes, the P2 cards -- over a 10-12 hour day -- somewhere in the "African bush" -- have got to be copied and the P2 cards erased. The HD(s) might, or might not, be used ON the NLE. Frankly, I think they go into a vault and the NLE uses a copy of the logged portions. Any time they are need they are remounted. There is no need, to copy the contents again to tape! "All that's needed is an accurate, searchable index. Efficient archive retrieval isn't a question of random access speed; it's a question of effective data management techniques." Meta-data, shmeta-data. We all know that we what we "should do" but we almost never do it. I'd much rather plug some drives into SATA connector than count on meta data. "If the NLE drives crash, then you'll need the entire contents of that tape anyway, so super fast random access is of no practical value there either." Again you are assuming a very different model where you edit the intermedite storage. I'm assuming a much more time efficient model where there are only P2 and HD. Both are fast and both are random access. In other words, I'm trying to keep the positive P2 attributes from beginning to end -- while overcoming the one P2 negative -- cost. Believe it or not, I'm trying to see how to make P2 practical for long-form work, field work. Picture Panasonic's 60B drive field unit. But, picture it with a slot for a REV cartridge of say 300GB. OK -- backup data tape may be cheaper, but if CHEAP was your primary goal you likely would not go down the P2 path to start with. P2 implies you value fast and random. OK -- you might just want an HVX200 and really not care about P2. But, Panasonic is betting its future not on the folks who want an HVX, but on folks wanting P2. And, it's archiving -- not HVX200 image quality or features that is the focus of this thread. |
Your model is *one* way to work. It's certainly not *the* way to work, and that's being generously diplomatic. The overriding point is that with this format it's all simply data in an IT stream from the moment of acquisition forward. And because of that, P2 adopters can choose from a variety of viable options for immediate backup and long-term storage purposes. To each his or her own.
I'm glad we can agree that archiving is the topic of this thread. |
Quote:
|
Quote:
Quote:
Quote:
Quote:
Quote:
Just curious... How much real world experience do you have with this type of stuff? Or do you just take what's printed in these run-of-the-mill, IT publications as the gospel? If you want, I can reference articles that refute what you have quoted here. No point in it... I've been everything from a hardware designer, to a software developer, a CG artist and more over the last 20 years and have run two web companies and a large datacenter. The most important thing Iv'e learned is that there's no such thing as an "expert" in the IT business. And over the years, I've seen more of these "experts" and publications turn out to be wrong in their predictions or accounting than they are right. Quote:
Doing incremental or intelligently structured backups becomes a very laborous task without special software. But my point may have been lost in what I was saying. Essentially, I was getting at using a proper archiving solution rather than typical backup software. Most backup softwares package up your data into proprietary containers, requiring you to have that same company's software to restore the data. Bad idea for a long-term archival solution. Once again, I'm not preaching DLT or any tape format. I'm pointing it out as a viable option. Too many people here have discounted tape as something of the past, yet many of those same people were the most vocal about the HVX not being HDV because they want to have their backups already there on a tape. Pretty ironic, really.... Some of the tape naysayers are also the guys talking about doing their backups on BluRay or HD-DVD. Pretty silly to me... They're going to say that VXA2 tape is too expensive because you have to buy a $1K tape drive and $22 per 160GB tape. And then they say they're going to buy a $1K BluRay drive and $25 per 25GB disc media. OK, fine, we'll all probably buy BluRay drives so we can author HD content. But as an archival solution does it make sense? It's not even an available product yet and nothing is proven about its reliability or viability as a long-term format... I've never discounted HDD as a backup/archival solution. I would question some of the plans posted here for long-term archival on HDD, just from a logistical point of view of having to deal with hard drives and connectors/cabling, etc... if you ever need to pull significant amounts of data from your archive. At least the way most people here are talking about using them (disconnect once the drive is full and wrap it in a static bag and bubble wrap and shelve it). Hmmm... If I was going to archive on HDD, I'd set up SATA enclosures with a RAID-5 config. Unplug the RAID once it's full and shelve it with a dust cloth or bag over it and connect a new one. But then we have to refigure the cost... Now it's buying 3 to 5 drives at a time plus a decent enclosure for them. Just like anything else, you get what you pay for and it usually costs a bit more to do something right. |
Quote:
|
Quote:
-------- I once had a tape backup system in which the tape cassette was not much bigger than a large DVCPRO cassette. Now, if this is today's DLT that writes at 80MBs, and if a tape can be mounted as a logical volume on the desktop for near random access of files, then it could be used anywhere. Each P2 card would incrementally be copied to tape. When one got ready to edit -- one would copy the files to disk. Then the low-cost tape goes on a shelf. Perhaps, a portable tape drive with a P2 slot would work well in the bush. |
Quote:
the thing to look for with p2 hdd acquistion is the trend towards bigger, cheaper hard drives... so you can always swap in a bigger, cheaper hdd, or get bigger, cheaper p2 cards, but with tape, the storage capacity limitations will never improve. |
What's missing is the integration you talk of that is needed.
Once a company puts the commodity drive in a package that includes a SATA interface, it won't be $40! And, if SATA isn't part of the package, then the disk package will too require the purchase of a "player" because the drivebay simply isn't part of the equation for many. There is a difference between putting "media" on a shelf and putting a drive+SATA INTERFACE on a shelf. |
All times are GMT -6. The time now is 06:22 PM. |
DV Info Net -- Real Names, Real People, Real Info!
1998-2025 The Digital Video Information Network