A buddy of mine raised an interesting question the other day and I'm not sure I have a good solution for his problem, I was wondering if someone here might.

The company he works for has a long standing relationship with a particular client who likes the idea of keeping every bit of footage shot for them around for future use. Problem is that a large portion of the early footage they have for this company was shot on Panasonic M2 tapes and the last remaining decks they have around that can play that format back are getting very old and are eating up space in the dub room for no apparent reason. They'd like to get all the footage off the old M2's and transfer them to something else for archival purposes but are stuck with making a decision of transferring 100's of tapes worth of footage to another tape format or placing all that footage on redundant drives.

What is a good way to go about doing something like this? If they go with drives, what codec would you recommend?

If they go with tapes or some other type of recording medium what would you recommend using that will be with us for quite some time?

We've been debating it the last few days and haven't come up with a good solution. Anyone know of anything?

BluRay data discs?(pricey per GB) LTO?(kinda old technology) Something I've never considered?

Took me a few moments to even find what this format was.

MII (videocassette format) - Wikipedia, the free encyclopedia (http://en.wikipedia.org/wiki/Panasonic_MII)

Given that it is standard definition, going with the DV codec could be a natural choice.
WD MyBook Essential 1 TB USB 2.0 external drives can be had for $125 USD.
If you use drives in pairs for redundancy, that's 72 hours of video storage for $250 USD.
And the space is far smaller than the boatload of tapes you have now.

I'm not sure how I feel about a long term archive "living" on drives, unless the usual archiving steps are followed ie. fire them up every xxx months/years and "exercise" them.

DV codec is certainly an option. I personally would probably look at a less lossy format, given that MII is a component format and should have quite reasonable quality, assuming that the issues that plague archived analog tapes haven't taken their toll.

I'm not sure how I feel about a long term archive "living" on drives, unless the usual archiving steps are followed ie. fire them up every xxx months/years and "exercise" them.
DV codec is certainly an option. I personally would probably look at a less lossy format, given that MII is a component format and should have quite reasonable quality, assuming that the issues that plague archived analog tapes haven't taken their toll.


Ahh, see now we're getting somewhere. I don't particularly like the idea of archiving on drives myself and this is why I have a hard time recommending this to my buddy. I also don't really think DV is the best codec route to go for the very reason you pointed out and that's why I was looking for other opinions. What codec would you use knowing what you do about MII?

As far as the problems of analog archive tapes, I think they are planning on cleaning the heads of the deck every few tapes to guard against too much buildup of the metal oxide or whatever magnetic substance that MII used since that stuff tends to separate from the plastic backing over the years.

If not drives, what would you go with? LTO tapes are an option but wouldn't they have some of the same problems storing on analog tape would have, namely the magnetic tape breaking down over time? Would BluRay be better because of this reason?

If not drives, what would you go with? LTO tapes arean option but wouldn't they have some of the same problems storing on analog tape would have, namely the magnetic tape breaking down over time? Would BluRay be better because of this reason?

LTO also has the added issue of being a smaller tape base which MIGHT lead to even more rapid physical media breakdown than the original MII tapes. The advantage is that the INFORMATION is digital and less prone to degradation than the original analog media but that 1/4" tape size... long term storage? umm....

I would recommend BluRay-Rom (or whatever it's called) IF there was greater market acceptance than there is right now. I'd hate to see it be a fizzle.

As far as codec, I'd probably go with uncompressed OR a good high bitrate digital intermediate. Of course, the more common the codec (uncompressed) the greater the likelihood that it will be supported when you need to grab media.

I wish I could provide better advice but I'd definitely try to hold on to as MUCH resolution (spatial and temporal) and colour information as absolutely possible.

Two years ago, I would have suggested a transfer to DigiBeta.

As far as codec, I'd probably go with uncompressed OR a good high bitrate digital intermediate. Of course, the more common the codec (uncompressed) the greater the likelihood that it will be supported when you need to grab media.

I wish I could provide better advice but I'd definitely try to hold on to as MUCH resolution (spatial and temporal) and colour information as absolutely possible.

Excellent point about uncompressed. The file sizes for 100 + tapes (many of which are the large 90 minute variety) would be staggering I'm afraid, but maybe that's just something theyll have to deal with.

What has been the accepted practice for this type of thing? Transferring to another tape format?

At my former 9 - 5, we had a bunch of old non-SP 3/4" U-Matic (in the 100's). What we would do with those was a transfer to DVD, mostly because the quality was SO terrible, as calls for access came up.

I'm not sure what I would do in similar circumstances. I've now walked away from 2 U-Matic archives and only need to face MY archives of Betacam, BetaSP, BetaSX, Hi-8, S-VHS, DV, DVCam and HDV now...

At the risk of sounding glib, any solution is a good one UNTIL it fails. Hard Drive, DLT, BD-R, digital video tape... All require SOMETHING to keep them viable into the future. Perhaps once Flash memory prices drop a bit more, THAT may be viable as well...

I wouldn't transfer them to another tape based format. That's just asking for trouble. Tape is on the way out. I also wouldn't transfer them to DV codec data files. I did a test the other day between the following SD codecs:

DV (180.7 mb)
ProRes (234.9 mb)
ProRes HQ (334.4 mb)
8-bit Uncompressed (840.8 mb)

It was amazing how poor quality the DV footage looked compared to the others. Most people had trouble telling the difference between ProRes, ProRes HQ and 8-bit Uncompressed, but EVERYONE could instantly tell when they were looking at the DV footage.

As MII is basically the same as Beta SP, I would recommend ProRes for the codec, but you'll have to decide how to archive the data. I've heard lots of people raving about the LaCie Rugged drives. Saw a guy speak about videography in Iraq and he stored everything on Rugged drives. Hasn't lost data yet and he's been using them for quite a while now in far more demanding conditions than we will ever use them.

The scariest thing I ever heard about archiving to hard drives is when a reputable source told me that storing a hard drive on the shelf is kind of like storing a car in the garage. Over many years, the car's internal engine components will start to rust from lack of use. He told me the same is true with a hard drive. He went on to say that his friends would get together once a month and have "hard drive spin up parties". (what a geek!)

Anyway, now you know everything I do about hard drive archiving. (but it's still what we're using. Western Digital My Studio portable FW800 drives. Buy them at the local Best Buy)

My only concern with ProRes is that it's tied so closely to Apple and FCP. If they were to change NLE's or ProRes support disappeared tomorrow then what? If I knew they'd be FCP for the next 10 years then ProRes sounds great. Maybe I'm over thinking it.

If they switched platforms (switching from Mac to PC) then that would be a problem. But ProRes is virtually guaranteed to stay in the Quicktime "bucket". When is the last time you can remember Apple ever "removing" a codec from Quicktime. I can't remember them ever doing it.

But yes, you're correct. If Apple was to declare bankruptcy, or get bought out by Microsoft (yikes!), then you'd have to transcode all that data. That would be a major pain. (let's hope that never happens)

I can see your point about DV. But you should probably chose DV Stream as it's a cross platform codec. I just can't get past the quality hit going to DV. MII actually looks pretty darn good. If I remember correctly, it was slightly better than Beta SP and they are both 4:2:2 color codecs.

I guess is the footage isn't that important, then the quality hit by going to DV wouldn't be so bad. But if it's not that important, why are you worried about archiving it? hehehehe :)

P.S. If you do the math, Blu-Ray is cheaper than DV tape.

It's not really my problem, I'm just trying to help out a former employer with his dilemma.

I'm not advocating using DV at all, the lowest I've recommend going is DV50 but even then I'd rather have something not in the DV family.

MII was one of those formats that was superior that didn't gain widespread acceptance. It was slightly better than Beta SP but for whatever reason SP won out. Happens.

Maybe I'll just advise him to go with ProRes and hope for the best. There isn't a good answer to this problem is there? I was hoping to post this on message boards and hear something I'd not considered but all this is doing is confirming my suspicions that 10bit uncompressed or ProRes is the way to go. I still feel a little funny about recommending LTO, but that seems to be the consensus on another forum too so maybe that's the answer since I don't trust hard drives even redundant ones.

I'd suggest DVCAM tape. It's relatively high quality (certainly equal to MII), would allow a component machine to machine dub (with TC), would probably be the most efficient both in cost and time and provides some future proofing. MII was a PIG and the reason that to this day I refuse to buy Panasonic. If you can get the MII decks to play (they barely worked on their best days anyway) then you will be lucky.

It's an interesting problem, and one not easily solved. One that Hollywood is facing as it brings it's catalog items forward from VERY old film. They've chosen Blu-Ray.

In my own work, I am bringing forward old material as well. About 100-150 hours of video. I also chose BluRay for a number of reasons.

1. The video/audio does NOT have to be stored in a playable format. This might seem trivial, but if the data has to be moved forward again, I won't need a special "player" to do it. Any computer with a converter can do it. This is one of the primary failings of tape-based formats.

2. Opticals are much less susceptible to atmospheric conditions. Heat, humidity, particles in the air, etc., can all affect tape and hard drives.

3. Magnetic sources like monitors do not affect optical media. Neither does X-Ray.

4. Water, even amounts that might flood a room, do not destroy opticals. Not so for tape, hard drive, or even flash.

5. With BluRay data storage, you can put the codec, the player, and a converter on EVERY disk. This greatly simplfies the method to move forward off the format. Essentially, you are making a self-contained transfer method right on every disk.


In terms of a codec, there are any number of good ones to use. Uncompressed is certainly an option, but probably totally unnecessary. Jpeg2000 is probably a reasonable option. You certainly want to pick something that is an SMPTE standard to ensure compatibility down the road. I tend to use Avid's DNxHD for my HD archiving, and will probably be using mpeg4, VC-1, or Jpeg2000 for SD. All compresses nicely and should allow you to put several hours of material on a single layer BluRay disk.

Just my thoughts on the matter.

I'd suggest DVCAM tape. It's relatively high quality (certainly equal to MII), would allow a component machine to machine dub (with TC), would probably be the most efficient both in cost and time and provides some future proofing. MII was a PIG and the reason that to this day I refuse to buy Panasonic. If you can get the MII decks to play (they barely worked on their best days anyway) then you will be lucky.

DVCAM and DV are virtually identical quality. They both use the same codec. The only difference is that the DVCAM tape drive runs much faster than DV so there's less of a chance of drop out. MII is a much better quality codec than DV/DVCAM. MII is basically 4:2:2 8-bit uncompressed.

DVCAM and DV are virtually identical quality. They both use the same codec. The only difference is that the DVCAM tape drive runs much faster than DV so there's less of a chance of drop out. MII is a much better quality codec than DV/DVCAM. MII is basically 4:2:2 8-bit uncompressed.

DV and DVCam ARE identical quality. No virtually about it, at least in terms of the video signal written to tape.

Agreed that MII is better than DV/DVCam but to call an analog format a "codec" would be incorrect. Codec is an acronym for COmpressor DECompressor, which indicates digital.

And yes, the chroma is recorded at half the frequency of the luminance channel.

I'd suggest DVCAM tape. It's relatively high quality (certainly equal to MII)

The issue with DV/DVCam is the colour sampling is at one quarter the luminance. MII is at one half. You're throwing away a lot of colour information as well as visual detail.

ADDENDUM: Of course, this all ASSUMES that the playback of the original material is high quality with little or no analog tape related issues and that the original program material is high in INITIAL quality...

MII is very close to Betacam, so what I did might be of use. When I was using DVCAM with DSR1800 decks which had component in and out, I transferred a bunch of Betacam SP tapes, via component, to DVCAM and comparing the original Betacam with the transferred DVCAM, nobody could tell the difference. That got my old tapes from analog into the digital realm and then I could do whatever I wanted with them. Obviously, DVCAM is not going to be around forever either, but it satisfied the requirement for the next few years. Much of that footage has been captured and re-edited into updates of old programs.

The reason I transferred to DVCAM was because we had DVCAM decks with component in. What I think I'd do today, however, is invest in a quality converter that takes component in, thus skipping the tape-to-tape transfer phase, and capture all the footage and save it as some QT format, ProRes, H.264, whatever you prefer. I'd store the files onto those gold coated expensive DVDs, which you can get at B&H. And I'd also keep the footage on hard drives. However, in my experience hard drives can simply die for no apparent reason when sitting on a shelf for a long time. So I'd consider the DVDs to be my "original."

In the digital world these days long term archiving is not a piece of equipment or a format--it's a process. Before DVD becomes extinct, the files probably would have to be transferred to some solid state device. Already, USB thumb drives are available in 32 gig sizes for around 40 bucks. Eventually it'll be cheaper to use them than DVDs, probably. However, no doubt they'll be extinct one day too. The main issue, I think, for your client is to get his footage into a digital form, regardless of what it is. It may be that 3 hour DVCAM tapes transferred from component to component would be the cheapest, easiest thing to do for right now. I stress the component--you do it with S-video or composite cable, there will be a degradation in quality.

My old Betacam SP tapes, which had been properly stored, were still in good shape after more than a dozen years, and the dropout compensation in the deck took care of most dropouts, which with old 3/4, Betacam and MII tapes is an issue. In my case many of the tapes had been handled a lot during early editing, on a variety of decks, so I expected more dropouts than I actually saw. Still, the sooner you get digital, the better off you are. Should your guy decide to go the tape route, make sure he only uses the best quality tape. In nearly 10 years of shooting DVCAM I used only Sony PDVM tapes and never saw a single dropout.

Another pretty good codec you could consider will be CINEFORM's codec. It got 10bits, 422 colour space, and uses wavelet compression. And is available for both PC and MAC platform.

As for physical media - hard-disk is out of question. Too many parts in there. The best I would think (that is available NOW) - BluRAY. It got the densest recording media so far for optical. Then in a couple of years time - when it becomes common place - halographic memories.

MII is a much better quality codec than DV/DVCAM.

Have you actually ever played back or watched MII? My DVCAM footage runs circles around anything I ever shot on Ikegami 53/57's with MII backs. After 10 record play cycles we had to replace the MII tapes because dropouts had become unbearable. MII had great specs that didn't translate into real world picture quality and that was if the audio worked. I've got 20 year old BetaSP tape that still plays and looks better than any MII ever did.

Have you actually ever played back or watched MII? My DVCAM footage runs circles around anything I ever shot on Ikegami 53/57's with MII backs. After 10 record play cycles we had to replace the MII tapes because dropouts had become unbearable. MII had great specs that didn't translate into real world picture quality and that was if the audio worked. I've got 20 year old BetaSP tape that still plays and looks better than any MII ever did.

It's all subjective. My former employer, the same guy who I'm inquiring about archival methods for, swore by MII and to this day will tell you that he prefers the look of that format to DVCAM. Every time we'd bust out the old stuff on MII he'd talk about the analogue look compared to the digital look and loved to point out how to his eye the component analogue signal was "smoother" and less "harsh". My guess is that he was seeing the difference between 4:2:2 of the MII and the 4:1:1 of the DVCAM.

It's a matter of opinion.

What I will say is having been around MII, Beta SP, and DVCAM that well shot MII held it's own against any of them. Your poor experience with the format is unfortunate.

The point of this thread isn't to denigrate this or that format, but to have a meeting of the minds and see if we can come up with some ideas for archiving old material. Let's leave it at that.

Back on topic:

Here's what I think will be the solution for archiving in the future:

DMN interstitial (http://www.broadcastnewsroom.com/articles/viewarticle.jsp?id=676133)

Once the prices come down (like they always do), we'll end up with cost effective, large storage capacity hard drives, without any of the hardware failures. I can't wait! :)

I wouldn't choose DV/DVCAM for the 4:1:1 reason.

But if I had to use it, I'd avoid any form of container (i.e., AVI, MOV etc) - I'd store it as a raw DV stream.

Likewise, for long term archiving, I'd avoid any proprietary codec no matter how wonderful they might be. Why? Because there is no guarantee that the codec will be available. Remember Indeo?

I'd also consider redundant formats for storage - e.g., tape and BR and hard drives etc. Personally, I place more faith in tape simply because the other options haven't stood the test of time yet - just look at the nightmares with recordable DVDs failing after just a couple of years. I have 20 year old Video8 tapes that play just as well as the day they were recorded on the original camcorder.

Re DV vs DVCAM - there is a difference in that DVCAM always records with locked audio. No big whoop, though. BUT - only a very few brands (of models thereof) can play DVCAM. Most consumer Sony DV camcorders can actually play DVCAM.

How about MPEG2 4:2:2 ?

MPEG2 is a nice, generic, high compression codec that people are going to understand for a long, long time to come.

Re DV vs DVCAM - there is a difference in that DVCAM always records with locked audio.

If this is in response to my quote above, you'll notice that I mention they are "identical quality" (exact same compression and bitrate, just different tap pitch) and further qualify myself by mentioning "in terms of the VIDEO written to tape".

Ah yes, Indeo... Nostalgia creeping in...

How about XDCAM Professional Disc system? Would this be a great way to archive? I'm already shooting XDCAM HD and are going to invest shortly in a PDW-U1 optical disc drive.

I also have lots of S-VHS Pro, Betacam SP and DVCAM tapes (actually who doesn't?) that I must archive for future use. The DVCAM tapes will be fine for the first years I guess. To me this looks way better, then storing it all back again on tape because it's tapeless. Much easier to access when the archiving is done. Or do I see this wrong? Any thoughts?

Really,

I think that in any archiving solution we need to look at two things.

The medium (Tape, hard drive, solid state, film), and the codec or means to read that file later.

Requiring that the video be stored in a playable format locks us to both. Allowing the file to be written as data opens up more flexibility.

For instance, if we say we were going to archive in "DVD" then we'd be locked to mpeg2 AND we'd be locked to optical. Saying we were going to store on optical would allow a great many codecs, or saying we were going to store as mpeg2 would let us write to any media.

I honestly believe that if the environment can be controlled (flood protection, heat protection, fire protection) then tape makes good sense. But in non-controlled scenarios, then optical makes more sense.

I would not lock myself to a single-vendor solution like Sony Professional Disk though. There are far too many standardized codecs and such to jump into bed with a single vendor solution. Generally, looking at what the library of congress is doing is a good start. They spend quite a lot of time thinking about these things. As to other preservation companies. Film is terrific because it's very stable, and all you need to do is to figure out how to shine a light through it to see the images.

I still contend that going to optical with a SMPTE standards based codec is going to hold up best over time.

FYI, QuantuM Corp will be having a webcast on archiving/long term data storage on 3/18

Here's the blurb that showed up in my e-mail

May be of interest so I'm passing this along

QUOTE____________________________

Join Quantum for our webcast entitled “Designing a Long-term Data Retention Strategy” and we’ll discuss shifting industry trends requiring more data to be retained for longer periods of time for both compliance and non-compliance requirements. A combination of tape and disk-based data protection, including archiving for data preservation, can help address power, cooling, floor-space and economics concerns while meeting other needs and issues.

In this webcast, hear about the issues and trends as well as how magnetic tape continues to evolve to support changing data protection and long-term data protection requirements while complementing and co-existing with disk-based data protection
technologies.

END QUOTE_____________________________________________

We have BetaSp tapes that are almost 20 years old that still work - but we did have to move all out old 2 inch & 1 inch tape over to DigiBeta... But I know that sooner rather than later we'll have to do the same with the SP... but to what?

Well NO WAY to a drive, I'd much rather go to M2 to DigiBeta - digi will be around for AGES....

I seem to remember a program that recored data to MiniDV tapes (only 13GB per) - maybe that tech has progressed.

BluRay - seems like a good choice.

We even talked about paying to SAVE an old 1 inch machine!!!! Doubt your friend will want to do that...

It does seems like in these modern digital days - archiving is a lost art.... If I had to pull that trigger right now - I would go for DigiBeta - but thats what we use here and have all the decks and so on....

Good luck.. let us know what the final choice is.

Actually he still has a pristine 1" machine sitting in a back room that he wont get rid of because he's convinced that he'll make $$ with it again one day when people get desperate to transfer old 1" stuff to some other format.

The thing looked brand spanking new.

Just because it looks - or IS - new when you put it on the shelf doesn't mean it will work when you need it. Time does all kinds of bad things to machines, just like it does to us. Plastics get brittle, rubber crumbles, lubricant dries out and turns to adhesive.

Sorry for the gloom and doom, but it's true!

I think it's also true that optical media is risky for long term storage and the higher the density and recording bit rate, the weaker the write signal becomes, implying the need for easier phase change in the medium, implying less resistance to reversion to the unrecorded phase state with time and temperature.

Of all rewriteable digital media known to man, tape probably has the best actual track record as a long term storage medium. Engraved stone tablets might beat it, but I don't recall seeing any stone tablet writers at Amazon lately.

And of course different kinds of tape have different long term archival characteristics. In today's world, LTO is about as good as it gets and there are millions of drives out there, so you should be able to find something to read back your tapes for the next 10 or 20 years.

And by the way, a key question when talking about "long term" should always be "how long".