p2 viewer copy for cleaning up corruptions... worked in one set of cases at DVinfo.net

Ali Husain · August 16th, 2006, 07:18 AM

just posting this in case it helps anyone else:

i've got a lot of p2 card images made via p2 store, and almost every last MXF file in the VIDEO directory produces a fatal error immediately upon my conversion attempt in BOTH raylight and cineform connect hd. the message is the same: that the format is unknown and that i should check my p2 viewer. now p2 viewer on the same machine opens these p2 card images fine, so...

i copied the "corrupted" p2 card image to a new p2 card image within p2 viewer. and the resulting copied files convert fine in cineform (haven't tried raylight yet).

so to summarize: if you have file "corruption" problems, then you might solve them by copying the p2 directory to a new one with p2 viewer. evidently, the panasonic software is a bit more robust when dealing with funky p2 metadata.

-----------

it seems there are highly inconvenient single points of failure in panasonic's p2 card scheme that make it non-ideal for uncertain field environments and for capture and ingest. if we were dealing with raw dvcproHD or raw DV25 or even raw audio, there is a fairly fine granularity at which corrupted data is resynced. in the simplest example, if you've got raw 16-bit stereo pcm you can lose a byte and at worst you've killed one sample.

if you accept that you're going to be losing bits and piece of your p2 image every now and then, which is an assumption i'd think a reasonable engineer would make, then if you get corruptions in your XML files or in any of the MXF headers, which you should assume you will, you have a problem.

i can see how the XML'icized P2 scheme is nice and elegant for very reliable and safe environments, like your desktop. but this seems like not a very good choice as the center of an abuse-prone yet critical acquisition scheme. if i get just the wrong byte corruption then my entire p2 card can't be read?

now i know there are redundencies in this scheme and you can still pluck data out with the right programs. it looks like panasonic's p2 viewer is the only tool that i have which deals with inconsistencies in the p2 card metadata in a non-tragic way.

if i were writing software to operate on p2 cards, the first thing i would do would be to write a front-end that would check the p2 card metadata against all the headers and lengths for consistancy and build-in a smart repair mechanism, a useful diagnostic output, and i would QA the snot out of it so that it never failed to return some output.

right now i don't think this is happening. i'm seeing common cases failing in alarming ways. for example, there is some p2 card manipulation software (i won't name it) and it dies if you insert an empty p2 card! how hard is that to test? here you go:

if i'm a p2 user i'm going to:
(1) stick in an empty card
(2) stick in a full card
(3) stick in a card with the absolute minimum amount of data
(4) minimum + 1
(5) soemthign in between
(6) max - 1
(7) etc.

if i'm a p2 user i'm going to pull out the card while it's being written (which i saw happen more than once in a single shoot). i'm going to forget to erase it. i'm going to use it to transfer wordprocessor files out in the field somewhere, and then i'm going to forget to erase them and then i'm going to put it back in the camera and try to start shooting.

panasonic and p2 software people: these are all BASIC cases that should be addressed in a graceful way.

dealing with annyoing p2 and mxf issues almost drove me to download the MXF SDK and start to cobble somethign together, until i discovered enough kludgey workarounds for the problems i'm having. anyway i'll stop ranting now. :)

Ali Husain · August 16th, 2006, 07:20 AM

just realized that i shouldn't have posted this in the cineform forum because it is not about their software. sorry about that. how do i move it somehwere more relevant?

Pete Bauer · August 16th, 2006, 08:54 AM

Done. Moved from Cineform to Panasonic HVX200 forum.

Jan Crittenden Livingston · August 16th, 2006, 09:48 AM

Quote:

Originally Posted by Ali Husain

just posting this in case it helps anyone else:
i've got a lot of p2 card images made via p2 store, and almost every last MXF file in the VIDEO directory produces a fatal error immediately upon my conversion attempt in BOTH raylight and cineform connect hd. the message is the same: that the format is unknown and that i should check my p2 viewer. now p2 viewer on the same machine opens these p2 card images fine, so...

So in your conversion, were you making the conversion directly from the P2 Store or were you making the conversion after you had transfered the data to another HDD?

I have noted the majority of experiences that sound like yours are making the conversions from the P2 device itself and not the copy of the P2. The fact that the P2 Viewer is able to read the file in question point so the robustness of the P2.

Try making the conversion from a copy of the data and not from the P2 Store, it may work better.

Best,

Jan

Ali Husain · August 19th, 2006, 02:34 AM

hi jan,

thanks for the reply. i was making the conversion from a harddrive copy of the p2 store. at some point, after i'm done with this project i'll put up an example somewhere, so that it might help developers to iron out these issues.

-------------
general comments for the forum (not directed at jan):

i'm guessing this corruption is because of operator error (there were people with varying skill level helpign with this shoot). and i bet 90+% of all the problems with the p2 scheme originate in operator error. but this is going ot happen and software developers would improve the happiness level of some, more vocal :) p2 users if they'd put a little more intelligence into the software.

for example, the p2 viewer will correctly playback a p2 image that bombs some other 3rd party software, but it reports that the file--from an 8g p2 card--is 1 hour and 2 mins long. somethign is corrupt in the file and the p2 viewer playback engine gets around it. if you copy that file to a virtual p2 card you'll get 24gig of data (from the original 8) because the filesystem routines are looking at evidently the part that is corrupt.

there is a length associated with the audio (since i believe it's continuous bit rate), and there is a length associated with the video (since it is also CBR). it's a straightforward enough check to verify the TIMECODE_END - TIMECODE_START length against the audio and video file sizes, also against the XML'd note of the originating p2 card's size. some of these might be corrupted, but through some intelligent checking you can arrive at what is *probably* the real file size and make people a lot happier. anyway just one example...

what would be great is if someone :) released some open source and ultra-robust C++ library routines for developers to use for p2 card and hvx originated mxf file access.

August 16th, 2006, 07:18 AM	#1
Ali Husain Regular Crew Join Date: Oct 2005 Location: San Jose, CA Posts: 141	p2 viewer copy for cleaning up corruptions... worked in one set of cases just posting this in case it helps anyone else: i've got a lot of p2 card images made via p2 store, and almost every last MXF file in the VIDEO directory produces a fatal error immediately upon my conversion attempt in BOTH raylight and cineform connect hd. the message is the same: that the format is unknown and that i should check my p2 viewer. now p2 viewer on the same machine opens these p2 card images fine, so... i copied the "corrupted" p2 card image to a new p2 card image within p2 viewer. and the resulting copied files convert fine in cineform (haven't tried raylight yet). so to summarize: if you have file "corruption" problems, then you might solve them by copying the p2 directory to a new one with p2 viewer. evidently, the panasonic software is a bit more robust when dealing with funky p2 metadata. ----------- it seems there are highly inconvenient single points of failure in panasonic's p2 card scheme that make it non-ideal for uncertain field environments and for capture and ingest. if we were dealing with raw dvcproHD or raw DV25 or even raw audio, there is a fairly fine granularity at which corrupted data is resynced. in the simplest example, if you've got raw 16-bit stereo pcm you can lose a byte and at worst you've killed one sample. if you accept that you're going to be losing bits and piece of your p2 image every now and then, which is an assumption i'd think a reasonable engineer would make, then if you get corruptions in your XML files or in any of the MXF headers, which you should assume you will, you have a problem. i can see how the XML'icized P2 scheme is nice and elegant for very reliable and safe environments, like your desktop. but this seems like not a very good choice as the center of an abuse-prone yet critical acquisition scheme. if i get just the wrong byte corruption then my entire p2 card can't be read? now i know there are redundencies in this scheme and you can still pluck data out with the right programs. it looks like panasonic's p2 viewer is the only tool that i have which deals with inconsistencies in the p2 card metadata in a non-tragic way. if i were writing software to operate on p2 cards, the first thing i would do would be to write a front-end that would check the p2 card metadata against all the headers and lengths for consistancy and build-in a smart repair mechanism, a useful diagnostic output, and i would QA the snot out of it so that it never failed to return some output. right now i don't think this is happening. i'm seeing common cases failing in alarming ways. for example, there is some p2 card manipulation software (i won't name it) and it dies if you insert an empty p2 card! how hard is that to test? here you go: if i'm a p2 user i'm going to: (1) stick in an empty card (2) stick in a full card (3) stick in a card with the absolute minimum amount of data (4) minimum + 1 (5) soemthign in between (6) max - 1 (7) etc. if i'm a p2 user i'm going to pull out the card while it's being written (which i saw happen more than once in a single shoot). i'm going to forget to erase it. i'm going to use it to transfer wordprocessor files out in the field somewhere, and then i'm going to forget to erase them and then i'm going to put it back in the camera and try to start shooting. panasonic and p2 software people: these are all BASIC cases that should be addressed in a graceful way. dealing with annyoing p2 and mxf issues almost drove me to download the MXF SDK and start to cobble somethign together, until i discovered enough kludgey workarounds for the problems i'm having. anyway i'll stop ranting now. :)

August 16th, 2006, 08:54 AM	#3
Pete Bauer Wrangler Join Date: May 2003 Location: Eagle River, AK Posts: 4,100	Done. Moved from Cineform to Panasonic HVX200 forum. __________________ Pete Bauer The most beautiful thing we can experience is the mysterious. It is the source of all true art and science. Albert Einstein Trying to solve a DV mystery? You may find the answer behind the SEARCH function ... or be able to join a discussion already in progress!

August 16th, 2006, 07:20 AM	#2
Ali Husain Regular Crew Join Date: Oct 2005 Location: San Jose, CA Posts: 141	just realized that i shouldn't have posted this in the cineform forum because it is not about their software. sorry about that. how do i move it somehwere more relevant?

August 19th, 2006, 02:34 AM	#5
Ali Husain Regular Crew Join Date: Oct 2005 Location: San Jose, CA Posts: 141	hi jan, thanks for the reply. i was making the conversion from a harddrive copy of the p2 store. at some point, after i'm done with this project i'll put up an example somewhere, so that it might help developers to iron out these issues. ------------- general comments for the forum (not directed at jan): i'm guessing this corruption is because of operator error (there were people with varying skill level helpign with this shoot). and i bet 90+% of all the problems with the p2 scheme originate in operator error. but this is going ot happen and software developers would improve the happiness level of some, more vocal :) p2 users if they'd put a little more intelligence into the software. for example, the p2 viewer will correctly playback a p2 image that bombs some other 3rd party software, but it reports that the file--from an 8g p2 card--is 1 hour and 2 mins long. somethign is corrupt in the file and the p2 viewer playback engine gets around it. if you copy that file to a virtual p2 card you'll get 24gig of data (from the original 8) because the filesystem routines are looking at evidently the part that is corrupt. there is a length associated with the audio (since i believe it's continuous bit rate), and there is a length associated with the video (since it is also CBR). it's a straightforward enough check to verify the TIMECODE_END - TIMECODE_START length against the audio and video file sizes, also against the XML'd note of the originating p2 card's size. some of these might be corrupted, but through some intelligent checking you can arrive at what is probably the real file size and make people a lot happier. anyway just one example... what would be great is if someone :) released some open source and ultra-robust C++ library routines for developers to use for p2 card and hvx originated mxf file access.