4:4:4 12-bit Uncompressed DVX100

[Luis Caffesse]

Juan,

Is it possible that Robert is on to something here?

I'm not saying it is bad CCD chips necessarily...

But, you aren't seeing any of this noise on the DV tape are you?
Is it all being filtered out in some way through the DV compression?

Is is possible that the noise is always there, on all DVX100s?
But, as Robert mentioned, it wouldn't matter because it's getting thrown out anyhow?

Probably not...but I figured it was worth mentioning.
You haven't seen any noise on the DV frames, right?

-Luis

[Jon Yurek]

I don't think that the noise is supposed to be there. Despite DV's compression ratio, it doesn't turn out a horrible picture, so a dark blue pixel will still be a dark blue pixel. And since there aren't that many extra vertical lines of resolution (what's the full res? 771x492 or something?), it can't all be interpolated away. And finally, the compression itself wouldn't cover for it because such a sharp change in contrast would be expected to be preserved by a compression scheme, not eliminated (imagine what it would do to text otherwise).

My guess is that he's got one of the higher bits loose on the CCDs' connections. If the bit is dead or intermittent then when everything lines up just right and that bit dies, the color value will drop to very low. For instance: if the color of a pixel of the sky is 1000000111 (519 decimal) and bit 10 (assuming big-endian) is flaky and happens to not make the connection on that clock cycle, then the color of the pixel would be returned as 0000000111 (7 decimal) which would return a very very dark pixel compared to what was supposed to be there.

And this makes perfect sense, since from what I have seen from the pictures, it's the very middle of the color range that's having the majority of the noise problem (and 512 is right in the middle of the color space) where you drop one bit and the color is changed dramatically.


Although, to be sure, the best thing to do would be to get another camera and try it on there as well to see if it's Juan's CCDs or if it's his method.

[Juan P. Pertierra]

Jon just pointed out what has been my best theory so far...however, i am pretty sure at this point that all the connections are solid and not physically intermittent....so my best guess at this point is that the clock signal which already drives the three A/D's, is degrading because I am also driving the capture card with it...i'm going to try and make a clock-follower that syncs up to the original clock and see if that helps.

Juan

[Robert Martens]

Well, I can see I'm out of my league with the noise theory, but I've got a couple other questions:

1.) Assuming that, once all is said and done, this procedure were possible on other cameras (let's say, the VX2000), how difficult is installation for a layman such as myself? Yes, yes, I know, you could tell me "You've never done anything worthwhile with your STANDARD camera, let alone with one of these fancy mods--focus on your content before you worry about your image", and you'd be right.

But just out of curiosity. Will one need a degree to do this? I mean, I've disassembled my camera several times, and have become familiar with the location of boards, screws, and the like, along with the required order of dis/reassembly, but I couldn't tell ya what chips perform which functions.

2.) What's the legal status of this project? Don't get me wrong, I see no MORAL problem here. I think this is a great idea, and am anxious to see some final results, but I worry about the ramifications of something like this should the companies catch on.

Sorta like the Linux on XBox project. I use this example 'cause I'm a videogame nerd, and I think it's analogous enough to work (keep in mind, however, that I can't remember if this scenario actually happened, or was mere speculation).

The idea was that the XBox, being essentially a mini computer, would make a perfect homegrown Linux platform, especially with its low price point. The fact that it's made by Microsoft was even more incentive for interested parties to get this done. Well, apparently someone figured it out, and in the process of telling others how to do this mod, got a cease-and-desist from MS. The explanation, as I remember it, was that while it was perfectly all right to modify the unit that you paid for, and do whatever you like to the thing, it was against the law to distribute this information to others, in spite of the fact that anyone wanting to do this still had to purchase their product to perform the modification.

Whether this was a rumor, I cannot say for sure, but I think it's worth mentioning, as you seem to be treading similar waters here.

Will this be an issue for your product?

[Isaac Brody]

Don't you basically just void your warranty when you open the unit up?

I'm not an expert on the legal ramifications, but this process doesn't seem that different than the BBC modification to the sound units on VX2000 and PD150's. The BBC hasn't been sued for that either. Perhaps because they're a corporation and not an individual. I don't know.

I say worry about it later. Microsoft was probably more concerned that the Xbox was being outfitted with Linux instead of Windows XP. Sort of like setting up the mayor's daughter with a convict.

[Jon Yurek]

The most they could conceivable get you on is a DMCA violation to circumvent copy protection (which is what MS used against the XBox Linux project). And since there's no copy protection that you're trying to circumvent, you're free to modify the equipment you legally own.

As for the actual mod... I imagine (although I can't say for certain since it's not even done let alone set in stone) it would be a piggyback chip, which either clips on or solders on. A clip on would be dead simple, you'd just have to know which chip to clip. A soldering would be harder but not impossible.

Juan,

Are you using the camera's clock to drive your chip and the capture card, or are you supplying your own to sync up and drive it? If you have your own, it could be that your clock is *slightly* out of phase and is ending up being systematically off on the important bits that result in the noise, but on in the surrounding bits (which wouldn't be subject to that nonsync).

[Juan P. Pertierra]

I am using the clock already on the camera, so i guess either the clock is getting degraded somewhere, or the actual data is. However, since switching the capture card sync from rising to falling edge increases the speckles, i think it's the clock.

[Robert Martens]

Ah, I see, good points. Guess it's not such a big deal after all.

<<<-- Originally posted by Isaac Brody : Don't you basically just void your warranty when you open the unit up?>>>

If you were referring to my fiddling with my camcorder, well, the warranty had expired before I did this disassembly, so there was no worry. Still risky, to be sure, but I have more curiosity than common sense.

Don't worry, though, I wouldn't do this to hardware I hadn't purchased myself; I won't be taking apart anyone else's cameras just to see what's inside. :P

[Steve Ipp]

Juan, much respect to what you're doing. Keep it up!
I'm sorry if this comment gets you away from the actual problem, but could it be that the "problem" isn't really present?
The thing is, ccds transfer charges to registers far from the actual sampling locations (pixels). Unlike CCDs, CMOS censors are more suitable for getting values/sampling at each pixels but raw values from these chips still require cunning algorithms to improve the quality of the final image.
My wild guess (I am not a professional) is that Pana encoding chip has additional functionality as "retouching" facility.
Forgive me if I'm off point, just trying to help.

[Juan P. Pertierra]

Steve,

Thanks for your input.

I've considered that possibility, and so far the biggest supporting evidence for it is that the speckels are very easy to get rid of with a smart median algorithm...something like using the magic wand in photoshop and then performing median on those specific pixels using the surrounding values(like a mosaic CCD).

However, i think so far there is far much more evidence to support otherwise, like the fact that the noise increases if i change the sync edge of the clock but stays in the same general areas, and the fact that i've seen the noise slightly reduced by using different grounding techhniques.

I'm keeping an open mind however.
-------
Also, to everyone else, i have a test this friday along with a robot vision project so this is why i've been disconnected, once i get done with that next week i'll back to work on this...

Just to put something else out there, i have uploaded the Photoshop CS file for the last low-speckles <g> frame i got. I think it's the best frame so far, because it was very dark to the naked eye yet it captured incredible detail in the sky and the dark areas.

This file is completely raw, i just put the layers in and i think i might've shifted them to where i thought they where aligned. If you guys want to post any of your color corrections that'd be awesome...even though this frame doesn't nearly use all of the dynamic range available.

http://expert.cc.purdue.edu/~pertierr/cap5_RAW.psd

Oh and steve is definitely right in that ~some~ processing is done to the raw frame...i'm not sure if this frame is slightly off focus, but it looks much better if you perform a Sharpen operation in PS...the color came out surprisingly good for a RAW frame.

[Obin Olson]

OMG Juan...I can do stuff with that image in PS that I have only DREAMED of!!!!! PLEASE PLEASE stay on this project...I have CASH in hand waiting for you to finish it!!!



guys you gota download that image and play with it....it's AMAZING how far you can push it!!!!

[Obin Olson]

Juan!!! got some news for ya! I just went outside and shot a DV test with my dvx100 VERY dark like your test.....guess what? I pushed it really hard in premiere pro color curves and it has the SAME dots yours does!!!

everyone do this and see if some cameras have dots more then others do....I think it's bad ccd's that ONLY show up when you shoot really dark and push it really hard...take a look:

http://www.dv3productions.com/uploads/dv_cap.jpg

[Obin Olson]

BTW, every frame has dots in a new location in the frame....very random sometimes more dots sometimes less

[Obin Olson]

more info...check this out Juan ..BOTH are INVERTED and I took the RGB curve all the way DOWN..ahh BOTH have RED GREEN and BLUE bad pixels..and they look like the same type of pixel....the DV version is just muddy and jaun's is VERY CRISP because he has NO compression....maybe????

http://www.dv3productions.com/upload...nRAW_16bit.jpg

http://www.dv3productions.com/upload...ed_dv_8bit.jpg

I hope this will help you find the answer...can you write some code that will "search and destroy" bad pixels? better yet try and expose your camera with a good light and see if the pixels go away....

[Obin Olson]

here is what we get if Juan can make this MOD for our cameras work:

http://www.dv3productions.com/upload..._colorwork.jpg
http://www.dv3productions.com/upload..._colorwork.jpg

I did the same color work to both pics

here is my original image:

http://www.dv3productions.com/upload...p_original.jpg