High Definition with Elphel model 333 camera

[Andrey Filippov]

The biggest noise visible on the CMOS (compared to CCD) is FPN (fixed pattern noise) caused by differences in the pixels. Actually the camera FPGA code in the 333 (and even in older 313) has the code to subtract array from the image and multiply - reading that correction data from the same video memory. This processing can be done in parallel with the compression so no slowing down.
But I never got the time to write some code to calculate that arrays without individual measurements for each particular settings (and sensor temperature) like taking an image, closing the cap on the lens and getting a dark one to subtract. It is possible to do the compensation without that - just while using the camera.

[Zsolt Hegyi]

Originally Posted by Andrey Filippov

taking an image, closing the cap on the lens and getting a dark one to subtract.

Sounds good; I mean there's room for improvement. I think I'm going to do that when the camera is ready. Seems quite simple, especially if integrated into the post-process instead of fpga.
If we use strictly interframe-difference compression then this noise won't decrease the encoding rates (because it's fpn). And if we use lossless, then we can reproduce these "black frames" without errors.

[Wayne Morellini]

Originally Posted by Zsolt Hegyi

Yes they're not bad but consider that these are interpolated rgb images, not bayer. Interpolation introduces noise and jpeg compression introduces more. This noise is algorithmical and can be removed later by software if correctly recognized.

There are far better ways then interpolation to remove noise, I understand they use a number of different methods including temporal, where they compare across frames.

But you have to remember that this is only for post processing MJPEG and Theora at the moment, not bayer, though I expect you can do bayer through one of the camera noise removers listed in that review link.

You're probably aware that the cmos chips we intend to use contain analog noise removal circuits and they're really good (removing the noise digitally is not nearly as efficient).

Yes, but the purpose is to clean up after shooting, or if I can find something suitably simple for FPGA, it is suitable for increasing compression. According to information I read, intra frame compression performance, and severely inter frame lossless compression, is limited because of noise. Extra non consistent complexity in the image limiting intra compression, and changes in pixel values between frames reducing inter performance.

By using correctly exposed sensors with normal analog gain levels we should have no significant noise on the raw bayer images. (Eg. use lighting if there's too low light around and always use manual exposure control.)

In a film set, yes, but using in the field, under various conditions, there are extremes of exposure that has to be used, as with most cameras producing noise. The SN is also not very high on these chips compared to Ibis (with proper external setup) Altasens, or ENG cameras, but probably close to a number of HDV cameras.

I personally want to increase aperture through a condenser 35mm adaptor, but these chips have a limit form their microlens that restricts maximum aperture (another reason why the IBIS5a was so good it had no microlens and you could use very fast apertures of less than 1 for more stops of light, and great latitude extension). I would like to minimise lighting (an personal experiment).


Thanks

Wayne.

[Zsolt Hegyi]

There are far better ways then interpolation to remove noise

I meant the source images not the results.

But you have to remember that this is only for post processing MJPEG and Theora at the moment, not bayer

Okay, I've got it now, sorry.

another reason why the IBIS5a was so good it had no microlens and you could use very fast apertures

And what do you think the maximum aperture is for the microns?

Switching topics: today I finished the development of the encoder. The source material for the tests were the following files:

http://3lib.ukonline.co.uk/cricket/imran1.mpg
http://www.elphel.com/3fhlo/samples/snakeriver.mov

I converted both of them to grayscale mpeg-s before processing. The java/jmf framework provided the decoded buffers with a bit depth of only 5 so I upconverted it to 12 bits (shifting 7 bits to the left, zeros coming in on the right) and replaced the lower 3 bits with random noise. I write this because it's not the ideal input - we could have better or worse when trying it with a real 12 bit stream.

With both of the above files I've got the targeted 3:1 ratio with lossless interframe compression. So my prognosis was good: the much lower frequency content of the time domain really helped the same encoder routine to provide better results than using it in intraframe mode.

I'd like to add that in java, the encoder routine is only 10 lines (!) long so I should have no problems implementing it in verilog... And it will be fast because it works with a 1 pixel per 1 clock cycle way. And because of its small size I could even instantiate an encoder for every pixel in a 5x5 block.

I have some more ideas for optimization but I don't want to implement them until I have a camera and the test results demand better compression performance than the current one.

Now I quit posting to this forum for a while; you'll probably hear from me again when I have a working camera in my hands. Hope this happens very soon.

Thanks,
Zsolt

[Andrey Filippov]

Originally Posted by Zsolt Hegyi

You're probably aware that the cmos chips we intend to use contain analog noise removal circuits and they're really good (removing the noise digitally is not nearly as efficient).Well, unless you push up the analog gain in the chip - that seems to be the case with several images posted on the above link. And the other set of images are just poorly jpeg-compressed.

What do you mean by this magic analog noise removal? And what kind of noise do you mean?

CMOS have on-chip black level compensation (still not as good as "digital" - subtracting individual pixel black levels). The only analog that can be better then digital (and it mostly applies to CCD) is true binning (with reduction of resolution, of course). The S/N gain compared to averaging of the output pixel data (can be done on chip for CMOS, always off-chip for CCD) can be better up to square root of the number of the pixels binned (e.g. 2 for 2x2 binning) The reason is that binning by combining pixel charges does not introduce any noise, so each cluster of pixels binned get noise from the output amplifier only once, and if you average pixel values after the amplifier each pixel will have the output amplifier noise added, so averaging of non-correlated noise will give you square root.

The above is true only for the noise of the output amplifier that can be a major factor with CCDs in some conditions (pixel thermal noise low - short exposure and/or cooling).

With CMOS the most visible is FPN that can be nicely removed by digital subtraction of the "dark" image.

[Zsolt Hegyi]

What do you mean by this magic analog noise removal?

It's not magic, I was talking exactly about the black level which has both analog and digital correction phases (at least in the micron). It is not amplifier noise that it removes but thermal noise generated by the pixels themselves. Sorry for not being so specific.

Anyway, are you talking about this method not being as good as the array substraction because the pixels used as the base of the black level are only surrounding the visible area but they're used for every pixel being read out?

Zsolt

[Andrey Filippov]

Originally Posted by Zsolt Hegyi

It's not magic, I was talking exactly about the black levels which has both analog and digital correction phases (at least in the micron). It is not amplifier noise that it removes but thermal noise generated by the pixels themselves. Sorry for not being so specific.
Zsolt

There is no way to reduce thermal noise but cooling the chip. Subtraction of the dark level as in Micron chips is OK as a first approximation, but measuring/calculating (and subtracting) per-pixel data produces much better results.

[Wayne Morellini]

Originally Posted by Zsolt Hegyi

And what do you think the maximum aperture is for the microns?

Andrey would know, I forget, but probably at around 1.4 it will start to wash out (or was that 3 chip, one was closer to 1.6 I think).

Switching topics: today I finished the development of the encoder. The source material for the tests were the following files:

http://3lib.ukonline.co.uk/cricket/imran1.mpg
http://www.elphel.com/3fhlo/samples/snakeriver.mov

I converted both of them to grayscale mpeg-s before processing. The java/jmf framework provided the decoded buffers with a bit depth of only 5 so I upconverted it to 12 bits (shifting 7 bits to the left, zeros coming in on the right) and replaced the lower 3 bits with random noise. I write this because it's not the ideal input - we could have better or worse when trying it with a real 12 bit stream.

With both of the above files I've got the targeted 3:1 ratio with lossless interframe compression. So my prognosis was good: the much lower frequency content of the time domain really helped the same encoder routine to provide better results than using it in intraframe mode.

I'd like to add that in java, the encoder routine is only 10 lines (!) long so I should have no problems implementing it in verilog... And it will be fast because it works with a 1 pixel per 1 clock cycle way. And because of its small size I could even instantiate an encoder for every pixel in a 5x5 block.

I am impressed by the ten lines of code, but exactly what sort of system are you doing, can you describe it? With only ten lines you could almost just use an array of small processing cells (look for Misc processor cores on the web less than 16K gates each) to achieve it without custom design.

Have you got any raw bayer files to test? There might be some RAW bayer footage on the Silicon imaging camera threads, or the Drake camera thread, or Obin Oslo's original camera thread. Compressed 4:2:0 files already have much of their information removed, and the most hard to compress data is in the last 2 bits.

[Zsolt Hegyi]

I am impressed by the ten lines of code, but exactly what sort of system are you doing, can you describe it?

I will, as soon as the code works in a camera and produces the results I have now. As I wrote earlier it's an integer-difference compression with a bitstream output.

With only ten lines you could almost just use an array of small processing cells (look for Misc processor cores on the web less than 16K gates each) to achieve it without custom design.

For ten lines one doesn't need to do really heavy design work... What I'm more interested in is that how this routine will fit into the Elphel design. If I take one of those cores then I might have a hard time fitting it into the 353.

There might be some RAW bayer footage on the Silicon imaging camera threads, or the Drake camera thread, or Obin Oslo's original camera thread.

I don't remember bumping into any raw videos when I was browsing through those threads. Anyway, in what kind of format they could've posted them, if they did?

Compressed 4:2:0 files already have much of their information removed, and the most hard to compress data is in the last 2 bits.

I know, that's why I inserted 3 bits of random noise into the pixels - which the encoder couldn't really compress effectively so basically I had the lower 3 bits uncompressed... Hence the only 3:1 ratio.

[Wayne Morellini]

Originally Posted by Zsolt Hegyi

I will, as soon as the code works in a camera and produces the results I have now. As I wrote earlier it's an integer-difference compression with a bit stream output.

Cool, no DCT.

For ten lines one doesn't need to do really heavy design work... What I'm more interested in is that how this routine will fit into the Elphel design. If I take one of those cores then I might have a hard time fitting it into the 353.

Doesn't the 353 have something like 1.2M gates, the Misc core is only around 16k (I think). www.ultratechnology.com/chips.htm has a archived list of forth chips that had the FPGA Misc variants (their are other sites on the web with the more info).

I don't remember bumping into any raw videos when I was browsing through those threads. Anyway, in what kind of format they could've posted them, if they did?

They did, I think they padded them in Tiffs and the like (their are still camera raw formats) and maybe custom formats. I know Obin's for certain. You could email him. But when you get your camera you can shoot small windowed raw frames output as monochrome bit-mapped. You could get somebody to do stills here even.

I know, that's why I inserted 3 bits of random noise into the pixels - which the encoder couldn't really compress effectively so basically I had the lower 3 bits uncompressed... Hence the only 3:1 ratio. Without the noise the ratios went up high as 12:1 (nonsense, the lower 7 bits were zeroes) and using the original 5 bit wide data without upscaling, the ratio was around 4:1.

If you are not compressing 4 blank padded bits we have a very promising start. The three bits of random noise should go to offsetting the lack of detail in the heavily compressed video. You could compare against the existing lossless video codecs, that I linked to on the web wiki before, even on their test footage.

Another interesting link I came across, if useful.

http://www.data-compression.info/Algorithms/RC/


Thanks for your efforts Zolt, keep it up. I wonder where everybody else has gone with their cameras?

Wayne.

[Wayne Morellini]

Over at the comp.compression thread, a commercial oriented guy from this company: http://www.algopix.com/ has turned up and is interested in developing noise removal for increasing lossless compression ratios.

He is aiming to do a simple two frame comparison, where he uses the noise removal to also identify inter frame movement, and then compresses lossless.


Thanks

Wayne.

[Wayne Morellini]

I did a search for something and accidentally came across that thread I posted on comp.compression newsgroup for reference:

http://groups.google.com/group/comp....ee766672dc7727

It seems to be turning up in different places:
http://archives.devshed.com/forums/c...r-1937388.html
http://groups.google.co.jp/group/com...lnk=raot&hl=ja


Thanks

Wayne.

[Matteo Pozzi]

look at this!

"Leveraging its heritage in high-speed, high-performance imaging, Micron will unveil a new 5-megapixel high-definition (HD) image sensor for mainstream digital cameras. The new sensor is capable of capturing video at 60 frames per second (fps) in 720p (progressive) format and 30 fps in 1080p format. Micron will begin sampling this sensor in the fall of 2006.

Designed specifically for digital video camcorder applications, Micron will also introduce a new HD video sensor that captures 60 fps in 720p format. Designed to work with long-range zoom lenses, the sensor was built using Micron’s stunning 2.2-micron pixel technology. Additionally, the sensor has additional pixel area for image stabilization, which reduces the effect of shaky and blurred images typically caused by jittery hands or camera-shake. This sensor is expected to begin sampling in the first quarter of 2007."

taken from:
http://www.micron.com/about/news/pre...A76239EFA2B68E

[Rob Scott]

Originally Posted by Matteo Pozzi

is there a way to view the graphic user interface ... so I've some idea for a touch screen user interface for a 7" monitor ...I don't know linux (for now) so what program I need, and to learn, to change the GUI of the elphel recording program? and is there someone that whant to help me?

I'm working on something similar -- ObscuraCam -- which is a "heads-up" recording program specifically for filmmaking; and also a post-processing program to convert the raw files into something useful.

I've just recently discovered this thread, and I find it very intriguing. The Elphel 353 could be an excellent choice for a DIY cinema camera.

[Wayne Morellini]

I just realised it wasn't Rob Loham posting to the thread. Long time no here, still got the same email?

So, still doing the Obsura, how is it going?


Actually, Zolt, what is happening after a month?


Matteo, I think that is the one Andrey aims to use.