4:4:4 10bit single CMOS HD project

[Eliot Mack]

<<<-- Originally posted by David Newman : >- Originally posted by Eliot Mack :

> It looks like the easiest way to capture is to RAW files. From there, the files can be color corrected and converted to a file format suitable for editing and compositing.

I see no reason not to capture directly into CineForm encoded file. We are doing this today from HD-SDI feeds.
-->>>

My understanding was that capture to RAW format allowed the Bayer filtering to be performed offline, which enables the use of better, more computationally expensive algorithms in performing the white balance/sharpening/etc. conversions. I especially liked the automated color balance from the image of a Macbeth target that Steve N. mentioned; this would negate the need to preset your camera for indoor/outdoor lighting to get the color right. Just expose for proper lighting levels and shoot a chart. This seems so much simpler than the usual methods that I'm surprised it's not more widely used.

The 10 bit 4:2:2 Aspect codec sounds like just the fit for what I need, which is a 1280x720 or 1920x1080 HD codec that I can use all the way through the compositing, editing, and final color correction stages without terabytes of storage and giant RAID requirements. If there is a codec plugin for Premiere Pro/After Effects/Shake then it would be a slam dunk.

How long would it take to get something like this to market?

Thanks,

Eliot

[Steve Nordhauser]

David,
In an ideal world you are correct, recording in the final format would be great. I don't know the amount of real-time processing that can be done on a 3GHz machine. That is clearly your area. This should be possible for 720p, 8 bit. I don't know about 10 bit. Would it scale to 1080p? In 12 bit mode @60fps, we are talking about a 300MB/sec (unpacked) data stream.

The word I hear in this group is that they don't want to compromise on what I've been calling pre-post processing - the Bayer filter, RGB->YUV, compression steps. If this can really be done in real-time, all the way up the camera scale, you have an impressive product (or will when the cheap camera link cameras are supported) and I would suggest that it is not worth anyone here writing their own code. $1K to record fully processed and compressed data with a preview window, maybe with basic camera controls would be great for the people here.

I think the biggest processing and recording issues are a complete understanding of any step that is a potential loss of quality - like the Bayer filter.

[Valeriu Campan]

It looks that the camera controller from RedLake has options for white balance settings and selectable output for various color spaces.

from the brochure:
"....White Balance Manual, Presets, and Semi-Automatic
Color Balance Real-time Advanced Color Filter Array Interpolation
Color Space Output RGB, sRGB, CIELab,YUV, or HSV..."

[David Newman]

Steve & others, I admit that we have yet to experiment with any direct signals from camera link, so much of my thoughts are theoretical and based on our experience with HD-SDI encoding. Our encoder and decoder pair function quite differently to other compression algorithms on the market (patents aren't granted yet so I can't tell you how.) I believe it would possible to encode 1920x1080 camera feed in 10 bits (we haven't done 12 yet) at about 30 to 40 fps on a 3GHz P4, and 10 bit 1280x720 feed at about 70-85fps. I know there will be other overheads so these numbers could be high, but I still believe 1920x1080 @ 24p would be possible on a P4. I know there are quality concerns but I believe I can overcome many of them (and I would like to try.)

Here what I would like to do (probably in my spare time); rather than getting a camera and link card (I'm still interested Steve) I would like a range of raw Bayer sequences from different cameras and resolutions of different sources (data pre-processing i.e. the type of data I would see across Camera Link.) Would anyone kindly send me a CD or DVD of raw material in any format, all I need to know is resolution and the packing format. From here I can experiment on encoding performance and offer more accurate numbers.

If anyone is interested please send a disk to:

David Newman
CineForm, Inc.
5315 Avenida Encinas Suite 230
Carlsbad CA 92008

Thanks.

[Les Dit]

David,
Would you be encoding the Bayer pattern directly?
The preferred Bayer demosaiking algorithms , like the variable gradient method, are adaptive algorithms. They are image dependent and take a lot of cycles. I don't think current CPU's, even with the most parallel SSE coding, can be made real time.
-Les

[David Newman]

Les,
I been doing some reading on various demosaiking algorithms, and I'm am aware that any conditionals are a pain for SSE optimization. Still it is worth a go. I would welcome links to any demosaiking algorithms you favor.

[Jason Rodriguez]

Hi David,

I've sent you a demosaicing algorithm that I've found along with sample MATLAB code that should be very interesting if it can be implemented. Another thing to note is that along with any demosaicing algorithm, there's going to have to be some sort of "false color" filter that can remove any vestages of the dreaded orange-blue color arifacts that occur once Nyquist has been exceeded for any chip. A good algorithm can reduce this color aliasing to almost nill, but there's still going to be some aliasing once Nyquist is exceeded, and that's going to happen with any sampling system out there, no matter how good your algorithm is. So again, we're also going to have to look at some sort of "false color" or de-moire/color-aliasing filter that can remove and color in the blue-orange areas with their surroundings.

[Les Dit]

David,
I've found that this paper was good at looking at the different methods of demosaiking:
http://www-ise.stanford.edu/~tingchen/main.htm

The sample code is in matlab, I got it working and tried them out. It's slow, but good for looking at the results.
There are a couple of other methods out there, one from Kodak ( patented, of course ) that seems to deal more with defect management than the actual demosaik itself.
Another method is 'Pixel grouping', but does not work as well as variable gradients for natural scenes.

On another topic, does Prem Pro 1.5 allow *any* higher bitdepth image to be imported without extra plugins? I tried tiff , rgb, and tga, and it does not seem to like any non 8-bit image from the get go. I guess it's a plug in I need to start testing with this idea. I have Cineons to start with.

-Les

[David Newman]

Les & Jason,

I'll look into the various algorithms to see what is both practical and good quality.

Les, Premiere Pro 1.x only includes an 8 bit engine. It allows for the replacement of its engine (importing, mixing and exporting can all be deeper than 8bit as we did with Prospect HD) which makes it very flexible, but not within the appropiate plug-ins.

David.

[Obin Olson]

david I am happy to send you raw files - what do you need from me?

[Les Dit]

Obin or Steve ,
How many noise free bits are coming from that cam ?

( subtracting 2 static 10 or 12 bit images, what's the average pixel value of the resulting close to black image ?)

Are we talking 9 bits at least , for a 1/48 sec integration?

-Les

[Rob Lohman]

Obin: can you please tell us how you got that movie? What
settings did you used for the camera (gain, FPS etc.). To what
format did you capture and how did you transform that into a
WM9 file? Did you do any color correction or other post work to
the footage? Which Bayer algorithm did you use?

What lens are you using for this system?

I'm wondering about your framerate issue with that WM9
encoded movie. Could it perhaps be that your harddisk could
not keep up with the data? To what harddisk system did you
record? (see my list below as well)

Steve: so can we do a full frame shutter with the chip or does
it only have a rolling shutter? Personally (as you know) I'm more
fond of getting a full frame than getting changing lines.

Regarding expensive bayer algorithms. As inidicated before
we can do bayer on the PC after capture if we want. An idea
that popped up in my mind was this that might be interesting:

1) get the RAW footage from the camera and dump this to disk

2) resize the footage to standard SD (include letterbox) using
some fast algorithm, do bayer (if needed) and do a simple
color shift for a crude white-balance. Send this to viewfinder
and realtime compress this in a second file that is being written
to disk as well.

The second file is a simple SD file that can be more easily
compressed and used in an edit system. We could then perhaps
edit the movie as normal and output a EDL that we could use
to conform the "big" raw uncompressed file?

The only issue remaining then is color correction, effects and
futher processing.

[Steve Nordhauser]

Rob,
The shuttering method is intregal to the chip design so you have to find a full frame shutter chip. This is the one I've seen mentioned on this site before:
http://www.micron.com/products/imagi...s/MT9M413.html

It has 12 micron pixels, 10 bit A/Ds, 15mm horizontal size, made my Micron (probably pretty low noise) and can do 720p. OK, maybe 100x faster also.

This would be an expensive camera since the chip is large and it is 10 tap. In 10 bit mode there are 100 parallel digital lines to be dealt with. Which would you rather have, gobal shutter or 1080p for the same $$?

[Rob Lohman]

I'd rather have global shutter... I think. As said I'm pretty new to
this whole (rolling) shutter system.

Since you have a lot more knowledge and experience with this
then I do; let me ask you the following:

1) do you think the camera will be able to give good enough pictures with a rolling shutter?

2) what type of images will we see skewing the most? I assume something fast moving?

3) is increasing the framerate and then dropping frames the only way we can get rid of this skewing

I'm not saying rolling shutter is bad etc. I'm just trying to
understand it all and see what the cons and pros are, as you
understand.

The "issue" I'm seeing now is that if you have fast moving
objects we will get skewing. The only way to get rid of it is by
moving the framerate up and dropping frames. But that loses
our edge of variable framerate recording for slow-motion which
almost always involve higher speeds.

So then the faster recording is not of any real use except to get
rid of skewing. Am I right in this conclusion?

I think the picture Obin showed looked very beautiful and I'm
very impressed with your chip. So that's a good sign for any
sells as far as I'm concerned!

Could we program the chip in such a manor that it will capture
say 48 fps but only sends 24 to the computer to get rid of skewing
and use less bandwidth since we don't use it anyway?

Thanks for your continued support on this matter. It IS greatly
appreciated by me and everyone else!

Personally I'm not too interested in HD, yet. There is no HD here
in Holland (basically). So the only reason would be blowing up to
film which might happen if we make a good enough movie. But
that is not garantueed. In that regard I'm not too scared since
they blowed up 28 days later as well and it still looked "okay" to
me.

BUT, ofcourse that was shot with 3 CCD's instead of 1 CMOS with
bayer pattern which ofcourse lowers the actual pixels.

The most important thing for me is uncompressed bits and the
larger 10 bit range. Ofcourse I'm going to run such a chip as yours
definitely at 1280x720 (progressive) to get the highest resolution
possible. That is neat to have, but is for me an extra.

I could care less about 1080p, personally. That is so much data
and bandwidth that I simply do not want to have. 720p will
already be a challenge enough in all of these regards so I'm
focussing on that first. Ofcourse in a couple of years I might want
to upgrade my chip to a higher resolution, but we'll see then
when we get there.

Ofcourse this is strictly for myself. Others seem to really want
the 1080p option. That's fine. Different people, different wishes.

[Rob Lohman]

I have a feeling not everyone is aware on how much data we are
actually talking about. So a small list below. I've included
recording time for a 200 GB harddisk (204800 MB) at these rates.

720 x 480 x 24 fps x 3 bytes (8 bit) = 23.73 MB/s (8630 sec / 143 min)
1280 x 720 x 24 fps x 3 bytes (8 bit) = 63.28 MB/s (3236 sec / 53 min)
1280 x 720 x 24 fps x 4 bytes (10 bit compact) = 84.38 MB/s (2427 sec / 40 min)
1280 x 720 x 24 fps x 6 bytes (10 bit) = 126.56 MB/s (1618 sec / 26 min)
1920 x 1080 x 24 fps x 3 bytes (8 bit) = 142.38 MB/s (1438 sec / 23 min)
1920 x 1080 x 24 fps x 4 bytes (10 bit compact) = 161.46 MB/s (1268 sec / 21 min)
1920 x 1080 x 24 fps x 6 bytes (10 bit) = 284.77 MB/s (719 sec / 11 min)

720 x 480 x 48 fps x 3 bytes (8 bit) = 47.46 MB/s (4315 sec / 71 min)
1280 x 720 x 48 fps x 3 bytes (8 bit) = 124.56 MB/s (1618 sec / 26 min)
1280 x 720 x 48 fps x 4 bytes (10 bit compact) = 168.76 MB/s (1213 sec / 20 min)
1280 x 720 x 48 fps x 6 bytes (10 bit) = 253.12 MB/s (809 sec / 13 min)
1920 x 1080 x 48 fps x 3 bytes (8 bit) = 284.76 MB/s (719 sec / 11 min)
1920 x 1080 x 48 fps x 4 bytes (10 bit compact) = 322.92 MB/s (634 sec / 10 min)
1920 x 1080 x 48 fps x 6 bytes (10 bit) = 569.54 MB/s (359 sec / 5 min)

Now even with raid I'd like to see us get over 100 - 150 MB/s
write rates. In any form I'd say compression is probably needed.