4:4:4 10bit single CMOS HD project

[Steve Nordhauser]

Rob,
The numbers that you show are appropriate for: pre-post processed single chip (Bayer de-mosaicing done but no compression) or 3 chip cameras. For single chip cameras like Obin's SI-1300, you are recording 1280x720x24fps, 10 bit, unpacked. (16 bits of transfer and storage) so you get about 44MB/sec *average* transfer or about 75minutes on a single drive.

Maybe Obin could do some test sequences of fast panning on objects with large vertical lines (buildings?) to test the rolling shutter artifacts at 24 and 48fps. I really don't know how objectionable the skew really is - everyone talks about it but I've never seen images other than packing labels on conveyor belts.
Steve

[Rob Lohman]

I think I missed the point that bayer is ofcourse a single color
per pixel indeed. So these numbers are indeed after bayer
conversion. When go to 10 bits we are indeed going to 2 bytes
per pixel instead of 1 (for 8-bits) or the 3 I had.

This futher indicates we would want to do bayer conversion
offline probably. This will greatly reduce bandwidth indeed.

Here is the new list storing the data BEFORE bayer conversion
(the previous list was AFTER):

1280 x 720 x 24 fps x 1 byte (8 bit) = 21.09 MB/s (9710 sec / 161 min)
1280 x 720 x 24 fps x 2 bytes (10 bit) = 42.19 MB/s (4854 sec / 90 min)
1920 x 1080 x 24 fps x 1 byte (8 bit) = 47.46 MB/s (4315 sec / 71 min)
1920 x 1080 x 24 fps x 2 bytes (10 bit) = 94.92 MB/s (2157 sec / 35 min)

1280 x 720 x 48 fps x 1 byte (8 bit) = 42.18 MB/s (4854 sec / 90 min)
1280 x 720 x 48 fps x 2 bytes (10 bit) = 84.38 MB/s (2427 sec / 40 min)
1920 x 1080 x 48 fps x 1 byte (8 bit) = 94.92 MB/s (2157 sec / 35 min)
1920 x 1080 x 48 fps x 2 bytes (10 bit) = 189.84 MB/s (1078 sec / 17 min)

We could perhaps lower this futher by packing data into all the
bits. So you get 4 pixels at 40 bits instead of 64 bits which is
another reduction in data bandwidth of 25%. If we re-shuffle
color bits we might even use a very fast lossless compression
method like RLE or something a bit more complex.

I will see if I can work something out on regards for an efficient
high-speed lossless algorithm for bayer patterns.

A quick search turned up that Dalsa is also using such an
algorithm which futhers enhances the feeling that his might be
the best way to go:

DALSA's implementation extends basic L3 to include near-lossless and visually lossless compression, as well as compression of raw Bayer pattern data (as opposed to 3-channel RGB).

It looks like we cannot use it since they hold a license on it, or
so they claim

[Rob Scott]

This futher indicates we would want to do bayer conversion
offline probably. This will greatly reduce bandwidth indeed.

Yes, that's the plan -- to store the raw pre-Bayer data to disk as quickly as possible. In my tests so far, I've been able to write 30 fps (1280 x 720 x 10 bits) with a single drive. (With nothing else going on.)

I don't have the camera SDK yet, but over the weekend I implemented a simple program to...

• "Grab" a frame (generates random data to simulate this)
• Write the frame to disk
• Do a simple Bayer filter (nearest neighbor)
• Slap the buffer up on the display (using SDL)

I'm doing a preview of 1 out of every 20 frames and I'm getting about 9 fps out to disk.

This sounds pretty bad -- and it is -- but I have done absolutely no optimization at all. The first thing I will do is split all these things into separate threads; that way, the "write to disk" thread can run at its lazy pace without slowing down capture or preview. I'll let you know how it goes.

Qt is a really nice C++ GUI framework

One of my next steps was to evaluate GTK and Qt (my two top choices for GUI frameworks), but this recommendation may push me over the edge into Qt. I'll evaluate them and let you know.

sync the frame grabs with incoming time code from an external time code generator

I will definitely keep that in mind for ... what, version 1.2? Not sure yet. :-)

I think the biggest processing and recording issues are a complete understanding of any step that is a potential loss of quality - like the Bayer filter.

I have a basic understanding ofthe Bayer filter -- I've implemented a "nearest neighbor" filter and have tested it to make sure it works properly (it does -- though it's a pretty brain-dead filter). I have several documents on these filters and I'll be reading up on them.

My plan is to have several filters to choose from during the processing step (which happen offline). This part of the project will definitely be open source, so anyone can audit the quality of the filters and add new ones if they like.

[Rob Lohman]

Good to see you on the move as well Rob <g>

I've done a bit of digging and I must say I could not find that
much about lossless bayer compression except for the Dalsa
link above and some paper that's a bit over my head in terms
of math. I can develop algorithms on my own, but not write it
down in math.

A Fast Vector Quantization Engine for CFA Data Compression

A lot of terms in regards to Dalsa's encoding and general
encoding came up:

Golomb coding
Entropy coding
Wavelet compression

It seems Dalsa is futher with me on my remark to maybe use
lower resolution/quality proxies to do the editing:

For editing purposes, the camera will be able to output a 2K proxy to be edited in a fashion that will re-define our concept of "offline editing" with the final EDL and associated metadata used to assemble the final master from the original 4K files.

Rob: In what did you code that test set?

I'm looking for RAW bayer encoded 10 bit (stored as 16 bit files
please) file to do some algorithm testing. It sounds like you
already have a basic setup working. Can you send me a RAW
files or compile program to make them or source?

I'm assuming you did store the 10 bits inside 16 bits when
you tested writing? If so you where doing 53 MB/s writing. I
don't think I have any disk here that will sustain that.

How do you know you wrote it to disk in realtime?

Steve: from the looks of it the 1300 chip has
1280 x 1024 = 1.310.720 pixels, and thus:

655.360 green pixels
327.680 red pixels
327.680 blue pixels

Right?

Steve: do you happen to have any RAW bayer 10 bit files from the
1300 that we can use and actually test if we get a correct
picture? Also with compression etc.

p.s. I've been wrapping my head around Bayer today and it is
sinking in, so that's a good thing.

[Rob Scott]

In what did you code that test set?

I wrote the code in C. The "test set" was two things:

• A "unit test" I wrote in C with arbitrary data values; I also coded the expected results, so this is self-checking.
• An 8-bit bitmap file I created with patterns I knew should generate certain colors. The test code read the "red" channel of the bitmap, "upscaled" it to 10 bits (thus simulating a raw image); then processed it and generated an 8-bit preview on screen.

I don't actually have any raw files yet, I'm just simulating them.

Does that answer your question?

I'm assuming you did store the 10 bits inside 16 bits when
you tested writing?

No, I'm writing the packed 10-bit raw data.

[Obin Olson]

Steve i would take global shutter ANYDAY over 1080P...I have seen the rolling shutter artifact and it is not what any of us want at all...

Rob and Rob tell me what you need and as soon as I get to work today i will try and upload stuff for you

[Rob Scott]

Originally posted by Obin Olson : Rob and Rob tell me what you need and as soon as I get to work today i will try and upload stuff for you

A raw (pre-Bayer) file would be helpful. That way, either of us Robs (heh) could work on a Bayer algorithm without requiring the camera system or SDK.

[Obin Olson]

anyone know of a HD DVD player that will play full on HD from a mpeg4 or windows media 9 endoded HD file on a dvd disk? I need a way to show this stuff in full blown HD and this seems like it would be a good answer if someone makes a player


A strange thing happens with the 1300 camera, if you shoot with NO gain on at all the whites are all grey..i must use about 3db gain to get whites and hotspots to be white not grey..anyone have an idea why this would be?

Rob Scott are you going to get a 1300 and capture board? Rob I think we need to take a very close look at Cineform for the Aspect HD stuff...we may need to use this because I sure can't edit any of this stuff at raw resolution on a standard 3ghz pc or over our gigabit network from our video file server

some info on the video clip I uploaded:

Everything is 8 bit - the software can't support 10 bit with my system at the moment.

The truck shot was exposed for the background that's in the sun and the truck was pulled up from the blacks - this 8 bit...you can do MUCH more in 10bit

The shot of me was exposed for my face - thats why it is so blown out in the background

Flowers are pretty much raw with no color work at all just a white balance

Everything has 3-4db gain on it and was shot close to 24fps

[Rob Scott]

Originally posted by Obin Olson : anyone know of a HD DVD player that will play full on HD from a mpeg4 or windows media 9 endoded HD file on a dvd disk? I need a way to show this stuff in full blown HD and this seems like it would be a good answer if someone makes a player

Product -
http://www.vinc.com/product.asp?ID=56&PID=21&SEQ=2

News articles -
http://www.cdfreaks.com/news/9852
http://news.designtechnica.com/article2925.html

[Obin Olson]

Rob Scott don't forget ROI in your program so we can capture SD at 720x480 if we want to from this camera

[Rob Lohman]

The only DVD player I know of that supports HD:
http://www.digitalproducer.com/2004/...05/vinc016.htm

Obin: are you referring to your older clip? We really would like
something 10 bit. Can your system handle it if you drop the
resolution to something lower?

Can you also please respond to my questions about how you
obtained that movie and whether you processed it or not?

[Rob Scott]

Originally posted by Obin Olson :
Rob Scott are you going to get a 1300 and capture board?

Yes, am placing my order today. How long did it take to get yours?

I think we need to take a very close look at Cineform for the Aspect HD stuff

Absolutely. My intention is to support any codec installed on the user's machine, but it would be good to be able to test with Aspect HD and other codecs. I wonder if we can get them to donate a few codecs for testing purposes ...

don't forget ROI in your program so we can capture SD at 720x480 if we want to from this camera

Well, version 0.1 will probably do strictly 1280x720. I'll add features after I get the core stuff working reliably. Feel free to maintain a list of your requirements -- on your wiki page, perhaps.

[Jason Rodriguez]

BTW, don't forget about the Mac people and FCP, especially when there are a lot of people editing with FCP. I understand that your bayer de-mosaic program will probably be running on Windows or Linux x86 (btw, if you're using GTK, we can get that running on OSX under X Windows), but make sure that whatever files you end up writing to, that they can be used on a Mac too. As of right now, Aspect HD, etc. won't run on Mac or in FCP.

[Richard Mellor]

obin I agree with the bravo d3. It will play up to 1080p wm9
this is a link to a highly rated 1280x720p projector it has the mustang chip found in much higher priced projectors .
you would be capturing in 720p playing a dvd in 720p and projecting in 720p this would look incredible

http://www.hometheaterhifi.com/volum...or-8-2003.html

[Rob Scott]

Originally posted by Jason Rodriguez : BTW, don't forget about the Mac people and FCP, especially when there are a lot of people editing with FCP.

Absolutely! The "Capture" program will start as Windows-only (and eventually Linux), primarily because the SDK is only available for Windows and Linux. It will have a bare-bones UI, just enough to get by.

The "Convert" program will be based on GTK or Qt, both of which are cross-platform. We're probably going with Qt, which runs natively on Mac OS X - http://www.trolltech.com/products/qt/mac.html

I have a cousin who works in the industry and uses Mac/FCP, and he has agreed to help test the system.

(Edit - I just realized that Qt is not available for GPL programs for Windows, only for Linux/X11 and Mac OS. For this reason I may have to go with GTK ...)