Les,
I been doing some reading on various demosaiking algorithms, and I'm am aware that any conditionals are a pain for SSE optimization. Still it is worth a go. I would welcome links to any demosaiking algorithms you favor.

Hi David,

I've sent you a demosaicing algorithm that I've found along with sample MATLAB code that should be very interesting if it can be implemented. Another thing to note is that along with any demosaicing algorithm, there's going to have to be some sort of "false color" filter that can remove any vestages of the dreaded orange-blue color arifacts that occur once Nyquist has been exceeded for any chip. A good algorithm can reduce this color aliasing to almost nill, but there's still going to be some aliasing once Nyquist is exceeded, and that's going to happen with any sampling system out there, no matter how good your algorithm is. So again, we're also going to have to look at some sort of "false color" or de-moire/color-aliasing filter that can remove and color in the blue-orange areas with their surroundings.

David,
I've found that this paper was good at looking at the different methods of demosaiking:
http://www-ise.stanford.edu/~tingchen/main.htm

The sample code is in matlab, I got it working and tried them out. It's slow, but good for looking at the results.
There are a couple of other methods out there, one from Kodak ( patented, of course ) that seems to deal more with defect management than the actual demosaik itself.
Another method is 'Pixel grouping', but does not work as well as variable gradients for natural scenes.

On another topic, does Prem Pro 1.5 allow *any* higher bitdepth image to be imported without extra plugins? I tried tiff , rgb, and tga, and it does not seem to like any non 8-bit image from the get go. I guess it's a plug in I need to start testing with this idea. I have Cineons to start with.

-Les

Les & Jason,

I'll look into the various algorithms to see what is both practical and good quality.

Les, Premiere Pro 1.x only includes an 8 bit engine. It allows for the replacement of its engine (importing, mixing and exporting can all be deeper than 8bit as we did with Prospect HD) which makes it very flexible, but not within the appropiate plug-ins.

David.

david I am happy to send you raw files - what do you need from me?

Obin or Steve ,
How many noise free bits are coming from that cam ?

( subtracting 2 static 10 or 12 bit images, what's the average pixel value of the resulting close to black image ?)

Are we talking 9 bits at least , for a 1/48 sec integration?

-Les

Obin: can you please tell us how you got that movie? What
settings did you used for the camera (gain, FPS etc.). To what
format did you capture and how did you transform that into a
WM9 file? Did you do any color correction or other post work to
the footage? Which Bayer algorithm did you use?

What lens are you using for this system?

I'm wondering about your framerate issue with that WM9
encoded movie. Could it perhaps be that your harddisk could
not keep up with the data? To what harddisk system did you
record? (see my list below as well)

Steve: so can we do a full frame shutter with the chip or does
it only have a rolling shutter? Personally (as you know) I'm more
fond of getting a full frame than getting changing lines.

Regarding expensive bayer algorithms. As inidicated before
we can do bayer on the PC after capture if we want. An idea
that popped up in my mind was this that might be interesting:

1) get the RAW footage from the camera and dump this to disk

2) resize the footage to standard SD (include letterbox) using
some fast algorithm, do bayer (if needed) and do a simple
color shift for a crude white-balance. Send this to viewfinder
and realtime compress this in a second file that is being written
to disk as well.

The second file is a simple SD file that can be more easily
compressed and used in an edit system. We could then perhaps
edit the movie as normal and output a EDL that we could use
to conform the "big" raw uncompressed file?

The only issue remaining then is color correction, effects and
futher processing.

Rob,
The shuttering method is intregal to the chip design so you have to find a full frame shutter chip. This is the one I've seen mentioned on this site before:
http://www.micron.com/products/imagi...s/MT9M413.html

It has 12 micron pixels, 10 bit A/Ds, 15mm horizontal size, made my Micron (probably pretty low noise) and can do 720p. OK, maybe 100x faster also.

This would be an expensive camera since the chip is large and it is 10 tap. In 10 bit mode there are 100 parallel digital lines to be dealt with. Which would you rather have, gobal shutter or 1080p for the same $$?

I'd rather have global shutter... I think. As said I'm pretty new to
this whole (rolling) shutter system.

Since you have a lot more knowledge and experience with this
then I do; let me ask you the following:

1) do you think the camera will be able to give good enough pictures with a rolling shutter?

2) what type of images will we see skewing the most? I assume something fast moving?

3) is increasing the framerate and then dropping frames the only way we can get rid of this skewing

I'm not saying rolling shutter is bad etc. I'm just trying to
understand it all and see what the cons and pros are, as you
understand.

The "issue" I'm seeing now is that if you have fast moving
objects we will get skewing. The only way to get rid of it is by
moving the framerate up and dropping frames. But that loses
our edge of variable framerate recording for slow-motion which
almost always involve higher speeds.

So then the faster recording is not of any real use except to get
rid of skewing. Am I right in this conclusion?

I think the picture Obin showed looked very beautiful and I'm
very impressed with your chip. So that's a good sign for any
sells as far as I'm concerned!

Could we program the chip in such a manor that it will capture
say 48 fps but only sends 24 to the computer to get rid of skewing
and use less bandwidth since we don't use it anyway?

Thanks for your continued support on this matter. It IS greatly
appreciated by me and everyone else!

Personally I'm not too interested in HD, yet. There is no HD here
in Holland (basically). So the only reason would be blowing up to
film which might happen if we make a good enough movie. But
that is not garantueed. In that regard I'm not too scared since
they blowed up 28 days later as well and it still looked "okay" to
me.

BUT, ofcourse that was shot with 3 CCD's instead of 1 CMOS with
bayer pattern which ofcourse lowers the actual pixels.

The most important thing for me is uncompressed bits and the
larger 10 bit range. Ofcourse I'm going to run such a chip as yours
definitely at 1280x720 (progressive) to get the highest resolution
possible. That is neat to have, but is for me an extra.

I could care less about 1080p, personally. That is so much data
and bandwidth that I simply do not want to have. 720p will
already be a challenge enough in all of these regards so I'm
focussing on that first. Ofcourse in a couple of years I might want
to upgrade my chip to a higher resolution, but we'll see then
when we get there.

Ofcourse this is strictly for myself. Others seem to really want
the 1080p option. That's fine. Different people, different wishes.

I have a feeling not everyone is aware on how much data we are
actually talking about. So a small list below. I've included
recording time for a 200 GB harddisk (204800 MB) at these rates.

720 x 480 x 24 fps x 3 bytes (8 bit) = 23.73 MB/s (8630 sec / 143 min)
1280 x 720 x 24 fps x 3 bytes (8 bit) = 63.28 MB/s (3236 sec / 53 min)
1280 x 720 x 24 fps x 4 bytes (10 bit compact) = 84.38 MB/s (2427 sec / 40 min)
1280 x 720 x 24 fps x 6 bytes (10 bit) = 126.56 MB/s (1618 sec / 26 min)
1920 x 1080 x 24 fps x 3 bytes (8 bit) = 142.38 MB/s (1438 sec / 23 min)
1920 x 1080 x 24 fps x 4 bytes (10 bit compact) = 161.46 MB/s (1268 sec / 21 min)
1920 x 1080 x 24 fps x 6 bytes (10 bit) = 284.77 MB/s (719 sec / 11 min)

720 x 480 x 48 fps x 3 bytes (8 bit) = 47.46 MB/s (4315 sec / 71 min)
1280 x 720 x 48 fps x 3 bytes (8 bit) = 124.56 MB/s (1618 sec / 26 min)
1280 x 720 x 48 fps x 4 bytes (10 bit compact) = 168.76 MB/s (1213 sec / 20 min)
1280 x 720 x 48 fps x 6 bytes (10 bit) = 253.12 MB/s (809 sec / 13 min)
1920 x 1080 x 48 fps x 3 bytes (8 bit) = 284.76 MB/s (719 sec / 11 min)
1920 x 1080 x 48 fps x 4 bytes (10 bit compact) = 322.92 MB/s (634 sec / 10 min)
1920 x 1080 x 48 fps x 6 bytes (10 bit) = 569.54 MB/s (359 sec / 5 min)

Now even with raid I'd like to see us get over 100 - 150 MB/s
write rates. In any form I'd say compression is probably needed.

Rob,
The numbers that you show are appropriate for: pre-post processed single chip (Bayer de-mosaicing done but no compression) or 3 chip cameras. For single chip cameras like Obin's SI-1300, you are recording 1280x720x24fps, 10 bit, unpacked. (16 bits of transfer and storage) so you get about 44MB/sec *average* transfer or about 75minutes on a single drive.

Maybe Obin could do some test sequences of fast panning on objects with large vertical lines (buildings?) to test the rolling shutter artifacts at 24 and 48fps. I really don't know how objectionable the skew really is - everyone talks about it but I've never seen images other than packing labels on conveyor belts.
Steve

I think I missed the point that bayer is ofcourse a single color
per pixel indeed. So these numbers are indeed after bayer
conversion. When go to 10 bits we are indeed going to 2 bytes
per pixel instead of 1 (for 8-bits) or the 3 I had.

This futher indicates we would want to do bayer conversion
offline probably. This will greatly reduce bandwidth indeed.

Here is the new list storing the data BEFORE bayer conversion
(the previous list was AFTER):

1280 x 720 x 24 fps x 1 byte (8 bit) = 21.09 MB/s (9710 sec / 161 min)
1280 x 720 x 24 fps x 2 bytes (10 bit) = 42.19 MB/s (4854 sec / 90 min)
1920 x 1080 x 24 fps x 1 byte (8 bit) = 47.46 MB/s (4315 sec / 71 min)
1920 x 1080 x 24 fps x 2 bytes (10 bit) = 94.92 MB/s (2157 sec / 35 min)

1280 x 720 x 48 fps x 1 byte (8 bit) = 42.18 MB/s (4854 sec / 90 min)
1280 x 720 x 48 fps x 2 bytes (10 bit) = 84.38 MB/s (2427 sec / 40 min)
1920 x 1080 x 48 fps x 1 byte (8 bit) = 94.92 MB/s (2157 sec / 35 min)
1920 x 1080 x 48 fps x 2 bytes (10 bit) = 189.84 MB/s (1078 sec / 17 min)

We could perhaps lower this futher by packing data into all the
bits. So you get 4 pixels at 40 bits instead of 64 bits which is
another reduction in data bandwidth of 25%. If we re-shuffle
color bits we might even use a very fast lossless compression
method like RLE or something a bit more complex.

I will see if I can work something out on regards for an efficient
high-speed lossless algorithm for bayer patterns.

A quick search turned up that Dalsa is also using such an
algorithm which futhers enhances the feeling that his might be
the best way to go:It looks like we cannot use it since they hold a license on it, or
so they claim

Yes, that's the plan -- to store the raw pre-Bayer data to disk as quickly as possible. In my tests so far, I've been able to write 30 fps (1280 x 720 x 10 bits) with a single drive. (With nothing else going on.)

I don't have the camera SDK yet, but over the weekend I implemented a simple program to...

• "Grab" a frame (generates random data to simulate this)
• Write the frame to disk
• Do a simple Bayer filter (nearest neighbor)
• Slap the buffer up on the display (using SDL)

I'm doing a preview of 1 out of every 20 frames and I'm getting about 9 fps out to disk.

This sounds pretty bad -- and it is -- but I have done absolutely no optimization at all. The first thing I will do is split all these things into separate threads; that way, the "write to disk" thread can run at its lazy pace without slowing down capture or preview. I'll let you know how it goes.

One of my next steps was to evaluate GTK and Qt (my two top choices for GUI frameworks), but this recommendation may push me over the edge into Qt. I'll evaluate them and let you know.

I will definitely keep that in mind for ... what, version 1.2? Not sure yet. :-)

I have a basic understanding ofthe Bayer filter -- I've implemented a "nearest neighbor" filter and have tested it to make sure it works properly (it does -- though it's a pretty brain-dead filter). I have several documents on these filters and I'll be reading up on them.

My plan is to have several filters to choose from during the processing step (which happen offline). This part of the project will definitely be open source, so anyone can audit the quality of the filters and add new ones if they like.

Good to see you on the move as well Rob <g>

I've done a bit of digging and I must say I could not find that
much about lossless bayer compression except for the Dalsa
link above and some paper that's a bit over my head in terms
of math. I can develop algorithms on my own, but not write it
down in math.

A Fast Vector Quantization Engine for CFA Data Compression

A lot of terms in regards to Dalsa's encoding and general
encoding came up:

Golomb coding
Entropy coding
Wavelet compression

It seems Dalsa is futher with me on my remark to maybe use
lower resolution/quality proxies to do the editing:Rob: In what did you code that test set?

I'm looking for RAW bayer encoded 10 bit (stored as 16 bit files
please) file to do some algorithm testing. It sounds like you
already have a basic setup working. Can you send me a RAW
files or compile program to make them or source?

I'm assuming you did store the 10 bits inside 16 bits when
you tested writing? If so you where doing 53 MB/s writing. I
don't think I have any disk here that will sustain that.

How do you know you wrote it to disk in realtime?

Steve: from the looks of it the 1300 chip has
1280 x 1024 = 1.310.720 pixels, and thus:

655.360 green pixels
327.680 red pixels
327.680 blue pixels

Right?

Steve: do you happen to have any RAW bayer 10 bit files from the
1300 that we can use and actually test if we get a correct
picture? Also with compression etc.

p.s. I've been wrapping my head around Bayer today and it is
sinking in, so that's a good thing.

I wrote the code in C. The "test set" was two things:

• A "unit test" I wrote in C with arbitrary data values; I also coded the expected results, so this is self-checking.
• An 8-bit bitmap file I created with patterns I knew should generate certain colors. The test code read the "red" channel of the bitmap, "upscaled" it to 10 bits (thus simulating a raw image); then processed it and generated an 8-bit preview on screen.

I don't actually have any raw files yet, I'm just simulating them.

Does that answer your question?

No, I'm writing the packed 10-bit raw data.

Steve i would take global shutter ANYDAY over 1080P...I have seen the rolling shutter artifact and it is not what any of us want at all...

Rob and Rob tell me what you need and as soon as I get to work today i will try and upload stuff for you

A raw (pre-Bayer) file would be helpful. That way, either of us Robs (heh) could work on a Bayer algorithm without requiring the camera system or SDK.

anyone know of a HD DVD player that will play full on HD from a mpeg4 or windows media 9 endoded HD file on a dvd disk? I need a way to show this stuff in full blown HD and this seems like it would be a good answer if someone makes a player


A strange thing happens with the 1300 camera, if you shoot with NO gain on at all the whites are all grey..i must use about 3db gain to get whites and hotspots to be white not grey..anyone have an idea why this would be?

Rob Scott are you going to get a 1300 and capture board? Rob I think we need to take a very close look at Cineform for the Aspect HD stuff...we may need to use this because I sure can't edit any of this stuff at raw resolution on a standard 3ghz pc or over our gigabit network from our video file server

some info on the video clip I uploaded:

Everything is 8 bit - the software can't support 10 bit with my system at the moment.

The truck shot was exposed for the background that's in the sun and the truck was pulled up from the blacks - this 8 bit...you can do MUCH more in 10bit

The shot of me was exposed for my face - thats why it is so blown out in the background

Flowers are pretty much raw with no color work at all just a white balance

Everything has 3-4db gain on it and was shot close to 24fps

Product -
http://www.vinc.com/product.asp?ID=56&PID=21&SEQ=2

News articles -
http://www.cdfreaks.com/news/9852
http://news.designtechnica.com/article2925.html

Rob Scott don't forget ROI in your program so we can capture SD at 720x480 if we want to from this camera

The only DVD player I know of that supports HD:
http://www.digitalproducer.com/2004/...05/vinc016.htm

Obin: are you referring to your older clip? We really would like
something 10 bit. Can your system handle it if you drop the
resolution to something lower?

Can you also please respond to my questions about how you
obtained that movie and whether you processed it or not?

Yes, am placing my order today. How long did it take to get yours?
Absolutely. My intention is to support any codec installed on the user's machine, but it would be good to be able to test with Aspect HD and other codecs. I wonder if we can get them to donate a few codecs for testing purposes ...
Well, version 0.1 will probably do strictly 1280x720. I'll add features after I get the core stuff working reliably. Feel free to maintain a list of your requirements -- on your wiki page, perhaps.

BTW, don't forget about the Mac people and FCP, especially when there are a lot of people editing with FCP. I understand that your bayer de-mosaic program will probably be running on Windows or Linux x86 (btw, if you're using GTK, we can get that running on OSX under X Windows), but make sure that whatever files you end up writing to, that they can be used on a Mac too. As of right now, Aspect HD, etc. won't run on Mac or in FCP.

obin I agree with the bravo d3. It will play up to 1080p wm9
this is a link to a highly rated 1280x720p projector it has the mustang chip found in much higher priced projectors .
you would be capturing in 720p playing a dvd in 720p and projecting in 720p this would look incredible

http://www.hometheaterhifi.com/volum...or-8-2003.html

Absolutely! The "Capture" program will start as Windows-only (and eventually Linux), primarily because the SDK is only available for Windows and Linux. It will have a bare-bones UI, just enough to get by.

The "Convert" program will be based on GTK or Qt, both of which are cross-platform. We're probably going with Qt, which runs natively on Mac OS X - http://www.trolltech.com/products/qt/mac.html

I have a cousin who works in the industry and uses Mac/FCP, and he has agreed to help test the system.

(Edit - I just realized that Qt is not available for GPL programs for Windows, only for Linux/X11 and Mac OS. For this reason I may have to go with GTK ...)

1280 x720p
 

more good news for the indie filmaker.
this camera could go straight to theatre.

http://www.filmfestivals.com/cgi-bin...&text_id=23982

Rob do you have a link to the wiki page? I lost it, sorry!

I will post stuff I think you need to write into the software on that page as I think of stuff...glad your getting the camera !!! awesome!

No problem - http://www.obscuracam.com/wiki/wiki

homemade screen
 

Hi everyone this is a link to a homemade screen I made about a year ago. It's made from material you can buy at joann fabric
it duplicates screens costing thousands the materials are less than $100. It's perfect for displaying 1280x720 material .

http://www.btinternet.com/~paulw77/

Perhaps we prefer not to think about this..... but
Can someone evaluate the noise on the 1.3 camera?
Just subtract two images and see what happens.

How many stops does it *really* have? It's easy to find out!

Premiere Pro 1.5
 

HI!

I'm new here... very excited about what you guys are doing..

I was wondering if any of these imaging devices could capture in an even wider format? like full cinemascope or something... maybe 1645 x 700 or something.. anything 2.35:1. Without having to buy a $10,000 anamorphic lens that is :)

Re: Premiere Pro 1.5
 

Most of these chips are 4:3 (1.33:1) aspect ratio, but you can certainly get 2.35:1 by ignoring some of the vertical resolution. The Silicon Imaging (http://www.siliconimaging.com) unit we're working with right now is 1280x1024 and we're planning (for the most part) to use it in 1280x720 mode to match the HDTV standard.

To get more horizonal pixels you'd have to go with a larger resolution chip. Silicon Imaging also has a model with 2048 x 1536 -- which means you could get 2048 x 870 in a "Cinemascope" mode.

You'd only need an anamorphic adapter if you really didn't want to waste vertical space on the chip and if you didn't mind getting rectangular pixels.

i guess 1280 x 545 would still kicks ass.

keep up the good work!

Gee, I get busy for a day and you guys go prolific on me. Rob, I'm very pleased that you are doing this project. I've never been happy with the software costs of the systems we have since we try hard to keep the prices down. I bugged Alison to get your software out the door.

I would like to suggest that anyone who wants to help Rob on software send him a list of skills and appoximate time they can donate. That way he can coordinate the effort and not do it all himself. It doesn't have to be all programming. It can be "wrapping your head" around some aspect. Or, in Obin's case, making test images and sequences.

Rob L: your numbers look good now but remember, they are average and not peak across the bus. You are correct about the color ratios on all Bayers.

Also, we we start benchmarking disk systems, remember, disks slow down on the inner tracks.

Obin,

do you see the rolling shutter artifacts at 24fps with a 1/24th second shutter? I didn't see what I would call any motion artifacts in your footage (although it did seem to have an annoying studder, like a shutter that was too short), so I'm wondering if even at 1/24th of a second (so that you're not dropping frames) they are present or noticeable (the rolling shutter "slanting" effect).

Software update
 

Just a quick update. Last night I split the three parts of the code --

• Frame generator (simulates incoming frames)
• Writer (to disk)
• Preview (quick-and-dirty Bayer filter)

... and now I'm consistently getting between 20-22 fps (I was getting 9 before). Again, I haven't done much in the way of optimization, I just shuffled things around and put them in separate threads so they don't keep each other waiting.

I am using buffered I/O right now; I'm going to switch to standard I/O and see if that makes a difference. I am also wondering if I'm getting a worst-case scenario on my hard drive speed -- they are almost full. I may need to buy a new one and see if I get better speed with an empty drive.

The preview is only getting 1-2 fps, but that code is badly written and I think I can get much better speed just by doing a bit more optimization.

I'm also using SDL for display of the preview. Using DirectX would probably be faster, but I had trouble linking with the libraries.

If I have time I'll put up a Wiki page with some further details about my progress.

BTW, should I start a new thread for this?

I think this thread is fine Rob, Jason i will try and get you an example of rolling shutter artifact today..you won't like it, it's ugly...we can avoid it if we set the camera mhz high

uggh...I posted what i have been doing with HD at dv.com and got a bunch of child-like responses. Guess I will stick to this site!


Rob I really know nothing about DirectX but from what I hear overlay is the way to go...if you can offload anything to a graphics card I would think that would be good because of the raw power that new graphics cards have

Re: Re: Software update
 

Yeah, I'll certainly look into that again once I have made a bit more progress. Thanks!

this may be worth looking at:

http://www.pluginz.com/news/970

" BitJazz Launches Real-Time Nondestructive Video Codec "

Mike Merten

I agree with you fully, see my posts in the other threads on custom camera designs and the viper. The camera here is Obin private experiment, it's also being used as a trial run for the others camera comming (see other threads).

Thanks

Wayne.