4:4:4 10bit single CMOS HD project

Wayne Morellini · June 16th, 2004, 01:33 PM

<<<-- Originally posted by Steve Nordhauser :Wayne: USB 2.0. To get high data rates, you eat a fair amount of CPU time. Also, transfer rates are highly dependent on the host controller. Intel ICH4 and 5 south bridges are 2-3x faster than external controllers. The typical USB interface chip has small FIFOs making real-time tough at high rates when you don't want to drop frames. We don't compress in the camera because either you want lossless, which only gets 2:1 or lossy and then you need to be able to select a lot of variables. CPU compression and raw recording are more cost effective. -->>>

I thought that might be the case, I thought I heard about them upgrading their specs so that it would off load the cpu ussage to a dedicated controll circuit (much like firewire), but I'm probably mistaken.

I like to say all this stuff with the read out peak speeds (with 1 GB you can average them out to disk) maxing out the interfaces is dissapionting, why don't they use buffer memory on the chip/camera (then you can run the next Micro camera at 216fps (24fps*9) and blank 8 out of 9 frames, that would solve much of the problem.

<<<-- Originally posted by Steve Nordhauser : Rob on BitJazz:2:1 lossless. Very image dependent though. And *Very* noise dependent. Remember that they are talking images after the Bayer de-mosaic step which puts that into the real-time path. I'm thinking the best method is to create a red, blue and green separate buffer and RLE each one. That should be doable at any resolution in real-time and save the Bayer for some real crunching. We need to hear from someone deep into Bayer. I'll try to get to the library and get a copy of that article. -->>>

This is what I was thinking too, the bayer has a pattern, but if you take all the pixels of the same colour you get three seperate gradient monochrome images, that the fax compression routine you mention could probably do a good job on (that was a looseless routine wanst it?).

About that compression performance of 5:1 (was it really 5:1). I think that any routine that reliably gets rid of niose before compression to get higher comrpession is acceptable, and it is prefferable to get rid of it at some stage anyway. RAW yes, but we are going to nuke this niose anyway, and a couple of other things. Am I right Steve, if we preproces the niose, Bayer (split to three mono files), and artifacts, we can archive higher than 2.5:1 average lossless compression?

Thanks

Wayne.

Rob Scott · June 16th, 2004, 01:45 PM

Quote:

Steve Nordhauser wrote
I'm thinking the best method is to create a red, blue and green separate buffer and RLE each one. That should be doable at any resolution in real-time.

I had that very idea! Once I get a camera in-hand I'm going to try separating the channels out into separate chunks and do zlib compression on them. I also found a real-time very fast compression library that I'm going to try out. Depending on the compression, we might be able to get far higher frame rates with a single drive.

Quote:

then you can run the next Micro camera at 216fps (24fps*9) and blank 8 out of 9 frames

I think that would make the shutter speed too fast and would tend to give unblurred frames. Good for sports, perhaps, or a "Saving Private Ryan" effect, but not for everything.

Les Dit · June 16th, 2004, 01:50 PM

Rob,
I've found that noise really limits the compression of digitized images. Using LZ compression on 10 bit images only gave me maybe 30% .
A lot of the codecs and compression schemes are a lot trickier when you have 10 or 12 bits of which the last few bits have a significant noise component. It's very important to keep those bits for color correcting later.
-Les

Steve Nordhauser · June 16th, 2004, 01:51 PM

Wayne on compression:
I think it was Obin that was talking about 500MB files in about 100MB of space. True lossless will max out around 2.5:1, no matter what you do. Certainly the less the black level offsets and gain non-linearities, the better the compression because then a smooth color will give smooth values.

Wayne on speeds: Yes, buffering in the camera or grabber would reduce peak speeds to average. Not sure if it is worth the cost.

Wayne on high speed: very fast is epensive - the Micron parts are rated at something like a 60MHz clock, Altasens at 75MHz. With internal A/Ds, you can't run much faster. Really fast parts like the Micron 1.3Mpix with global shutter have more taps out of the array and run them in parallel - that is a 10 tap.

Jason on rates: We have a programmable clock generator that works off of a base frequency. If we can't get to the exact number, we can change the base frequency. Using 23.976 as the target frequency, we can hit 23.977 with a 45.5ppm error. For Obin and Rob, that is a value of 0x35BD8F to the clock generator.

Rob Scott · June 16th, 2004, 02:00 PM

Quote:

Les Dit wrote:
Using LZ compression on 10 bit images only gave me maybe 30%

Well, that might still give me enough breathing room to get 30-40 fps on a single hard drive. Or prevent hiccups when Windows has to "do something" during capture.

Quote:

Steve Nordhauser wrote:
Using 23.976 as the target frequency, we can hit 23.977 with a 45.5ppm error

Interesting. Does anyone know how much error we can tolerate? Is there some tolerance specified in the NTSC and/or ATSC standards?

If we can't get close enough, I guess we'd just have to adjust the speed audio to match the video, like the "old" days of shooting silent film and wild sound. That's OK with me, doing indie filmmaking, but I'm not sure it's going to be OK with everyone...

Les Dit · June 16th, 2004, 02:02 PM

A good test for the rolling shutter artifact is panning across some vertical objects, like some buildings or a picket fence.
The image will look 'trapezoidal', as if for each scan line the image is offset by a pixel or so.
Most consumer still digi cams *do* have a shutter.
I have an el-cheapo CMOS one that has a 2MP sensor, and it has horrible rolling shutter problems, because it has no shutter. I took it apart, and I think it has either a Zoran or Micron sensor in there. Too bad it Jpegs it all, I can't see the image quality.

You may not see problems on a passing car, as cars these days don't have many straight edges, they look more like a used bar of soap :).

I think it will be essential to allow a longer integration time, and a short readout. Otherwise the image will look like it was made of Jello as you pan around, for lack of better words!

Any luck posting two Bayer images that we can look at for noise content?

-Les

Steve Nordhauser · June 16th, 2004, 02:06 PM

Les,
You are sure you were seeing rolling shutter problems and not interlacing? Interlaced images have their own problems with panning - the odd and even fields are one frame off so you get sawtooth edges - much worse than a RS effect.

Les Dit · June 16th, 2004, 02:08 PM

Indie film makers get away with shooting 25 fps in PAL and transfering it to film at 24 with NO CONVERSION at all !

Close enough for them !

Steve,
Most defiantly rolling shutter.
It's a SiPix still camera, a sub $100 still camera.

-Les

<<<-- Originally posted by Steve Nordhauser : Les,
You are sure you were seeing rolling shutter problems and not interlacing? Interlaced images have their own problems with panning - the odd and even fields are one frame off so you get sawtooth edges - much worse than a RS effect. -->>>

Rob Scott · June 16th, 2004, 02:22 PM

Quote:

Les Dit :
Indie film makers get away with shooting 25 fps in PAL and transfering it to film at 24 with NO CONVERSION at all !

Close enough for them !

True enough! There is plenty of software available now (I think Audacity will handle it) for adjusting audio without changing the pitch. (And a minor change in the pitch is usually no big deal -- for dialogue anyway.)

Steve Nordhauser · June 16th, 2004, 02:40 PM

Les: the two are not related. Rolling shutter is the sequential reading lines in a frame while the other lines integrate. Interlacing is reading every other line each frame so you traverse the image top to bottom twice as fast. The opposite of interlaced is progressive - no skipped lines for each frame read. You can have interlaced rolling shutter or progressive. All of ours are progressive.

Rob Lohman · June 16th, 2004, 03:39 PM

I'm pretty sure that 23.976 or 23.977 will be good enough. But
then again, I ain't in NTSC land and we have rock solid frame
rates here at exact 25 fps for example.

Steve & Rob: we are all on the same line in regards to first
seperating the planes and doing compression on them seperate.

I've just written a TIFF to RAW Bayer conversion program so we
can now test with any 16 bit TIFF basically. This gives us some
freedom. The program is in Rob S's posession as well.

Wayne: I did my first programming in low-level assembler before
moving to C(++) and then on to the Windows platform. I'm pretty
sure I know exactly how a PC works internally and how Windows
works as well. I've written assembler boot-loaders and some
low-level Windows stuff. The only thing I ain't really good at is
anything with Unix/Linux on the PC. Oh well...

Steve: do you happen to know if the 16-bit bytes are coming
in BIG or LITTLE endian order? Or do we get the high byte first
or the low?

A question regarding future camera's. Will they be controllable
and connectable in a similar fashion? In other words minor
changes to the software?

Obin Olson · June 16th, 2004, 04:30 PM

ok streampix works well..I shot 3500 frames in one set today at 12bit 24fps..this very good...workflow from stream pix is hell and goes like this:

1 Open xcap and set the camera exposure frame period gain ROI and mhz for each and every shot we shoot, save fmt camera file from xcap
2 Open streampix and load fmt file so that the camera is ready for the shot
3 Make new file for sequence
4 Shoot
5 Open shot black and white sequence with streampix
6 Load Bayer converter filter and make sure files will output in the right place with the right Bayer RGGB
7 Save sequence to disk for a long long time as the black and white sequence is being converted to color sequence
8 Open HUGE color sequence in streampix and output tiff files or avi or jpeg and wait for a very long time
9 Open what you have just saved and compress with some codec like SheerVideo so you can edit and finish project

almost all steps above should be this:

1 Open stream pix and in streampix set camera for shot incl. shutter fps etc
2 Shoot to sequence file
3 After shooting open sequence file and de-Bayer the image to color and save with a codec like SHeerVideo
4 Edit final footage

what's up with this board??

oh ya, almost forgot it takes over 45min to deal with the above files to get them into a SheerVIdeo codec quicktime compressed file! this is silly and will never work for professional work unless we can make fewer steps to get a sheervideo file

and don't forget the 22min it takes to transfer all the tiff files to the videoserver over gigabit lan

Steve Nordhauser · June 16th, 2004, 05:01 PM

Obin,
In terms of LAN speed, we have found that streaming over a gigE lan is limited to about 250Mbps from one machine to another using the Windows drivers. Even the raw data for 3500 frames is 1280x720x16x3500 = 51Gbps
At 250Mbps, that should take 206 seconds or 3.5 minutes. That is 16 bits per pixel.

For bigger shoots, if you don't need a raid or maybe only a two drive, use removables (slide into a chassis slot right on the IDE bus, not firewire) to move the data around. Just watch the heat - I don't think the removables cool as well as the fixed.

Jim Lafferty · June 16th, 2004, 05:30 PM

Is anyone looking into building a simple control system that would send the footage as is to some stacked drives?

I'm following these threads the best I can and keep reading about Obin's plans to build a small miniATX machine, but why not just have SATA drives in an enclosure that's within reasonable size/weight/power consumption specs ala the Vance Cam?

What sort of intercept system would have to be built to catch the stream and compress it on the fly?

What I'm getting at is -- I'd rather capture footage to disk without lugging a computer around with the cam. I want to just get the footage on site and then bring it back to a machine for post.

Is this possible? If so, can it be done with our resources?

Also, I mentioned elsewhere that it would be neat if custom "looks" could be generated as small files -- like BIOS flash files -- that we could all trade online as filmmakers. Based on what I know of the project as is, this isn't really feasible given that the camera is technically already built -- we're just figuring out innovative ways to harness its footage.

To my mind it would be amazing to be able to assemble an interface that would apply different effects to the footage with a way to prerserve and select among settings. These prefs would be output to a small file, and could be downloaded and installed to the cam. In this way you could change the looks of your footage as easily as you do ringtones on your cell phone. A community of open-source mod'ers could build visual "filter" toolkits this way.

Steve -- what kind of control is possible for the ambitiout enduser with regards to the image's lattitude and other characteristics? Is it possible that they could be changed on the fly and plugins could be built for them?

Thanks for all contributing,

- jim

Obin Olson · June 16th, 2004, 05:42 PM

ouch..it says it will take 58min to convert 16bit tif file's into the beta sheer codec....guess this could be because it's on the network drive and or it's dealing with a high bit depth.

Jim your thinking big, and thats good, but we have very basic problems at the moment like what to do with all the data this camera spits out...how to deal with high bit depth images and the list goes on and on and on...arggghh I am getting tired of it! I wish we had a magic button for this!

what the heck am I doing with 16bit tiff files anyway? the 1300 camera outputs 10bit that gets captured at 12bit and converted to 16bit tiff...what the HECK is that about ?? anyone? why does After Effects edit in 8 or 16bit ONLY what about 10 and 12bit?

maybe we should stay with simple 8bit files...everything works with them....argghh

June 16th, 2004, 01:51 PM	#274
Steve Nordhauser Silicon Imaging, Inc. Join Date: May 2004 Location: Troy, NY USA Posts: 325	Wayne on compression: I think it was Obin that was talking about 500MB files in about 100MB of space. True lossless will max out around 2.5:1, no matter what you do. Certainly the less the black level offsets and gain non-linearities, the better the compression because then a smooth color will give smooth values. Wayne on speeds: Yes, buffering in the camera or grabber would reduce peak speeds to average. Not sure if it is worth the cost. Wayne on high speed: very fast is epensive - the Micron parts are rated at something like a 60MHz clock, Altasens at 75MHz. With internal A/Ds, you can't run much faster. Really fast parts like the Micron 1.3Mpix with global shutter have more taps out of the array and run them in parallel - that is a 10 tap. Jason on rates: We have a programmable clock generator that works off of a base frequency. If we can't get to the exact number, we can change the base frequency. Using 23.976 as the target frequency, we can hit 23.977 with a 45.5ppm error. For Obin and Rob, that is a value of 0x35BD8F to the clock generator. __________________ Silicon Imaging, Inc. We see the Light! http://www.siliconimaging.com

June 16th, 2004, 02:02 PM	#276
Les Dit Major Player Join Date: Oct 2003 Location: Southern Cal-ee-for-Ni-ya Posts: 608	rolling shutter test A good test for the rolling shutter artifact is panning across some vertical objects, like some buildings or a picket fence. The image will look 'trapezoidal', as if for each scan line the image is offset by a pixel or so. Most consumer still digi cams do have a shutter. I have an el-cheapo CMOS one that has a 2MP sensor, and it has horrible rolling shutter problems, because it has no shutter. I took it apart, and I think it has either a Zoran or Micron sensor in there. Too bad it Jpegs it all, I can't see the image quality. You may not see problems on a passing car, as cars these days don't have many straight edges, they look more like a used bar of soap :). I think it will be essential to allow a longer integration time, and a short readout. Otherwise the image will look like it was made of Jello as you pan around, for lack of better words! Any luck posting two Bayer images that we can look at for noise content? -Les

June 16th, 2004, 02:06 PM	#277
Steve Nordhauser Silicon Imaging, Inc. Join Date: May 2004 Location: Troy, NY USA Posts: 325	Les, You are sure you were seeing rolling shutter problems and not interlacing? Interlaced images have their own problems with panning - the odd and even fields are one frame off so you get sawtooth edges - much worse than a RS effect. __________________ Silicon Imaging, Inc. We see the Light! http://www.siliconimaging.com

June 16th, 2004, 02:08 PM	#278
Les Dit Major Player Join Date: Oct 2003 Location: Southern Cal-ee-for-Ni-ya Posts: 608	frame rates Indie film makers get away with shooting 25 fps in PAL and transfering it to film at 24 with NO CONVERSION at all ! Close enough for them ! Steve, Most defiantly rolling shutter. It's a SiPix still camera, a sub $100 still camera. -Les <<<-- Originally posted by Steve Nordhauser : Les, You are sure you were seeing rolling shutter problems and not interlacing? Interlaced images have their own problems with panning - the odd and even fields are one frame off so you get sawtooth edges - much worse than a RS effect. -->>>

June 16th, 2004, 02:40 PM	#280
Steve Nordhauser Silicon Imaging, Inc. Join Date: May 2004 Location: Troy, NY USA Posts: 325	Les: the two are not related. Rolling shutter is the sequential reading lines in a frame while the other lines integrate. Interlacing is reading every other line each frame so you traverse the image top to bottom twice as fast. The opposite of interlaced is progressive - no skipped lines for each frame read. You can have interlaced rolling shutter or progressive. All of ours are progressive. __________________ Silicon Imaging, Inc. We see the Light! http://www.siliconimaging.com

June 16th, 2004, 01:33 PM	#271
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	<<<-- Originally posted by Steve Nordhauser :Wayne: USB 2.0. To get high data rates, you eat a fair amount of CPU time. Also, transfer rates are highly dependent on the host controller. Intel ICH4 and 5 south bridges are 2-3x faster than external controllers. The typical USB interface chip has small FIFOs making real-time tough at high rates when you don't want to drop frames. We don't compress in the camera because either you want lossless, which only gets 2:1 or lossy and then you need to be able to select a lot of variables. CPU compression and raw recording are more cost effective. -->>> I thought that might be the case, I thought I heard about them upgrading their specs so that it would off load the cpu ussage to a dedicated controll circuit (much like firewire), but I'm probably mistaken. I like to say all this stuff with the read out peak speeds (with 1 GB you can average them out to disk) maxing out the interfaces is dissapionting, why don't they use buffer memory on the chip/camera (then you can run the next Micro camera at 216fps (24fps9) and blank 8 out of 9 frames, that would solve much of the problem. <<<-- Originally posted by Steve Nordhauser : Rob on BitJazz:2:1 lossless. Very image dependent though. And Very* noise dependent. Remember that they are talking images after the Bayer de-mosaic step which puts that into the real-time path. I'm thinking the best method is to create a red, blue and green separate buffer and RLE each one. That should be doable at any resolution in real-time and save the Bayer for some real crunching. We need to hear from someone deep into Bayer. I'll try to get to the library and get a copy of that article. -->>> This is what I was thinking too, the bayer has a pattern, but if you take all the pixels of the same colour you get three seperate gradient monochrome images, that the fax compression routine you mention could probably do a good job on (that was a looseless routine wanst it?). About that compression performance of 5:1 (was it really 5:1). I think that any routine that reliably gets rid of niose before compression to get higher comrpession is acceptable, and it is prefferable to get rid of it at some stage anyway. RAW yes, but we are going to nuke this niose anyway, and a couple of other things. Am I right Steve, if we preproces the niose, Bayer (split to three mono files), and artifacts, we can archive higher than 2.5:1 average lossless compression? Thanks Wayne.

June 16th, 2004, 01:50 PM	#273
Les Dit Major Player Join Date: Oct 2003 Location: Southern Cal-ee-for-Ni-ya Posts: 608	Rob, I've found that noise really limits the compression of digitized images. Using LZ compression on 10 bit images only gave me maybe 30% . A lot of the codecs and compression schemes are a lot trickier when you have 10 or 12 bits of which the last few bits have a significant noise component. It's very important to keep those bits for color correcting later. -Les

June 16th, 2004, 03:39 PM	#281
Rob Lohman RED Code Chef Join Date: Oct 2001 Location: Holland Posts: 12,514	I'm pretty sure that 23.976 or 23.977 will be good enough. But then again, I ain't in NTSC land and we have rock solid frame rates here at exact 25 fps for example. Steve & Rob: we are all on the same line in regards to first seperating the planes and doing compression on them seperate. I've just written a TIFF to RAW Bayer conversion program so we can now test with any 16 bit TIFF basically. This gives us some freedom. The program is in Rob S's posession as well. Wayne: I did my first programming in low-level assembler before moving to C(++) and then on to the Windows platform. I'm pretty sure I know exactly how a PC works internally and how Windows works as well. I've written assembler boot-loaders and some low-level Windows stuff. The only thing I ain't really good at is anything with Unix/Linux on the PC. Oh well... Steve: do you happen to know if the 16-bit bytes are coming in BIG or LITTLE endian order? Or do we get the high byte first or the low? A question regarding future camera's. Will they be controllable and connectable in a similar fashion? In other words minor changes to the software? __________________ Rob Lohman, visuar@iname.com DV Info Wrangler & RED Code Chef Join the DV Challenge \| Lady X Search DVinfo.net for quick answers \| Buy from the best: DVinfo.net sponsors

June 16th, 2004, 04:30 PM	#282
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	ok streampix works well..I shot 3500 frames in one set today at 12bit 24fps..this very good...workflow from stream pix is hell and goes like this: 1 Open xcap and set the camera exposure frame period gain ROI and mhz for each and every shot we shoot, save fmt camera file from xcap 2 Open streampix and load fmt file so that the camera is ready for the shot 3 Make new file for sequence 4 Shoot 5 Open shot black and white sequence with streampix 6 Load Bayer converter filter and make sure files will output in the right place with the right Bayer RGGB 7 Save sequence to disk for a long long time as the black and white sequence is being converted to color sequence 8 Open HUGE color sequence in streampix and output tiff files or avi or jpeg and wait for a very long time 9 Open what you have just saved and compress with some codec like SheerVideo so you can edit and finish project almost all steps above should be this: 1 Open stream pix and in streampix set camera for shot incl. shutter fps etc 2 Shoot to sequence file 3 After shooting open sequence file and de-Bayer the image to color and save with a codec like SHeerVideo 4 Edit final footage what's up with this board?? oh ya, almost forgot it takes over 45min to deal with the above files to get them into a SheerVIdeo codec quicktime compressed file! this is silly and will never work for professional work unless we can make fewer steps to get a sheervideo file and don't forget the 22min it takes to transfer all the tiff files to the videoserver over gigabit lan

June 16th, 2004, 05:01 PM	#283
Steve Nordhauser Silicon Imaging, Inc. Join Date: May 2004 Location: Troy, NY USA Posts: 325	Obin, In terms of LAN speed, we have found that streaming over a gigE lan is limited to about 250Mbps from one machine to another using the Windows drivers. Even the raw data for 3500 frames is 1280x720x16x3500 = 51Gbps At 250Mbps, that should take 206 seconds or 3.5 minutes. That is 16 bits per pixel. For bigger shoots, if you don't need a raid or maybe only a two drive, use removables (slide into a chassis slot right on the IDE bus, not firewire) to move the data around. Just watch the heat - I don't think the removables cool as well as the fixed. __________________ Silicon Imaging, Inc. We see the Light! http://www.siliconimaging.com

June 16th, 2004, 05:30 PM	#284
Jim Lafferty Major Player Join Date: Dec 2003 Location: Brooklyn, NY Posts: 636	Is anyone looking into building a simple control system that would send the footage as is to some stacked drives? I'm following these threads the best I can and keep reading about Obin's plans to build a small miniATX machine, but why not just have SATA drives in an enclosure that's within reasonable size/weight/power consumption specs ala the Vance Cam? What sort of intercept system would have to be built to catch the stream and compress it on the fly? What I'm getting at is -- I'd rather capture footage to disk without lugging a computer around with the cam. I want to just get the footage on site and then bring it back to a machine for post. Is this possible? If so, can it be done with our resources? Also, I mentioned elsewhere that it would be neat if custom "looks" could be generated as small files -- like BIOS flash files -- that we could all trade online as filmmakers. Based on what I know of the project as is, this isn't really feasible given that the camera is technically already built -- we're just figuring out innovative ways to harness its footage. To my mind it would be amazing to be able to assemble an interface that would apply different effects to the footage with a way to prerserve and select among settings. These prefs would be output to a small file, and could be downloaded and installed to the cam. In this way you could change the looks of your footage as easily as you do ringtones on your cell phone. A community of open-source mod'ers could build visual "filter" toolkits this way. Steve -- what kind of control is possible for the ambitiout enduser with regards to the image's lattitude and other characteristics? Is it possible that they could be changed on the fly and plugins could be built for them? Thanks for all contributing, - jim __________________ Realism, anyway, is never exactly the same as reality, and in the cinema it is of necessity faked. -- J-L G

June 16th, 2004, 05:42 PM	#285
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	ouch..it says it will take 58min to convert 16bit tif file's into the beta sheer codec....guess this could be because it's on the network drive and or it's dealing with a high bit depth. Jim your thinking big, and thats good, but we have very basic problems at the moment like what to do with all the data this camera spits out...how to deal with high bit depth images and the list goes on and on and on...arggghh I am getting tired of it! I wish we had a magic button for this! what the heck am I doing with 16bit tiff files anyway? the 1300 camera outputs 10bit that gets captured at 12bit and converted to 16bit tiff...what the HECK is that about ?? anyone? why does After Effects edit in 8 or 16bit ONLY what about 10 and 12bit? maybe we should stay with simple 8bit files...everything works with them....argghh