4:4:4 10bit single CMOS HD project

Rob Scott · June 11th, 2004, 01:44 PM

Quote:

... I would simply get a frame in the frame grabbers buffer. I thought that was the main reason it was there. In other words, asks for 24 fps and you get a buffer that gets filled with information 24 times a second.

I think you can control to with a finer degree than that by setting the vertical blanking interval (etc.). I'm not quite certain exactly how this works, 'cause I haven't read through the docs thoroughly yet. I'm going to shut up now because I am at the limit of my knowledge about this. :-)

Steve Nordhauser · June 11th, 2004, 01:59 PM

Mainly, this is a degree of what you want to know at what level of detail. Once you get to the PC side, yes, you request a frame, you get the next frame appearing in a buffer. But, because what the camera is doing will influence your imaging, you might want to understand the camera side.

Basically, there are two types of sensor shutters - full frame (also called global and asynchronous) and rolling (synchronous). Full frame works like film exposure - expose the entire surface, stop exposing, and then readout. With few exceptions (Micron Truesnap being one and some CCD architectures), the two are sequential so you can't expose for a frame time. But, there is no difference in time for a pixel a the top of an image to the bottom. A pencil held vertically and moved horizontally will be blurred by the motion during exposure time but that is all. To do 24fps with a reasonable exposure on an IBIS5, you need to expose for 1/48 sec and readout in 1/48th of a sec.

Rolling shutter, is different. Each line is read and reset. You roll down through the image. You can also roll a second reset to get different exposure times, but there is always one frame time difference from a top line to a bottom. Most objects don't fill the screen so the effect isn't too much on smaller objects. Or pencil, though will have both blur from motion during exposure and look slanted due to the time difference. On the Micron, you can expose for up to 1/24th of a second.

Obin's plan is to run the Micron at 48fps. Take every other frame and chuck it. You now have 24fps. You also have frames that read out in 48th of a sec, minimizing the rolling shutter effect. Rob suggested having a longer blanking time (dead time between frames) of the regular time plus 1/48th of a sec - no second frame read out to lose later. Lower the average data rate and reduce the storage size.

Jason Rodriguez · June 11th, 2004, 02:01 PM

<<<-- Originally posted by Obin Olson : Rob I was up till 4am last night reading shit and it sure looks like you could (in the future) build a box with a FPGA that could be programmed to capture and spit out image files...don't let this stray you from the path if BASIC software..but think how awesome that would be in the future..no PC at all jsut camera FPGA system and disk drives! -->>>

It's called the Kinetta HD camera. Should be shipping by the end of the fall/winter.

Obin Olson · June 11th, 2004, 02:54 PM

guys I have some footage to show...I am very impressed
I will encode it to windowsmedia 9 HD

Mike Metken · June 11th, 2004, 03:06 PM

Guys,

Just a little thought. You should look at this from the commercial point of view. If you create a small storage unit that could be made to work with box (POV) cameras, you can really cash on this.

There is a Sony HDC-X300 1080p camera that costs $15K. Canon HD auto focus/manual lens for it is $7K. It outputs HD SDI. If you can make the storage box work with this camera, you've made a Sony F900 killer.

That's why I think that this should be as mainstream as possible. If the world out there is all HD SDI, this project should be too.

If you make your storage HD SDI and sell 2,000 of them at $2K profit each to the industry, you've made 40,000,000 and you can buy each of us one of the complete systems as a little tip. You normally tip 15%. In your case we'll settle for a lot less.

My idea is to keep to industry standards. The same as for the storage goes for those industrial cameras.

You do something non-standard and you could infringe on somebody's patent and will not even realize it. But if the all industry does it and there are no patent labels on the other cameras, you're most likely OK.

Mike

Obin Olson · June 11th, 2004, 03:19 PM

link to HD wmv file...framerate got jacked somehow..duno

www.dv3productions.com/Video Clips/HD-test.wmv

Dennis Jakobsen · June 11th, 2004, 03:39 PM

That looks very impressive...

Could easily have fooled me into thinking it was film. The contrast seems great. The colors seems very natural, and clean...

Hope you are able to fix the framerate...

Jason Rodriguez · June 11th, 2004, 03:43 PM

Obin,

Very cool stuff.

BTW, is the "shutter" speed too high? That's what it seems like from the looks of things.

Also I take it you were recording at 8-bit instead of 10-bit? Just wondering, because again, some really harsh clipping in the highlights. There's no knee on these cameras, everything I assume is perfectly linear, so there's nothing to simulate the S-curve of film. So when you slapping into the highlights, you are literally "slapping" right into them-and boy can they hurt! :-)

Mike Metken · June 11th, 2004, 03:52 PM

I don't think that you can have a linear output unless you are using something like a 16 bit sampling. You need to have the knee at least for the highlights.

Mike

Jason Rodriguez · June 11th, 2004, 03:56 PM

Quote:

Could easily have fooled me into thinking it was film. The contrast seems great.

Whoa there, I sure hope not, not with those highlights. Now I think when we get into 10 or 12-bit that we'll be talking about quality there. But for right now I see the promise, but not exactly the results. Then again, I totally understand that this is a project in the works, and I'm really excited to see where this is going.

BTW Obin, that's a really nice board you found, one question though is will this require a huge power supply like normal PC's (300W+)? If so, that could definitely take out the portability aspect of things.

Also where are you doing your bayer conversion right now? Inside XCAP? Rob, if you're going to write software, what type of algorithm are you planning to incorporate? I think right now the weakest link is the bayer algorithms out there that either produce 'stepppy' or 'stripped-dot' edges, and or blue/orange color aliasing. Actually normal DSLR cameras do this too when they reach the limiting resolution of the chip, but the conversion software has a special "false color filter" (at least that's what Canon calls it), that removes the blue/orange color moire problems where ever they do occur. So I think a good algorithm (even the best algorithms will produce color aliasing when the limit of the sensor is reached) combined with this approach (false color filtering) should give some really nice results.

Hi Steve,

Are you saying that with global syncronous shutters you can't expose for the entire duration of the frame, like a 360 degree shutter on film? For instance, in the Panasonic, Sony, etc. you can turn the shutter "off", so that you are exposing for the entire duration of the frame, i.e., 1/24th of a second for 24fps. This produces a lot of motion blur. Now if you can't do this with a global shutter, then none of the current HD cameras is using a global shutter, they must be using a roling shutter, unless their CCD's are setup with a full-frame that's over black-I've seen CCD's like that, that have one part exposed and then other other part is blanked over. But I don't believe most are like that, so therefore they probably have rolling shutters. If so, I've never seen what would appear like strange "slanting" artifacts, typically because there's so much motion blur when you have the "shutter" open that long. Does anybody know anything else about this?

Obin,

Is MicroATX small enough? 9.6" by 9.6" is pretty big IMHO. At least too big to make a nice box that you can carry on your shoulders without it sticking up way to high. Mini-ITX seems better at around 6.8x6.8, but there's nothing that size with a 64-bit PCI-X slot. Hmm, oh well . . .

Obin Olson · June 11th, 2004, 04:57 PM

Jason i want you to know this is ALL 8 bit and 8 bit SUCKS hard....BUT even with that in mind take a look at the truck jib like shot, the truck was almost totaly dark in the forground and the background is hard sunlight..EVEN in 8 bit I was able to pull the truck UP so you could see it...in 10bit you have so much more BUT even in 8 bit that is darn good in my book...notice the shot with me in it(ya that is me after 2 hours of sleep lastnight) that shot was exposed IN 8 bit for my face and you see the background is GONE if I exposed like I did for the truck then we would still have the background and could pull my face up and out of the blacks..try that with the Varicam!...bottom line is EVEN in 8 bit this thing is amazing in how much you can pull from the darks....this is from the fact that it's 4:4:4 so the darks have NO compression in them.....if only I could capture 10bit 24fps!!!!!!!!!!!! One thing I notice is this chip(cmos) has a MUCH softer look then ccd chips do when you hit the upper limits, I like it!

it's great to see some footage at last eh?(even 8 stinkin bits!!)

i am going with microatx..it's good enough and I am betting that microatx board with pci-x is $1,000 or more because they are the only company that makes one! so I will go with standard pci and dual sata for a 2 disk drive raid using a 3ghz p4 2gigs ram 1 ide os disk and 2 sata drives inside a microatx case and use a 7inch 1024x768 display that mounts on the camera and a 2nd "control" display for the system...maybe 17inch 1280x1024? and a good graphics card with dual head and svhs out for a producton monitor on-set...now if we can just feed Rob enough TacoBell cheesy-bean-and-rice we can get some software to run it all!!!!!

oh yes the days of minidv are so so so OVER!

Eliot Mack · June 11th, 2004, 05:09 PM

It's great to see the progress, Obin. Can 2 SATA striped drives handle 1280x720 10 bit uncompressed raw footage?

When do you think you'll have the full system pieces together to start integrating/bug fixing?

Thanks,

Eliot

Obin Olson · June 11th, 2004, 05:12 PM

10bit 1280x720 on raid sata? easy.

maybe even 48fps or more

Steve Nordhauser · June 11th, 2004, 07:54 PM

That micro ATX motherboard chip set can handle either the ICH5 or ICH5R southbridge. It would be great with the ICH5R since that would give you the built in RAID capabilities. Obin, you might want to ask. Otherwise, it is perfect.

Jason, on CCDs, there are a number of different architectures that allow you to overlap exposure and readout - interline transfer - you trade fill factor for extra storage between pixels, frame transfer - those are the ones you describe with an entire extra sensor split between two sides - much more expensive. Micron Truesnap is the only CMOS solution I've seen. We are releasing a VGA 250fps camera with Truesnap, but not applicable here.

Hey, when it comes time to make some money, don't forget me. If a product can be developed, I'd like to be involved in the camera head, system whatever.

Just a thought anyway. At some point in my life I may have to justify all this to my boss.

To be fair, if this gets carried away from where I am, let's say into HD-SDI, that is OK.

Jason Rodriguez · June 11th, 2004, 08:28 PM

So an IT, or FIT type of CCD does not use a rolling shutter, or doesn't have to? Also frame transfer CCD's don't have to use rolling Shutters?

And I'm sorry to beat this horse to death, but again, don't still cameras have rolling shutters (the curtain shutter), but I've never noticed any problems with those systems? And film cameras have a mechanical shutter that does not expose the film at the same time, it sweeps across the front of the film, again in a sideways "rolling" manner, so some areas of the film have an exposure later than other areas of the film-I haven't seen any weird smearing from that effect either.

June 11th, 2004, 01:59 PM	#167
Steve Nordhauser Silicon Imaging, Inc. Join Date: May 2004 Location: Troy, NY USA Posts: 325	Mainly, this is a degree of what you want to know at what level of detail. Once you get to the PC side, yes, you request a frame, you get the next frame appearing in a buffer. But, because what the camera is doing will influence your imaging, you might want to understand the camera side. Basically, there are two types of sensor shutters - full frame (also called global and asynchronous) and rolling (synchronous). Full frame works like film exposure - expose the entire surface, stop exposing, and then readout. With few exceptions (Micron Truesnap being one and some CCD architectures), the two are sequential so you can't expose for a frame time. But, there is no difference in time for a pixel a the top of an image to the bottom. A pencil held vertically and moved horizontally will be blurred by the motion during exposure time but that is all. To do 24fps with a reasonable exposure on an IBIS5, you need to expose for 1/48 sec and readout in 1/48th of a sec. Rolling shutter, is different. Each line is read and reset. You roll down through the image. You can also roll a second reset to get different exposure times, but there is always one frame time difference from a top line to a bottom. Most objects don't fill the screen so the effect isn't too much on smaller objects. Or pencil, though will have both blur from motion during exposure and look slanted due to the time difference. On the Micron, you can expose for up to 1/24th of a second. Obin's plan is to run the Micron at 48fps. Take every other frame and chuck it. You now have 24fps. You also have frames that read out in 48th of a sec, minimizing the rolling shutter effect. Rob suggested having a longer blanking time (dead time between frames) of the regular time plus 1/48th of a sec - no second frame read out to lose later. Lower the average data rate and reduce the storage size. __________________ Silicon Imaging, Inc. We see the Light! http://www.siliconimaging.com

June 11th, 2004, 03:06 PM	#170
Mike Metken Join Date: Jun 2004 Location: Carlsbad, CA Posts: 73	$40,000,000 for Your Efforts Guys, Just a little thought. You should look at this from the commercial point of view. If you create a small storage unit that could be made to work with box (POV) cameras, you can really cash on this. There is a Sony HDC-X300 1080p camera that costs $15K. Canon HD auto focus/manual lens for it is $7K. It outputs HD SDI. If you can make the storage box work with this camera, you've made a Sony F900 killer. That's why I think that this should be as mainstream as possible. If the world out there is all HD SDI, this project should be too. If you make your storage HD SDI and sell 2,000 of them at $2K profit each to the industry, you've made 40,000,000 and you can buy each of us one of the complete systems as a little tip. You normally tip 15%. In your case we'll settle for a lot less. My idea is to keep to industry standards. The same as for the storage goes for those industrial cameras. You do something non-standard and you could infringe on somebody's patent and will not even realize it. But if the all industry does it and there are no patent labels on the other cameras, you're most likely OK. Mike

June 11th, 2004, 07:54 PM	#179
Steve Nordhauser Silicon Imaging, Inc. Join Date: May 2004 Location: Troy, NY USA Posts: 325	That micro ATX motherboard chip set can handle either the ICH5 or ICH5R southbridge. It would be great with the ICH5R since that would give you the built in RAID capabilities. Obin, you might want to ask. Otherwise, it is perfect. Jason, on CCDs, there are a number of different architectures that allow you to overlap exposure and readout - interline transfer - you trade fill factor for extra storage between pixels, frame transfer - those are the ones you describe with an entire extra sensor split between two sides - much more expensive. Micron Truesnap is the only CMOS solution I've seen. We are releasing a VGA 250fps camera with Truesnap, but not applicable here. Hey, when it comes time to make some money, don't forget me. If a product can be developed, I'd like to be involved in the camera head, system whatever. Just a thought anyway. At some point in my life I may have to justify all this to my boss. To be fair, if this gets carried away from where I am, let's say into HD-SDI, that is OK. __________________ Silicon Imaging, Inc. We see the Light! http://www.siliconimaging.com

June 11th, 2004, 02:01 PM	#168
Jason Rodriguez Trustee Join Date: Mar 2003 Location: Virginia Beach, VA Posts: 1,095	<<<-- Originally posted by Obin Olson : Rob I was up till 4am last night reading shit and it sure looks like you could (in the future) build a box with a FPGA that could be programmed to capture and spit out image files...don't let this stray you from the path if BASIC software..but think how awesome that would be in the future..no PC at all jsut camera FPGA system and disk drives! -->>> It's called the Kinetta HD camera. Should be shipping by the end of the fall/winter.

June 11th, 2004, 02:54 PM	#169
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	guys I have some footage to show...I am very impressed I will encode it to windowsmedia 9 HD

June 11th, 2004, 03:19 PM	#171
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	link to HD wmv file...framerate got jacked somehow..duno www.dv3productions.com/Video Clips/HD-test.wmv

June 11th, 2004, 03:39 PM	#172
Dennis Jakobsen New Boot Join Date: Aug 2003 Location: Denmark Posts: 13	That looks very impressive... Could easily have fooled me into thinking it was film. The contrast seems great. The colors seems very natural, and clean... Hope you are able to fix the framerate...

June 11th, 2004, 03:43 PM	#173
Jason Rodriguez Trustee Join Date: Mar 2003 Location: Virginia Beach, VA Posts: 1,095	Obin, Very cool stuff. BTW, is the "shutter" speed too high? That's what it seems like from the looks of things. Also I take it you were recording at 8-bit instead of 10-bit? Just wondering, because again, some really harsh clipping in the highlights. There's no knee on these cameras, everything I assume is perfectly linear, so there's nothing to simulate the S-curve of film. So when you slapping into the highlights, you are literally "slapping" right into them-and boy can they hurt! :-)

June 11th, 2004, 03:52 PM	#174
Mike Metken Join Date: Jun 2004 Location: Carlsbad, CA Posts: 73	I don't think that you can have a linear output unless you are using something like a 16 bit sampling. You need to have the knee at least for the highlights. Mike

June 11th, 2004, 04:57 PM	#176
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	Jason i want you to know this is ALL 8 bit and 8 bit SUCKS hard....BUT even with that in mind take a look at the truck jib like shot, the truck was almost totaly dark in the forground and the background is hard sunlight..EVEN in 8 bit I was able to pull the truck UP so you could see it...in 10bit you have so much more BUT even in 8 bit that is darn good in my book...notice the shot with me in it(ya that is me after 2 hours of sleep lastnight) that shot was exposed IN 8 bit for my face and you see the background is GONE if I exposed like I did for the truck then we would still have the background and could pull my face up and out of the blacks..try that with the Varicam!...bottom line is EVEN in 8 bit this thing is amazing in how much you can pull from the darks....this is from the fact that it's 4:4:4 so the darks have NO compression in them.....if only I could capture 10bit 24fps!!!!!!!!!!!! One thing I notice is this chip(cmos) has a MUCH softer look then ccd chips do when you hit the upper limits, I like it! it's great to see some footage at last eh?(even 8 stinkin bits!!) i am going with microatx..it's good enough and I am betting that microatx board with pci-x is $1,000 or more because they are the only company that makes one! so I will go with standard pci and dual sata for a 2 disk drive raid using a 3ghz p4 2gigs ram 1 ide os disk and 2 sata drives inside a microatx case and use a 7inch 1024x768 display that mounts on the camera and a 2nd "control" display for the system...maybe 17inch 1280x1024? and a good graphics card with dual head and svhs out for a producton monitor on-set...now if we can just feed Rob enough TacoBell cheesy-bean-and-rice we can get some software to run it all!!!!! oh yes the days of minidv are so so so OVER!

June 11th, 2004, 05:09 PM	#177
Eliot Mack Regular Crew Join Date: Jun 2004 Location: Somerville, MA Posts: 37	It's great to see the progress, Obin. Can 2 SATA striped drives handle 1280x720 10 bit uncompressed raw footage? When do you think you'll have the full system pieces together to start integrating/bug fixing? Thanks, Eliot

June 11th, 2004, 05:12 PM	#178
Obin Olson Trustee Join Date: Jan 2003 Location: Wilmington NC Posts: 1,414	10bit 1280x720 on raid sata? easy. maybe even 48fps or more

June 11th, 2004, 08:28 PM	#180
Jason Rodriguez Trustee Join Date: Mar 2003 Location: Virginia Beach, VA Posts: 1,095	So an IT, or FIT type of CCD does not use a rolling shutter, or doesn't have to? Also frame transfer CCD's don't have to use rolling Shutters? And I'm sorry to beat this horse to death, but again, don't still cameras have rolling shutters (the curtain shutter), but I've never noticed any problems with those systems? And film cameras have a mechanical shutter that does not expose the film at the same time, it sweeps across the front of the film, again in a sideways "rolling" manner, so some areas of the film have an exposure later than other areas of the film-I haven't seen any weird smearing from that effect either.