4:4:4 10bit single CMOS HD project

Ben Syverson · August 2nd, 2004, 10:23 AM

Hopefully there's some way to control the slopes (Sumix mentioned a multislope setup, with control over each color), because as it is, the dual-slope looks extremely unnatural...

Steve Nordhauser · August 2nd, 2004, 12:52 PM

Ben:
This is one of my beefs with the IBIS-5. In global shutter mode, the integration time is sequential with the readout time. Let's say that they can move data over USB 2.0 at 40MB/sec and that the 10 bit data is packed. So, you are streaming at 320Mbps or 32Mpix/sec max rate. The real transfer time you need is 1280x720 x 24 = 22Mpix/sec. That leftover time is all you get for integration. The IBIS-5 isn't that sensitive to start, so crank up the gain or the lights in global shutter mode.

With CCD interline transfer or CMOS Truesnap you can overlap the integation time and readout time.

The IBIS-5 doesn't exactly control slope. You can set a knee point and any pixel that is saturated at the knee is reset. In my lamp example, the light is reset very shortly before closing the shutter. This lets the bright areas integrate for a short period while allowing the full integration for the rest of the image. Unless it is finely tuned, it does look very artificial. Very nice for machine vision though.

Ben Syverson · August 2nd, 2004, 03:46 PM

Steve -- interesting. Thanks for the illumination. I did do a couple of global shutter tests recently (with still frames) and I found it to be much noisier (and darker) than rolling shutter with the same settings (even without gain). I'm not 100% sure what's going on there, but it was enough to dissuade me from global shutter for now.

I'm implementing a 16 pixel Spline interpolation for linBayer. The results are extremely promising.

This is the normal (bilinear) interpolation

This is the Spline16 interpolation

I should note that the process is still 100% lossless, even with Spline16 -- the original pixel values are not changed, so you could extract the original sensor data from the RGB data and do a different de-Bayer routine in the future if you so desired...

It's an interesting problem, because any interpolator has to fill in 75% of the image data in the R + B channels. Spline seems to do a really good job. I don't think Spline36 would be worth it, for all the extra processing.

- ben

Jason Rodriguez · August 2nd, 2004, 03:49 PM

Hi Ben,

That spline-16 looks very nice-will this also be implemented on the green channel as well as the red and blue?

Ben Syverson · August 2nd, 2004, 03:56 PM

Jason, I'll give it a shot. Maybe Spline is all we need to get rid of our gridding and zippering problems, since it sees a neighborhood of 16 pixels for every pixel interpolated. However, it means that processing 720p really will be slow, 1080 even more so. Bilinear is an integer operation, whereas spline really needs to take place in floating point. That in itself is not the speed hit, it's the type conversion to/from integer from/to float -- 512 times for every pixel.

At this point, it might make sense to read the image into a floating point array and do everything in float. No type conversion means no slowdown...

Jason et al,

Spline doesn't help at all with zippering. In fact, it looks almost identical to bilinear, since it doesn't have to do very much interpolating (50% of the data is already there).

Our de-zippering stuff still beats anything I've seen, but I'm open to suggestions as always!

Jason Keenan · August 2nd, 2004, 07:37 PM

Wayne:

Yes Raavin is Jason Keenan. I'm using my normal handle just so when people reply to me, others don't get confused with the other Jason. I've seen cameras cheaper from www.videoguys.com.au. They have a good selection too.

Ben:

From a lay point of view, I agree with you about leaving the raw data alone and converting it with a codec. It just seems to make sense. I think it was you who replied to my comment about using avisynth. I suppose that's more what I was thinking.

I'm still trying to get my head around the 8 bit versus 10 bit stuff. Having a look at the raw file in a hex editor and using some flawed logic, it seemed to me that when you drop the raw file to 8 bit you halve the size of the original because the original 10 bits needs a full 16 bits to fit into. Is that right?

On another line, just thinking out loud, is it possible to stream the file through a fantasy 8 bit conversion codec, basically just dropping the 2 extra bits on the fly for preview and editing, then be able to change codecs for colour grading and final rendering. I'm assuming though that you need at least most of the bits to do the bayer conversion though. It just seems that in order to keep things flexible and compatible you need to have the ability to open the file in anything and at the full frame rate.

Anyone got any links to how bitmaps are arranged in hex format for me???

Raavin :)

Jason Rodriguez · August 2nd, 2004, 08:02 PM

Quote:

Spline doesn't help at all with zippering

Did it help at all with the "gridding"? I thought your de-zipper system was already really good, so I definitely couldn't see much improvement there, but the "gridding" in low contrast areas/out-of-focus/gradient areas was really the problem for me at least.

Ben Syverson · August 2nd, 2004, 08:06 PM

The gridding is an artifact of the de-zippering that only occurs when the two green offets or gains don't quite match. So since I wasn't doing any de-zippering, there wasn't any gridding. However, since the two greens on the shot I was working on didn't quite match, you could see a slight horizontal line pattern (just as you would with bilinear)...

Basically, the solution is to correctly calibrate your shot before you record. Otherwise, you'll either get a horizontal line pattern. Or gridding when you de-zipper.

- ben

Jason Keenan · August 2nd, 2004, 10:31 PM

Another couple of stupid questions

1) In the '10 bit' raw bayer file, does that mean that in each pixel, there are 1024 shades of grey?

2) Anyone care to give a breif explaination of how these pixel values are represented in the file???

Raavin :)

Ben Syverson · August 2nd, 2004, 10:41 PM

Raavin, yeah -- 10 bit represents 1024 values. You can always determine the number of values by doing
2^x
Where x = the number of bits. So 2^10 == 1024

I don't know what you mean by "represented in the file," but any native 10-bit format will do bit packing. That is, since a byte is 8bits, and since the byte is how we write to files, it means that uncompressed 10 bit would be written like this. I'll use "1" to represent the first 10-bit value, "2" to represent the second 10-bit value, etc. Each group of 8 numbers is a byte written to the file.

11111111____11222222____22223333____33333344____44444444

etc

That means it won't make much sense in a hex viewer. However, one extremely inefficient way to store 10bit data is to pad it to 2 bytes (16bit). That would look like this:

0000001111111111___0000002222222222___0000003333333333___0000004444444444

As you can see, we're using far more bits/bytes to represent the same data.

- ben

Wayne Morellini · August 2nd, 2004, 11:22 PM

<<<-- Originally posted by Ben Syverson :
But I'd rather not work directly on Rob's app, as I think he's focusing on a PC-based solution, whereas everything I do is on the Mac. Also, I think Rob's sw is basically capture software specifically for CameraLink

@Wanye: the fastest codec (huffy??)

A 10 or 12 bit JPEG/Wavelet would be fine for 99.9999999999% of the people on this board, but it would be slower than lossless, so what's the point?

- ben -->>>

Only mentioned it because those two are popular topics here and support the range of compression ratios that people want to see.

I think it was Jaun that spoke about a Bayer version of Huff.

The wavelet stuff is not the fastest but one of the best people like here, and able to do completely lossless.

Rob is trying to do a Universal version of capture software that works with any codec, on the PC first then MAC/Linux. I think he intends to support USB and Gigabit versions of cameralink cameras, but I am unsure. So the codec and capture side would still be seperate.

Wayne.

Wayne Morellini · August 2nd, 2004, 11:23 PM

<<<-- Originally posted by Jason Keenan : Wayne:

confused with the other Jason. I've seen cameras cheaper from www.videoguys.com.au. They have a good selection too.
-->>>

Thanks.

Jason Keenan · August 2nd, 2004, 11:35 PM

Ok, I think I've got it.

So, in your example-

"11111111____11222222____22223333____33333344____44444444"

-if we had 4 pixels which were alternating white and black, it would be

11111111____11000000____00001111____11111100____00000000

yeah???

Would this then show in a hex editor as

ff C0 0f fc 00 ?????? if it was true 10 bit and not padded to 16 bit????

Has it been ascertained whether the bin captures are padded to 16 bit or not. Every other byte has a leading zero so if it's the 'rawest' format available I'm guessing it is.

eg "00 00 90 05 ac 06 18 05 54 06 38 06 28 07 28"
from raw_cap.bin

Thanks for the answers, I'm just trying to get my head around it.

Cheers,

Raavin

Ben Syverson · August 2nd, 2004, 11:48 PM

@Raavin: Would this then show in a hex editor as ff C0 0f fc 00 ?????? if it was true 10 bit and not padded to 16 bit????

Yes, precisely. As Rob clarified, the 10-bit data coming off the SI cameras is padded to 16bits in the following manner: (this would represent 1 value)
0000##########00
Where the hashes represent bits with data. So it's 10-bits of data, padded out to 12-bits, within a 16bit number. It seems very strange to me.

But yes, that's why every pixel will look like: 0X XX in hex, because the first four bits are always 0. And the last two bits are always zeroes, so the last hex value of every pixel will always be C, D, E or F.

But who cares about hex? :)

- ben

Edgar Rodriguez · August 3rd, 2004, 12:16 AM

Hi, this is my first post but I've been following this thread since the start. Congratulations everyone for the great work!!!

-Avid has a codec called DNxHD http://www.avid.com/DNxHD/
6:1 full raster (sampling every pixel on the image) It works for 8-bit and 10-bit at 4:2:2
this will give us uncompressed quality at almost uncompressed SD size

I think this realy would be the best codec to go for, with this project.

August 2nd, 2004, 12:52 PM	#1277
Steve Nordhauser Silicon Imaging, Inc. Join Date: May 2004 Location: Troy, NY USA Posts: 325	Ben: This is one of my beefs with the IBIS-5. In global shutter mode, the integration time is sequential with the readout time. Let's say that they can move data over USB 2.0 at 40MB/sec and that the 10 bit data is packed. So, you are streaming at 320Mbps or 32Mpix/sec max rate. The real transfer time you need is 1280x720 x 24 = 22Mpix/sec. That leftover time is all you get for integration. The IBIS-5 isn't that sensitive to start, so crank up the gain or the lights in global shutter mode. With CCD interline transfer or CMOS Truesnap you can overlap the integation time and readout time. The IBIS-5 doesn't exactly control slope. You can set a knee point and any pixel that is saturated at the knee is reset. In my lamp example, the light is reset very shortly before closing the shutter. This lets the bright areas integrate for a short period while allowing the full integration for the rest of the image. Unless it is finely tuned, it does look very artificial. Very nice for machine vision though. __________________ Silicon Imaging, Inc. We see the Light! http://www.siliconimaging.com

August 2nd, 2004, 10:23 AM	#1276
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	Hopefully there's some way to control the slopes (Sumix mentioned a multislope setup, with control over each color), because as it is, the dual-slope looks extremely unnatural...

August 2nd, 2004, 03:46 PM	#1278
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	Steve -- interesting. Thanks for the illumination. I did do a couple of global shutter tests recently (with still frames) and I found it to be much noisier (and darker) than rolling shutter with the same settings (even without gain). I'm not 100% sure what's going on there, but it was enough to dissuade me from global shutter for now. I'm implementing a 16 pixel Spline interpolation for linBayer. The results are extremely promising. This is the normal (bilinear) interpolation This is the Spline16 interpolation I should note that the process is still 100% lossless, even with Spline16 -- the original pixel values are not changed, so you could extract the original sensor data from the RGB data and do a different de-Bayer routine in the future if you so desired... It's an interesting problem, because any interpolator has to fill in 75% of the image data in the R + B channels. Spline seems to do a really good job. I don't think Spline36 would be worth it, for all the extra processing. - ben

August 2nd, 2004, 03:49 PM	#1279
Jason Rodriguez Trustee Join Date: Mar 2003 Location: Virginia Beach, VA Posts: 1,095	Hi Ben, That spline-16 looks very nice-will this also be implemented on the green channel as well as the red and blue?

August 2nd, 2004, 03:56 PM	#1280
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	Jason, I'll give it a shot. Maybe Spline is all we need to get rid of our gridding and zippering problems, since it sees a neighborhood of 16 pixels for every pixel interpolated. However, it means that processing 720p really will be slow, 1080 even more so. Bilinear is an integer operation, whereas spline really needs to take place in floating point. That in itself is not the speed hit, it's the type conversion to/from integer from/to float -- 512 times for every pixel. At this point, it might make sense to read the image into a floating point array and do everything in float. No type conversion means no slowdown... Jason et al, Spline doesn't help at all with zippering. In fact, it looks almost identical to bilinear, since it doesn't have to do very much interpolating (50% of the data is already there). Our de-zippering stuff still beats anything I've seen, but I'm open to suggestions as always!

August 2nd, 2004, 07:37 PM	#1281
Jason Keenan New Boot Join Date: Jul 2004 Posts: 19	Wayne: Yes Raavin is Jason Keenan. I'm using my normal handle just so when people reply to me, others don't get confused with the other Jason. I've seen cameras cheaper from www.videoguys.com.au. They have a good selection too. Ben: From a lay point of view, I agree with you about leaving the raw data alone and converting it with a codec. It just seems to make sense. I think it was you who replied to my comment about using avisynth. I suppose that's more what I was thinking. I'm still trying to get my head around the 8 bit versus 10 bit stuff. Having a look at the raw file in a hex editor and using some flawed logic, it seemed to me that when you drop the raw file to 8 bit you halve the size of the original because the original 10 bits needs a full 16 bits to fit into. Is that right? On another line, just thinking out loud, is it possible to stream the file through a fantasy 8 bit conversion codec, basically just dropping the 2 extra bits on the fly for preview and editing, then be able to change codecs for colour grading and final rendering. I'm assuming though that you need at least most of the bits to do the bayer conversion though. It just seems that in order to keep things flexible and compatible you need to have the ability to open the file in anything and at the full frame rate. Anyone got any links to how bitmaps are arranged in hex format for me??? Raavin :)

August 2nd, 2004, 08:06 PM	#1283
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	The gridding is an artifact of the de-zippering that only occurs when the two green offets or gains don't quite match. So since I wasn't doing any de-zippering, there wasn't any gridding. However, since the two greens on the shot I was working on didn't quite match, you could see a slight horizontal line pattern (just as you would with bilinear)... Basically, the solution is to correctly calibrate your shot before you record. Otherwise, you'll either get a horizontal line pattern. Or gridding when you de-zipper. - ben

August 2nd, 2004, 10:31 PM	#1284
Jason Keenan New Boot Join Date: Jul 2004 Posts: 19	Another couple of stupid questions 1) In the '10 bit' raw bayer file, does that mean that in each pixel, there are 1024 shades of grey? 2) Anyone care to give a breif explaination of how these pixel values are represented in the file??? Raavin :)

August 2nd, 2004, 10:41 PM	#1285
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	Raavin, yeah -- 10 bit represents 1024 values. You can always determine the number of values by doing 2^x Where x = the number of bits. So 2^10 == 1024 I don't know what you mean by "represented in the file," but any native 10-bit format will do bit packing. That is, since a byte is 8bits, and since the byte is how we write to files, it means that uncompressed 10 bit would be written like this. I'll use "1" to represent the first 10-bit value, "2" to represent the second 10-bit value, etc. Each group of 8 numbers is a byte written to the file. 11111111____11222222____22223333____33333344____44444444 etc That means it won't make much sense in a hex viewer. However, one extremely inefficient way to store 10bit data is to pad it to 2 bytes (16bit). That would look like this: 0000001111111111___0000002222222222___0000003333333333___0000004444444444 As you can see, we're using far more bits/bytes to represent the same data. - ben

August 2nd, 2004, 11:22 PM	#1286
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	<<<-- Originally posted by Ben Syverson : But I'd rather not work directly on Rob's app, as I think he's focusing on a PC-based solution, whereas everything I do is on the Mac. Also, I think Rob's sw is basically capture software specifically for CameraLink @Wanye: the fastest codec (huffy??) A 10 or 12 bit JPEG/Wavelet would be fine for 99.9999999999% of the people on this board, but it would be slower than lossless, so what's the point? - ben -->>> Only mentioned it because those two are popular topics here and support the range of compression ratios that people want to see. I think it was Jaun that spoke about a Bayer version of Huff. The wavelet stuff is not the fastest but one of the best people like here, and able to do completely lossless. Rob is trying to do a Universal version of capture software that works with any codec, on the PC first then MAC/Linux. I think he intends to support USB and Gigabit versions of cameralink cameras, but I am unsure. So the codec and capture side would still be seperate. Wayne.

August 2nd, 2004, 11:23 PM	#1287
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	<<<-- Originally posted by Jason Keenan : Wayne: confused with the other Jason. I've seen cameras cheaper from www.videoguys.com.au. They have a good selection too. -->>> Thanks.

August 2nd, 2004, 11:35 PM	#1288
Jason Keenan New Boot Join Date: Jul 2004 Posts: 19	Ok, I think I've got it. So, in your example- "11111111____11222222____22223333____33333344____44444444" -if we had 4 pixels which were alternating white and black, it would be 11111111____11000000____00001111____11111100____00000000 yeah??? Would this then show in a hex editor as ff C0 0f fc 00 ?????? if it was true 10 bit and not padded to 16 bit???? Has it been ascertained whether the bin captures are padded to 16 bit or not. Every other byte has a leading zero so if it's the 'rawest' format available I'm guessing it is. eg "00 00 90 05 ac 06 18 05 54 06 38 06 28 07 28" from raw_cap.bin Thanks for the answers, I'm just trying to get my head around it. Cheers, Raavin

August 2nd, 2004, 11:48 PM	#1289
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	@Raavin: Would this then show in a hex editor as ff C0 0f fc 00 ?????? if it was true 10 bit and not padded to 16 bit???? Yes, precisely. As Rob clarified, the 10-bit data coming off the SI cameras is padded to 16bits in the following manner: (this would represent 1 value) 0000##########00 Where the hashes represent bits with data. So it's 10-bits of data, padded out to 12-bits, within a 16bit number. It seems very strange to me. But yes, that's why every pixel will look like: 0X XX in hex, because the first four bits are always 0. And the last two bits are always zeroes, so the last hex value of every pixel will always be C, D, E or F. But who cares about hex? :) - ben

August 3rd, 2004, 12:16 AM	#1290
Edgar Rodriguez Tourist Join Date: Apr 2004 Posts: 1	Hi, this is my first post but I've been following this thread since the start. Congratulations everyone for the great work!!! -Avid has a codec called DNxHD http://www.avid.com/DNxHD/ 6:1 full raster (sampling every pixel on the image) It works for 8-bit and 10-bit at 4:2:2 this will give us uncompressed quality at almost uncompressed SD size I think this realy would be the best codec to go for, with this project.