4:4:4 10bit single CMOS HD project

Les Dit · August 2nd, 2004, 01:43 AM

Ben, what are you basing this frame rate estimate on? How long does the software package the still photo guys use to demosaik take? I forget the name of the app, but it's for doing work on still camera RAW images.
-Les

<<<-- Originally posted by Ben Syverson : You could process footage way, way, way faster than 1fps on reasonably decent hardware. Even if you're doing pretty sophisticated de-Bayering. Especially if the code is optimized (possibly with vectorization/SIMD).

Totally unoptimized, it would probably run at about 2-3fps. Vectorized, your main bottleneck becomes the drive speed. I'm sure you could get between 10 and 15fps. -->>>

Juan M. M. Fiebelkorn · August 2nd, 2004, 01:55 AM

Well, a long time ago I made a resizing filter for Avisynth that had a speed of around 18 fps on an Athlon 2000 for a resulting image of 1600x1200.Obviously it worked on 8 bit depth images.
It was really un-optimized and very alpha with lots of comparisons, multiplications and divisions.It uses some techniques that are well suited for demosaicking (in fact I was upsampling color vectors based on Luma info, or Red and Blue based on Green for RGB).Maybe if I can get the time to code again I could release something in a month or two......

Ben Syverson · August 2nd, 2004, 01:59 AM

linBayer is totally unoptimized, and 100% single-thread -- ie, it's only using one of your processors, and using it extremely inefficiently. I won't optimize it until the code is a little more finalized.

Multithreading alone will give you an almost 100% speed increase, because it'll start using that second processor -- that would put you anywhere from 2fps to 2.4fps. Simple code optimization could probably give at least a 25% boost, to around 2.75 - 3

But linBayer is a prime candidate for vectorization, which could boost the speed even more, and for each processor. On a 2x2ghz G5, I wouldn't expect anything less than 8 - 10fps once it's fully optimized, and that's conservative.

The real bottleneck is disk speed, not processing speed. AE is not designed for fast disk reads -- rather than reading a bunch of frames into memory at a time, it determines file dependencies on a frame-by-frame basis. So. Every. Frame. Is. Read. In. Separately. That's necessary to keep AE flexible, but if you have a specialized need like we do, it's unecessary.

A well-coded standalone app could read in a bunch of frames at a time, churn through them with multithreaded and possibly vectorized code, and spit them to disk in sequence. That way, the drive gets to do a nice big sequential read (which is where you get your burst data transfer rates), your processor gets to work directly from RAM, and then it goes back to disk in a big fast sequential write.

Additionally, processing will probably be faster in general than in AE, because you won't have quite as much overhead associated with being a plug-in within a large app.

These are the reasons why 10 - 15fps is totally possible, given some nice disk throughput.

Add to this fact that MPEG-4 compresses in better than realtime on a G5, and you could probably write out a low-res editing proxy at the same time as your Bayer->RGB conversion with a small speed hit.

But the bigger question is, why convert Bayer->RGB this way? It seems to me that the data should be stored as Bayer information, and de-Bayered by a codec as it's played back. That way you get the storage benefits of compressed Bayer (lossless or lossy, either way), but you can see the image in RGB in all your apps. (Including AE, FCP, etc)

It makes absolutely no sense to store uncompressed 4:4:4 RGB versions of Bayer images on your hard drive, unless you just love eating up 3X the disk space and throughput for absolutely NO gains...

Rai Orz · August 2nd, 2004, 02:13 AM

Today i will order the Sumix SMX 150C, because we think the IBIS5 sensor is mutch better than others (in this price range = 2/3" and global shutter). Ben, how much did you pay for it, exactly? It seems, Sumix want to co-operate with us (software and hardware changes).
I know the bad software and the slow USB2.0, but this will be not a problem, we have other solutions so we will use the camera at 24fps with 10Bit and global shutter.

Global shutter was the reason for the Sumix decision. Because the SI rolling shutter is absolutely unfit. We made test with a german camera head (with same sensor characteristics). At 24fps, but also at 48fps, the rolling shutter artefacts destroys each moving picture.
The altasens will have also a rolling shutter, but mutch faster (not the whole fps-time). This produces mutch less artefacts. The altasens supported also a external shutter, so on this way the artefacts will disappear completely.

Ben Syverson · August 2nd, 2004, 02:19 AM

Rai,

I agree that 2/3" is totally key. The rolling shutter is not such a show-stopper for me, but the IBIS-5 can indeed do global shutter. What remains to be seen is whether the SMX-150c can deliver 24fps of global shutter of USB2. I have my doubts, but I'll be interested to hear how your experiments go.

If you email Sumix and mention my name, they should give you the same deal they gave me. They're very eager to work with filmmakers and figure out our needs.

I should note that they're taking a bit of a summer vacation right now, so you may not get an immediate response. I think they get back in full force in a week or two.

- ben

Wayne Morellini · August 2nd, 2004, 02:49 AM

I thought we were working together towards an eventual cheap solution, step by step? I think we are still going well, with more of the steps getting done. If anyone wants to jump the gun and experiment, or wants broadcast compliance, before then it is going to cost you more.

My 2 cents, we are fine, and if you read Rob's Developement log, he is making progress on speeding up the software (I did warn everybody that it wasn't easy to program for the best speed).

Juan and Ben, it is obviouse you want to do bayer and compression codecs, why don't you volunteer to work on them with Rob. He is allready doing capture software you could each volunteer to do bayer and compression? It maybe crazy economics, but I think it is worth it.

Jaun, you wrote about people shooting you down about jpeg2000, but I say go ahead great (but remember the BBC version). We are looking at universal codec support, but for now we could use a few codecs, the fastest codec (huffy??) and the best (wavelet based lossless, near lossless, and 20:1).

Everybody has different needs, all we need to do is support the best needs.

Ben you wrote a lot of wisdom there, unless we are passing to broadcast, display, some non-bayer NLE, or using 3 chips, there is no reason to debayer.

Rai, been sick in bed for days because of stomach bug, I will try to email you by the end of the day.

Somebody posted here (or one of the other threads, even photo camera to video mod threads) that there was a number of SD cameras you could catch "frames" from live through firewire. Which models I don't know but worth asking around about.

Thanks

Wayne.

http://www.cameraaction.com.au/, should be able to do a better price on an XL1S.

Jason Rodriguez · August 2nd, 2004, 04:45 AM

Unless your app captures directly to Quicktime Ben, you're going to have to convert from RAW to Quicktime, but maybe Rob's app could capture directly to QT, although dropped frames will be a new concern (ala FCP captures).

Keep in mind that QT on Windows (where the framegrabbers are) is horribly slow, much slower than the Mac.

Also for HD 1920x1080 I'm definitely NOT getting better than real-time for MPEG4 transcodes. On SD footage yes, but not HD-it could be disk-write speeds, but again, I'm back at the good 'ole 2-3fps for pure transcodes/resizing.

Rai Orz · August 2nd, 2004, 05:24 AM

<<<-- Originally posted by Ben Syverson: What remains to be seen is whether the SMX-150c can deliver 24fps of global shutter of USB2. I have my doubts, but I'll be interested to hear how your experiments go-->>>

Fist, we can change the hardware inside the camera. For example in the past we changed canon video cameras (up-side-down) to work with our 35mm solution, but without a prism. So we can go arround the USB2.0. But...

...second, read this part of a email from Sumix:
"...In the present SMX-150C pixels are sampled at 10 bit. Then a look up table (which can be arbitrarily programed) converts the 10bit to 8bit. In effect at present SMX-150 user has access to all 10 bit by choosing the look up table accordingly. However we have a new version (a software/firmware upgrade) that transfers directly 10 bit pixel data to PC. This upgrade will be ready in 2-3 weeks. In addition this version has the option of using the multi-gain (muti-slope) sampling features in the IBIS5 sensor used in the cameras. This muti-sampling at different gains provides defective 12 bit pixels (compressed in 10 bits for transfer to PC.) This upgrade will be free of charge for people who have purchased the present SMX-150C...
...Both global and rolling shutters can be used for video streaming..."

Juan M. M. Fiebelkorn · August 2nd, 2004, 05:28 AM

I would like to see how the dual slope operation will look like..... :)

Defective or Efective??

I'm still thinking they should give us an SDK.It would be good both for us and for them....

Rob Scott · August 2nd, 2004, 05:43 AM

Quote:

Jason Rodriguez wrote:
Unless your app captures directly to Quicktime Ben, you're going to have to convert from RAW to Quicktime, but maybe Rob's app could capture directly to QT, although dropped frames will be a new concern (ala FCP captures).

I suppose that is possible, but my plan right now is to write to a very simple raw format for speed. QuickTime conversion will be done offline.

Quote:

Ben Syverson wrote:
It seems to me that the data should be stored as Bayer information, and de-Bayered by a codec as it's played back.

If we had such an on-the-fly Bayer codec, the offline conversion would be very fast; it would just rewrite the video stream into QuickTime format without any further processing.

Mostly, though, I expect the conversion stage to be used to Bayer-process the footage, apply gamma correction (etc.) and then write to QuickTime using a near-lossless codec for a reasonable compress ratio.

Rai Orz · August 2nd, 2004, 06:25 AM

Write in a very simple RAW format is the key.

I dont look for other ways, because all other ways will "change" the image-information (differnt decoder = different bayer-pixel-resolution). If you store the original RAW format, you can select the best bayer decoder for your images later.

I think a real-time decoder is need only for the viewfinder or camera-display. But maybe you can live with the B&W RAW format.

Juan M. M. Fiebelkorn · August 2nd, 2004, 07:02 AM

what's the problem of seeing a 1280x720 camera at 640x360 in color on the viewfinder??
Anyway you would need to view a MODIFIED version of the RAW image on the Viewfinder, unless you have some kind of gamma correction suitable for this kind of data inside your Brain...

I'm getting lost again.....

And , What's the problem of LOSSLESS transformations on the RAW Bayer? (LOSSLESS means you always have the EXACT original data)

Ben Syverson · August 2nd, 2004, 09:38 AM

@Jason: Unless your app captures directly to Quicktime Ben

The Sumix app can capture directly to AVI, a frame sequence, or a "SMX" video stream, which is basically a stream of raw data. The easiest solution is to generate a grayscale Bayer AVI, and convert that to Quicktime. I guess I don't understand what the problem with that is.

@Jason: Keep in mind that QT on Windows (where the framegrabbers are) is horribly slow, much slower than the Mac.

Doesn't matter much to me -- my PC laptop is just a capture slave. Everything else in my pipeline happens on a Mac.

@Jason: Also for HD 1920x1080 I'm definitely NOT getting better than real-time for MPEG4 transcodes.

I was talking about SD (low res) proxy footage for editing purposes. Just a throwaway thought. It's just one possibility...

@Wayne: Juan and Ben, it is obviouse you want to do bayer and compression codecs, why don't you volunteer to work on them with Rob.

I already have -- I have an open invitation to anyone working on an free or open source integrated solution to use my de-Bayering code. But it's just as well -- I'd like to get it in better shape before I let anyone see it.

But I'd rather not work directly on Rob's app, as I think he's focusing on a PC-based solution, whereas everything I do is on the Mac. Also, I think Rob's sw is basically capture software specifically for CameraLink (Rob, correct me if I'm wrong), which doesn't apply to my camera. It may make sense to share certain techniques and code snippets, but if I decide to go forward with a Quicktime codec, a lot of that code will be fairly specific.

I don't believe Rob is working on a codec, but maybe he has plans for one...

@Wanye: the fastest codec (huffy??)

Wayne, remember that HuffYUV is... YUV. To get YUV from Bayer, you have to do Bayer->RGB->YUV. And remember that YUV @ 4:2:2 takes up TWICE as much space as Bayer. There's no reason to use HuffYUV.

I'm not sure how well Huffman encoding would work on raw Bayer channels (ie, 1/2 size G, 1/4 size R+B). HuffYUV probably works so well because U and V are mostly gray....

A 10 or 12 bit JPEG/Wavelet would be fine for 99.9999999999% of the people on this board, but it would be slower than lossless, so what's the point?

The single best option in my eyes is RLE/Huff on Bayer data, inside a codec that displays RGB for you. That way, you get the 15-20MB/sec file size, and you never have to worry about de-Bayering, because the codec does it for you. That would be the way to go, and it's what I'm currently investigating via a Quicktime Component. If anyone wants to help, they're more than welcome to.

- ben

Steve Nordhauser · August 2nd, 2004, 10:16 AM

At the risk of supporting my competitors' sales, here is an application note on dual slope with the IBIS-5:
http://www.siliconimaging.com/Specifications/Dual-Slope%20Ap%20Note%20001.pdf

Here is another test:
I just did a quick experiment with the dual slope capabilities of the SI-1280F using XCAP. I placed a 20W halogen in a fairly dark scene, pointing directly at the camera.

(all files are 1.3MB Tiff files)
http://www.siliconimaging.com/Sample...osec%20exp.tif
http://www.siliconimaging.com/Sample...msec%20exp.tif
Then I turned on the dual slope:
http://www.siliconimaging.com/Sample...ight%20exp.tif

I didn't spend a lot of time tweaking this but you can see that the details of the lamp are there along with the background image in the dual slope capture. I'm sure it would be possible to improve the contrast of the dark image, but this will give you an idea of what can be done.

Having provided this public service, I'll drop in a commercial. If you find that when you move to 10 bit and global shutter, USB 2.0 can't hack it anymore, think about our camera link version. It is true 12 bit and clocks to 60MHz (important in global shutter mode). It is still an IBIS-5 so it comes with all the other baggage (noise and washed out colors), but that is sensor level stuff.

Ben Syverson · August 2nd, 2004, 10:19 AM

Woah -- Steve, what was your gain set to? I've never seen footage that noisy unless my gain was set through the roof...

August 2nd, 2004, 10:16 AM	#1274
Steve Nordhauser Silicon Imaging, Inc. Join Date: May 2004 Location: Troy, NY USA Posts: 325	Dual slope on IBIS-5 At the risk of supporting my competitors' sales, here is an application note on dual slope with the IBIS-5: http://www.siliconimaging.com/Specifications/Dual-Slope%20Ap%20Note%20001.pdf Here is another test: I just did a quick experiment with the dual slope capabilities of the SI-1280F using XCAP. I placed a 20W halogen in a fairly dark scene, pointing directly at the camera. (all files are 1.3MB Tiff files) http://www.siliconimaging.com/Sample...osec%20exp.tif http://www.siliconimaging.com/Sample...msec%20exp.tif Then I turned on the dual slope: http://www.siliconimaging.com/Sample...ight%20exp.tif I didn't spend a lot of time tweaking this but you can see that the details of the lamp are there along with the background image in the dual slope capture. I'm sure it would be possible to improve the contrast of the dark image, but this will give you an idea of what can be done. Having provided this public service, I'll drop in a commercial. If you find that when you move to 10 bit and global shutter, USB 2.0 can't hack it anymore, think about our camera link version. It is true 12 bit and clocks to 60MHz (important in global shutter mode). It is still an IBIS-5 so it comes with all the other baggage (noise and washed out colors), but that is sensor level stuff. __________________ Silicon Imaging, Inc. We see the Light! http://www.siliconimaging.com

August 2nd, 2004, 01:43 AM	#1261
Les Dit Major Player Join Date: Oct 2003 Location: Southern Cal-ee-for-Ni-ya Posts: 608	Ben, what are you basing this frame rate estimate on? How long does the software package the still photo guys use to demosaik take? I forget the name of the app, but it's for doing work on still camera RAW images. -Les <<<-- Originally posted by Ben Syverson : You could process footage way, way, way faster than 1fps on reasonably decent hardware. Even if you're doing pretty sophisticated de-Bayering. Especially if the code is optimized (possibly with vectorization/SIMD). Totally unoptimized, it would probably run at about 2-3fps. Vectorized, your main bottleneck becomes the drive speed. I'm sure you could get between 10 and 15fps. -->>>

August 2nd, 2004, 01:55 AM	#1262
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	Well, a long time ago I made a resizing filter for Avisynth that had a speed of around 18 fps on an Athlon 2000 for a resulting image of 1600x1200.Obviously it worked on 8 bit depth images. It was really un-optimized and very alpha with lots of comparisons, multiplications and divisions.It uses some techniques that are well suited for demosaicking (in fact I was upsampling color vectors based on Luma info, or Red and Blue based on Green for RGB).Maybe if I can get the time to code again I could release something in a month or two......

August 2nd, 2004, 01:59 AM	#1263
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	linBayer is totally unoptimized, and 100% single-thread -- ie, it's only using one of your processors, and using it extremely inefficiently. I won't optimize it until the code is a little more finalized. Multithreading alone will give you an almost 100% speed increase, because it'll start using that second processor -- that would put you anywhere from 2fps to 2.4fps. Simple code optimization could probably give at least a 25% boost, to around 2.75 - 3 But linBayer is a prime candidate for vectorization, which could boost the speed even more, and for each processor. On a 2x2ghz G5, I wouldn't expect anything less than 8 - 10fps once it's fully optimized, and that's conservative. The real bottleneck is disk speed, not processing speed. AE is not designed for fast disk reads -- rather than reading a bunch of frames into memory at a time, it determines file dependencies on a frame-by-frame basis. So. Every. Frame. Is. Read. In. Separately. That's necessary to keep AE flexible, but if you have a specialized need like we do, it's unecessary. A well-coded standalone app could read in a bunch of frames at a time, churn through them with multithreaded and possibly vectorized code, and spit them to disk in sequence. That way, the drive gets to do a nice big sequential read (which is where you get your burst data transfer rates), your processor gets to work directly from RAM, and then it goes back to disk in a big fast sequential write. Additionally, processing will probably be faster in general than in AE, because you won't have quite as much overhead associated with being a plug-in within a large app. These are the reasons why 10 - 15fps is totally possible, given some nice disk throughput. Add to this fact that MPEG-4 compresses in better than realtime on a G5, and you could probably write out a low-res editing proxy at the same time as your Bayer->RGB conversion with a small speed hit. But the bigger question is, why convert Bayer->RGB this way? It seems to me that the data should be stored as Bayer information, and de-Bayered by a codec as it's played back. That way you get the storage benefits of compressed Bayer (lossless or lossy, either way), but you can see the image in RGB in all your apps. (Including AE, FCP, etc) It makes absolutely no sense to store uncompressed 4:4:4 RGB versions of Bayer images on your hard drive, unless you just love eating up 3X the disk space and throughput for absolutely NO gains...

August 2nd, 2004, 02:13 AM	#1264
Rai Orz Regular Crew Join Date: Jun 2004 Location: Germany Posts: 136	Today i will order the Sumix SMX 150C, because we think the IBIS5 sensor is mutch better than others (in this price range = 2/3" and global shutter). Ben, how much did you pay for it, exactly? It seems, Sumix want to co-operate with us (software and hardware changes). I know the bad software and the slow USB2.0, but this will be not a problem, we have other solutions so we will use the camera at 24fps with 10Bit and global shutter. Global shutter was the reason for the Sumix decision. Because the SI rolling shutter is absolutely unfit. We made test with a german camera head (with same sensor characteristics). At 24fps, but also at 48fps, the rolling shutter artefacts destroys each moving picture. The altasens will have also a rolling shutter, but mutch faster (not the whole fps-time). This produces mutch less artefacts. The altasens supported also a external shutter, so on this way the artefacts will disappear completely.

August 2nd, 2004, 02:19 AM	#1265
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	Rai, I agree that 2/3" is totally key. The rolling shutter is not such a show-stopper for me, but the IBIS-5 can indeed do global shutter. What remains to be seen is whether the SMX-150c can deliver 24fps of global shutter of USB2. I have my doubts, but I'll be interested to hear how your experiments go. If you email Sumix and mention my name, they should give you the same deal they gave me. They're very eager to work with filmmakers and figure out our needs. I should note that they're taking a bit of a summer vacation right now, so you may not get an immediate response. I think they get back in full force in a week or two. - ben

August 2nd, 2004, 02:49 AM	#1266
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	I thought we were working together towards an eventual cheap solution, step by step? I think we are still going well, with more of the steps getting done. If anyone wants to jump the gun and experiment, or wants broadcast compliance, before then it is going to cost you more. My 2 cents, we are fine, and if you read Rob's Developement log, he is making progress on speeding up the software (I did warn everybody that it wasn't easy to program for the best speed). Juan and Ben, it is obviouse you want to do bayer and compression codecs, why don't you volunteer to work on them with Rob. He is allready doing capture software you could each volunteer to do bayer and compression? It maybe crazy economics, but I think it is worth it. Jaun, you wrote about people shooting you down about jpeg2000, but I say go ahead great (but remember the BBC version). We are looking at universal codec support, but for now we could use a few codecs, the fastest codec (huffy??) and the best (wavelet based lossless, near lossless, and 20:1). Everybody has different needs, all we need to do is support the best needs. Ben you wrote a lot of wisdom there, unless we are passing to broadcast, display, some non-bayer NLE, or using 3 chips, there is no reason to debayer. Rai, been sick in bed for days because of stomach bug, I will try to email you by the end of the day. Somebody posted here (or one of the other threads, even photo camera to video mod threads) that there was a number of SD cameras you could catch "frames" from live through firewire. Which models I don't know but worth asking around about. Thanks Wayne. http://www.cameraaction.com.au/, should be able to do a better price on an XL1S.

August 2nd, 2004, 04:45 AM	#1267
Jason Rodriguez Trustee Join Date: Mar 2003 Location: Virginia Beach, VA Posts: 1,095	Unless your app captures directly to Quicktime Ben, you're going to have to convert from RAW to Quicktime, but maybe Rob's app could capture directly to QT, although dropped frames will be a new concern (ala FCP captures). Keep in mind that QT on Windows (where the framegrabbers are) is horribly slow, much slower than the Mac. Also for HD 1920x1080 I'm definitely NOT getting better than real-time for MPEG4 transcodes. On SD footage yes, but not HD-it could be disk-write speeds, but again, I'm back at the good 'ole 2-3fps for pure transcodes/resizing.

August 2nd, 2004, 05:24 AM	#1268
Rai Orz Regular Crew Join Date: Jun 2004 Location: Germany Posts: 136	<<<-- Originally posted by Ben Syverson: What remains to be seen is whether the SMX-150c can deliver 24fps of global shutter of USB2. I have my doubts, but I'll be interested to hear how your experiments go-->>> Fist, we can change the hardware inside the camera. For example in the past we changed canon video cameras (up-side-down) to work with our 35mm solution, but without a prism. So we can go arround the USB2.0. But... ...second, read this part of a email from Sumix: "...In the present SMX-150C pixels are sampled at 10 bit. Then a look up table (which can be arbitrarily programed) converts the 10bit to 8bit. In effect at present SMX-150 user has access to all 10 bit by choosing the look up table accordingly. However we have a new version (a software/firmware upgrade) that transfers directly 10 bit pixel data to PC. This upgrade will be ready in 2-3 weeks. In addition this version has the option of using the multi-gain (muti-slope) sampling features in the IBIS5 sensor used in the cameras. This muti-sampling at different gains provides defective 12 bit pixels (compressed in 10 bits for transfer to PC.) This upgrade will be free of charge for people who have purchased the present SMX-150C... ...Both global and rolling shutters can be used for video streaming..."

August 2nd, 2004, 05:28 AM	#1269
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	I would like to see how the dual slope operation will look like..... :) Defective or Efective?? I'm still thinking they should give us an SDK.It would be good both for us and for them....

August 2nd, 2004, 06:25 AM	#1271
Rai Orz Regular Crew Join Date: Jun 2004 Location: Germany Posts: 136	Write in a very simple RAW format is the key. I dont look for other ways, because all other ways will "change" the image-information (differnt decoder = different bayer-pixel-resolution). If you store the original RAW format, you can select the best bayer decoder for your images later. I think a real-time decoder is need only for the viewfinder or camera-display. But maybe you can live with the B&W RAW format.

August 2nd, 2004, 07:02 AM	#1272
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	what's the problem of seeing a 1280x720 camera at 640x360 in color on the viewfinder?? Anyway you would need to view a MODIFIED version of the RAW image on the Viewfinder, unless you have some kind of gamma correction suitable for this kind of data inside your Brain... I'm getting lost again..... And , What's the problem of LOSSLESS transformations on the RAW Bayer? (LOSSLESS means you always have the EXACT original data)

August 2nd, 2004, 09:38 AM	#1273
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	@Jason: Unless your app captures directly to Quicktime Ben The Sumix app can capture directly to AVI, a frame sequence, or a "SMX" video stream, which is basically a stream of raw data. The easiest solution is to generate a grayscale Bayer AVI, and convert that to Quicktime. I guess I don't understand what the problem with that is. @Jason: Keep in mind that QT on Windows (where the framegrabbers are) is horribly slow, much slower than the Mac. Doesn't matter much to me -- my PC laptop is just a capture slave. Everything else in my pipeline happens on a Mac. @Jason: Also for HD 1920x1080 I'm definitely NOT getting better than real-time for MPEG4 transcodes. I was talking about SD (low res) proxy footage for editing purposes. Just a throwaway thought. It's just one possibility... @Wayne: Juan and Ben, it is obviouse you want to do bayer and compression codecs, why don't you volunteer to work on them with Rob. I already have -- I have an open invitation to anyone working on an free or open source integrated solution to use my de-Bayering code. But it's just as well -- I'd like to get it in better shape before I let anyone see it. But I'd rather not work directly on Rob's app, as I think he's focusing on a PC-based solution, whereas everything I do is on the Mac. Also, I think Rob's sw is basically capture software specifically for CameraLink (Rob, correct me if I'm wrong), which doesn't apply to my camera. It may make sense to share certain techniques and code snippets, but if I decide to go forward with a Quicktime codec, a lot of that code will be fairly specific. I don't believe Rob is working on a codec, but maybe he has plans for one... @Wanye: the fastest codec (huffy??) Wayne, remember that HuffYUV is... YUV. To get YUV from Bayer, you have to do Bayer->RGB->YUV. And remember that YUV @ 4:2:2 takes up TWICE as much space as Bayer. There's no reason to use HuffYUV. I'm not sure how well Huffman encoding would work on raw Bayer channels (ie, 1/2 size G, 1/4 size R+B). HuffYUV probably works so well because U and V are mostly gray.... A 10 or 12 bit JPEG/Wavelet would be fine for 99.9999999999% of the people on this board, but it would be slower than lossless, so what's the point? The single best option in my eyes is RLE/Huff on Bayer data, inside a codec that displays RGB for you. That way, you get the 15-20MB/sec file size, and you never have to worry about de-Bayering, because the codec does it for you. That would be the way to go, and it's what I'm currently investigating via a Quicktime Component. If anyone wants to help, they're more than welcome to. - ben

August 2nd, 2004, 10:19 AM	#1275
Ben Syverson Major Player Join Date: Mar 2004 Location: chicago Posts: 434	Woah -- Steve, what was your gain set to? I've never seen footage that noisy unless my gain was set through the roof...