Mirage Recorder at DVinfo.net

Take Vos · March 24th, 2007, 07:56 PM

Hello everyone,

This is the thread about my new Mac OS X software that I am writing to record 16 bit raw bayer data from an IIDC camera. In this case I am planning to use the Pike 210C from www.alliedvisiontec.com.

At this moment I can already see a live image from a stand-in camera, the Fire-i from unibrain, it does not do bayer but I wanted to start cheap.

I use libdc1394 for the camera feed. The 8 bit images are converted to 32 bit floating point using the vImage (accelerated image processing) library. These 32 bit images are then send to OpenGL as a luminance texture map.

I am converting to 32 bit floating point images as OpenGL is extremely slow when working with 16 bit integers. There are also more accelerated functions that can work on 32 bit floating point number than there are for 16 bit integer.

I am now planning on to compress the images before they will be added to the ring buffer. Compressing the images before hand would allow for more room in the ring buffer.

I have not yet decided on the compression algorithm to use. I will probably make a prediction algorithm that uses left and top neighbor pixels. The error values (real - predicted) are then converted using a sort of huffman code.

I have this weird idea of creating a set of static huffman dictionaries based on normal distributions. For each line or block of pixels the average error value is calculated and this average will select the dictionary to use for compression. It would be better to calculate the median, but an average is much faster to calculate (doesn't need a sort operation).

Cheers,
Take Vos
developer of Boom Recorder

Take Vos · March 25th, 2007, 04:42 PM

Hello everyone,

I just designed and implemented the compression algorithm.
It first predicts the value of a pixel by taking a sort of average of its four neighbor pixels, of the same color, positioned above and to its left.

The actual value is subtracted from the predicted value. When there is not to much high resolution contrast in the image the error values should on average be low.

These values are than written down using the exponential-golomb encoding, this encoding writes down small values in less bits than high values. With my 640x480 camera with lots of noise I mostly get a compression ratio between 1:1 and 1:2.

I've decided against implementing most of the compression pre-processing into OpenGL and write it in straight C. The amount of CPU time used is still within limits, but tight.

Cheers,
Take

Take Vos · March 29th, 2007, 02:29 PM

Hi again.

I've made the compression algorithm simpler, it now just looks at the pixel two positions (because of bayer) to the left of the current pixel and subtract. I still use exponential-golomb encoding. The cpu usage for 640x480@15fps is now between 10% and 11% for the conversion from RGB to Bayer (for testing with a RGB camera), compression and displaying the video.

I have just created a fragment program in OpenGL that does realtime demosaic with linear interpolation. The linear interpolation is not only used for debayering, but also for getting a smooth image when the image is displayed on the screen using a non-integer zoom factor.

Next step, recording.

Cheers,
Take

Wayne Morellini · March 30th, 2007, 03:41 AM

Congradulations, are there any HD webcams for the mac this would work mon. If you are using the portable or mini, the Intel chipset emaulates many GPU fucntions, next chipset (GM965) hopefully should not.

Good luck.

Take Vos · March 30th, 2007, 04:52 AM

Hello Wayne,

Thank you, I am pretty happy with the results right now.

The software now is specifically written for the Fire-i camera, which is a 640x480 RGB webcam using the IIDC Firewire protocol.

An HD camera that uses IIDC needs a lot of bandwidth and a Firewire 800 controller is required, which currently is only available on the MacBook Pro.
The sustained transfer rate required for the disks is around 60 MB/sec, this can not be done using a Firewire interface as it is already in use by the camera. Thus a eSATA ExpressCard is need, ExpressCard is only available on the MacBook Pro.

Cheers,
Take

Juan M. M. Fiebelkorn · April 2nd, 2007, 03:17 PM

It sounds to me quite a lot simillar to just how Huffyuv compresses video.

Take Vos · April 2nd, 2007, 03:22 PM

Indeed, it isn't exactly rocket science.

I am changing my mind slightly though.
To make it myself easy I am thinking of actually letting my recorder write OpenEXR files directly, probably without compression :-(

Cheers,
Take

Take Vos · April 8th, 2007, 12:30 PM

Good and bad news.

I've been able to record the images to disk in OpenEXR format. Bad news, OpenEXR is quite slow, for something that should be a simple copy operation it uses a tad to much CPU power.

So it is back to the exponential golomb coding and writing my own type of file.

Cheers,
Take

Take Vos · April 11th, 2007, 11:27 AM

Hello everyone,

The recorder is working, it is creating my own digital negative format.
I will publish specifications on this file format at release to source code to read and write these files.

Of course I can not yet actually see if it is recording correctly, that is my next task.

Cheers,
Take

Solomon Chase · April 11th, 2007, 07:08 PM

Cool, keep posting updates :)

Take Vos · April 25th, 2007, 12:36 PM

Hello everyone,

Another update, I have ordered the Pike 210C as I am really interested to see if it will all work.

Also I will build the video recorder into Boom Recorder instead of a separate program. This will allow you to enter metadata, such as scene and take numbers. It will also allow timecode signal via an audio input and a lot more features.

Cheers,
Take

Take Vos · May 3rd, 2007, 05:40 PM

I received the camera, it took me some time to get a picture from it, but it is working now. I currently have it set up to send quite a large 2.35:1 picture at 14 bits depth.

It seems that the camera by default does not do any compensation as I hoped it would not, so the picture is quite a mess. It should not be much of a problem to get all that sensor unevenness removed.

My current problem is that I had an old camera with a 1/3" CS mount lens, and I am using that lens on this camera. That does not make for a sharp picture without vignetting (although the inside of the lens and the three diaphragm blades are in focus).

As for computer performance, I found out that my compression routine was using quite a lot of CPU time and it did not actually compress the data much (actually it became larger). So instead of Delta-Golomb compression it now simply packs the 14 bits tightly.

Next step is finding a good lens and start working on writing to disk.

Cheers,
Take

Take Vos · May 8th, 2007, 12:46 PM

An other update.

Boom Recorder can now write the sensor data to disk, I just did a full scale test: 1800 x 750 (1:2.40 ratio), 16bits @ 24fps. I recorded for 9 minutes onto an external SATA disk, smooth as an android's bottom.

I only configured a video ring buffer of 128 MByte (I should put more memory in my computer), but it kept up rather well as the maximum was around 50% filled.

It was also pretty light on the CPU, the computer did not even turn on the fans.

Next step: QuickTime component so that you can natively edit the footage on Final Cut Pro, it must do real-time debayer and compensate for fixed sensor noise. I also have to make an intermediate encoder, so that you can render the footage in Final Cut Pro to a 16 bits depth.

QuickTime does have 16 bit depth capability, but it is not linear, rather it uses a gamma correction of 1.8. I guess Final Cut Pro converts it to linear floating point before doing any calculations. To bad QuickTime does not support linear floating point pixel types.

Cheers,
Take

Take Vos · May 11th, 2007, 06:53 AM

Hello everyone,

I finally have a decent lens, it is an zoom lens from an old Minolta 35mm photo camera. I had to do some back focus adjustments (the pike supports this by rotating the C-mount in its body-mount) to get it to focus through zoom. Strangely enough the focus marks on the lens are off.

It zooms between 35 and 70 mm and the iris is between f1/22 - f1/4.5, so it is rather slow.

Anyway I made three screenshots of my application. The pictures are not corrected for sensor anomalies or color. the only corrections applied are demosaic using a simple linear algorithm, gamma correction of 1.8 and resized to the smaller screen size.

The first image is the park from my open window, it was raining outside. f1/11, 35mm, infinite focus.
http://www.vosgames.nl/products/MirageRecorder/park.png

The second image is of some bicycles in the park, same settings.
http://www.vosgames.nl/products/Mira...er/bicycle.png

And everyone likes to see a depth of field shot. F1/4.5, 35mm, focused on the blue perfume sampler around 1 meter away. The green perfume sample on the left is 20 cm nearer and a mobile phone is 20 cm nearer than that. On the right the orange lens converter box is 20 cm further and the small C/S mount lens on the far right is 20 cm further than that.
The background wallpaper is around 3 meters from the lens. The photo camera bag is about 1.5 meters from the lens.
http://www.vosgames.nl/products/MirageRecorder/DOF.png

Cheers,
Take

Take Vos · May 27th, 2007, 01:02 AM

Hi there,

I have another update for you. The QuickTime component is working, it can read the raw digital negative and demosaic it during playing. It asks for a 16 bit per-component drawable, so with a bit of luck Final Cut Pro can use all the 14 bits from the camera.

I found in the Final Cut Pro manual that you only get fast (extreme) performance when the timeline uses the same codec and codec-settings as the clips that are imported. So I will make this QuickTime component to also be used on the timeline. Footage that needs to be coded by Final Cut Pro will use 48 bit RGB and not bayer, in other word it will silently use other settings without Final Cut Pro knowing about it.

At this time the QuickTime component is rather slow as all the demosaicing and other processing is done in software. The plan is to use OpenGL for processing the footage before it enters Final Cut Pro. With a bit of luck you will be able to edit the footage in real-time.

But the next programming challenge is an application that can make calibration data to be used by the QuickTime component and Mirage Recorder. The calibration data will consist of a dark-image offset and slope (based on the exposure time), and a light image offset and slope, and lastly a pixel repair list.

light-image calibration image can be done with any "white"-light that has enough intensity in each red, green and blue component, as long as the same light is smeared equally over the whole sensor. Some kind of diffuser would be handy, or take the lens off the camera. As the calibration is color agnostic, I can use the color conversion matrix for this camera without change. White balance is then the job of Final Cut Pro (or Color if that works).

March 24th, 2007, 07:56 PM	#1
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	Mirage Recorder Hello everyone, This is the thread about my new Mac OS X software that I am writing to record 16 bit raw bayer data from an IIDC camera. In this case I am planning to use the Pike 210C from www.alliedvisiontec.com. At this moment I can already see a live image from a stand-in camera, the Fire-i from unibrain, it does not do bayer but I wanted to start cheap. I use libdc1394 for the camera feed. The 8 bit images are converted to 32 bit floating point using the vImage (accelerated image processing) library. These 32 bit images are then send to OpenGL as a luminance texture map. I am converting to 32 bit floating point images as OpenGL is extremely slow when working with 16 bit integers. There are also more accelerated functions that can work on 32 bit floating point number than there are for 16 bit integer. I am now planning on to compress the images before they will be added to the ring buffer. Compressing the images before hand would allow for more room in the ring buffer. I have not yet decided on the compression algorithm to use. I will probably make a prediction algorithm that uses left and top neighbor pixels. The error values (real - predicted) are then converted using a sort of huffman code. I have this weird idea of creating a set of static huffman dictionaries based on normal distributions. For each line or block of pixels the average error value is calculated and this average will select the dictionary to use for compression. It would be better to calculate the median, but an average is much faster to calculate (doesn't need a sort operation). Cheers, Take Vos developer of Boom Recorder

April 25th, 2007, 12:36 PM	#11
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	Boom Recorder Hello everyone, Another update, I have ordered the Pike 210C as I am really interested to see if it will all work. Also I will build the video recorder into Boom Recorder instead of a separate program. This will allow you to enter metadata, such as scene and take numbers. It will also allow timecode signal via an audio input and a lot more features. Cheers, Take

May 11th, 2007, 06:53 AM	#14
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	First light with pictures Hello everyone, I finally have a decent lens, it is an zoom lens from an old Minolta 35mm photo camera. I had to do some back focus adjustments (the pike supports this by rotating the C-mount in its body-mount) to get it to focus through zoom. Strangely enough the focus marks on the lens are off. It zooms between 35 and 70 mm and the iris is between f1/22 - f1/4.5, so it is rather slow. Anyway I made three screenshots of my application. The pictures are not corrected for sensor anomalies or color. the only corrections applied are demosaic using a simple linear algorithm, gamma correction of 1.8 and resized to the smaller screen size. The first image is the park from my open window, it was raining outside. f1/11, 35mm, infinite focus. http://www.vosgames.nl/products/MirageRecorder/park.png The second image is of some bicycles in the park, same settings. http://www.vosgames.nl/products/Mira...er/bicycle.png And everyone likes to see a depth of field shot. F1/4.5, 35mm, focused on the blue perfume sampler around 1 meter away. The green perfume sample on the left is 20 cm nearer and a mobile phone is 20 cm nearer than that. On the right the orange lens converter box is 20 cm further and the small C/S mount lens on the far right is 20 cm further than that. The background wallpaper is around 3 meters from the lens. The photo camera bag is about 1.5 meters from the lens. http://www.vosgames.nl/products/MirageRecorder/DOF.png Cheers, Take

March 25th, 2007, 04:42 PM	#2
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	Hello everyone, I just designed and implemented the compression algorithm. It first predicts the value of a pixel by taking a sort of average of its four neighbor pixels, of the same color, positioned above and to its left. The actual value is subtracted from the predicted value. When there is not to much high resolution contrast in the image the error values should on average be low. These values are than written down using the exponential-golomb encoding, this encoding writes down small values in less bits than high values. With my 640x480 camera with lots of noise I mostly get a compression ratio between 1:1 and 1:2. I've decided against implementing most of the compression pre-processing into OpenGL and write it in straight C. The amount of CPU time used is still within limits, but tight. Cheers, Take

March 29th, 2007, 02:29 PM	#3
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	Hi again. I've made the compression algorithm simpler, it now just looks at the pixel two positions (because of bayer) to the left of the current pixel and subtract. I still use exponential-golomb encoding. The cpu usage for 640x480@15fps is now between 10% and 11% for the conversion from RGB to Bayer (for testing with a RGB camera), compression and displaying the video. I have just created a fragment program in OpenGL that does realtime demosaic with linear interpolation. The linear interpolation is not only used for debayering, but also for getting a smooth image when the image is displayed on the screen using a non-integer zoom factor. Next step, recording. Cheers, Take

March 30th, 2007, 03:41 AM	#4
Wayne Morellini Inner Circle Join Date: May 2003 Location: Australia Posts: 2,762	Congradulations, are there any HD webcams for the mac this would work mon. If you are using the portable or mini, the Intel chipset emaulates many GPU fucntions, next chipset (GM965) hopefully should not. Good luck.

March 30th, 2007, 04:52 AM	#5
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	Hello Wayne, Thank you, I am pretty happy with the results right now. The software now is specifically written for the Fire-i camera, which is a 640x480 RGB webcam using the IIDC Firewire protocol. An HD camera that uses IIDC needs a lot of bandwidth and a Firewire 800 controller is required, which currently is only available on the MacBook Pro. The sustained transfer rate required for the disks is around 60 MB/sec, this can not be done using a Firewire interface as it is already in use by the camera. Thus a eSATA ExpressCard is need, ExpressCard is only available on the MacBook Pro. Cheers, Take

April 2nd, 2007, 03:17 PM	#6
Juan M. M. Fiebelkorn Major Player Join Date: Jun 2004 Location: Buenos Aires , Argentina Posts: 444	It sounds to me quite a lot simillar to just how Huffyuv compresses video.

April 2nd, 2007, 03:22 PM	#7
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	Indeed, it isn't exactly rocket science. I am changing my mind slightly though. To make it myself easy I am thinking of actually letting my recorder write OpenEXR files directly, probably without compression :-( Cheers, Take

April 8th, 2007, 12:30 PM	#8
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	Good and bad news. I've been able to record the images to disk in OpenEXR format. Bad news, OpenEXR is quite slow, for something that should be a simple copy operation it uses a tad to much CPU power. So it is back to the exponential golomb coding and writing my own type of file. Cheers, Take

April 11th, 2007, 11:27 AM	#9
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	Hello everyone, The recorder is working, it is creating my own digital negative format. I will publish specifications on this file format at release to source code to read and write these files. Of course I can not yet actually see if it is recording correctly, that is my next task. Cheers, Take

April 11th, 2007, 07:08 PM	#10
Solomon Chase Regular Crew Join Date: Jan 2006 Location: Woodstock, Georgia Posts: 154	Cool, keep posting updates :)

May 3rd, 2007, 05:40 PM	#12
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	I received the camera, it took me some time to get a picture from it, but it is working now. I currently have it set up to send quite a large 2.35:1 picture at 14 bits depth. It seems that the camera by default does not do any compensation as I hoped it would not, so the picture is quite a mess. It should not be much of a problem to get all that sensor unevenness removed. My current problem is that I had an old camera with a 1/3" CS mount lens, and I am using that lens on this camera. That does not make for a sharp picture without vignetting (although the inside of the lens and the three diaphragm blades are in focus). As for computer performance, I found out that my compression routine was using quite a lot of CPU time and it did not actually compress the data much (actually it became larger). So instead of Delta-Golomb compression it now simply packs the 14 bits tightly. Next step is finding a good lens and start working on writing to disk. Cheers, Take

May 8th, 2007, 12:46 PM	#13
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	An other update. Boom Recorder can now write the sensor data to disk, I just did a full scale test: 1800 x 750 (1:2.40 ratio), 16bits @ 24fps. I recorded for 9 minutes onto an external SATA disk, smooth as an android's bottom. I only configured a video ring buffer of 128 MByte (I should put more memory in my computer), but it kept up rather well as the maximum was around 50% filled. It was also pretty light on the CPU, the computer did not even turn on the fans. Next step: QuickTime component so that you can natively edit the footage on Final Cut Pro, it must do real-time debayer and compensate for fixed sensor noise. I also have to make an intermediate encoder, so that you can render the footage in Final Cut Pro to a 16 bits depth. QuickTime does have 16 bit depth capability, but it is not linear, rather it uses a gamma correction of 1.8. I guess Final Cut Pro converts it to linear floating point before doing any calculations. To bad QuickTime does not support linear floating point pixel types. Cheers, Take

May 27th, 2007, 01:02 AM	#15
Take Vos Major Player Join Date: Mar 2007 Location: Amsterdam The Netherlands Posts: 200	Hi there, I have another update for you. The QuickTime component is working, it can read the raw digital negative and demosaic it during playing. It asks for a 16 bit per-component drawable, so with a bit of luck Final Cut Pro can use all the 14 bits from the camera. I found in the Final Cut Pro manual that you only get fast (extreme) performance when the timeline uses the same codec and codec-settings as the clips that are imported. So I will make this QuickTime component to also be used on the timeline. Footage that needs to be coded by Final Cut Pro will use 48 bit RGB and not bayer, in other word it will silently use other settings without Final Cut Pro knowing about it. At this time the QuickTime component is rather slow as all the demosaicing and other processing is done in software. The plan is to use OpenGL for processing the footage before it enters Final Cut Pro. With a bit of luck you will be able to edit the footage in real-time. But the next programming challenge is an application that can make calibration data to be used by the QuickTime component and Mirage Recorder. The calibration data will consist of a dark-image offset and slope (based on the exposure time), and a light image offset and slope, and lastly a pixel repair list. light-image calibration image can be done with any "white"-light that has enough intensity in each red, green and blue component, as long as the same light is smeared equally over the whole sensor. Some kind of diffuser would be handy, or take the lens off the camera. As the calibration is color agnostic, I can use the color conversion matrix for this camera without change. White balance is then the job of Final Cut Pro (or Color if that works).