Possible future CMOS Image Sensor ....

[Bob Diaz]

When it come to image sensors, there's always a trade-off between CCD and CMOS. CCD offers good images, but requires a more complex power supply, is less flexible, suffers from vertical smear with super bright lights, and draws more power. CMOS seems to have the upper hand in several key areas over CCD, but suffers from the rolling shutter. However, we could be one generation away from seeing a major change in CMOS technology. What follows is a simplified explanation of how Sony stands to change the world of video with a simple improvement to an existing design. Keep in mind, I can tell you what is possible, BUT I have no information as to what Sony's Marketing Department is planning on doing next. I would hope that the marketing department of Sony is going to make this next move, but until it happens or not, I have no way of knowing for sure. For those who want to skip all the technical details, just skip down to the section called "The Bottom Line".

Conventional design would be to connect a single High Speed A/D (Analog to Digital) converter to the output of CCD or CMOS image sensor (most CMOS sensors have the A/D converter on the same chip). In order to be able to convert all the pixels fast enough, the converter must be able to read and convert each pixel to digital at 75 MHz. In this case, a Flash Converter is used.

Inside the Flash Converter chip is many Comparators. A Comparator compares 2 input voltages, on Input A, we have the unknown voltage from our image sensor. On Input B we have a fixed or reference voltage. If the voltage on Input A is greater than the voltage on Input B, the output is "1". Otherwise, the output is "0".

In the case of a 10 Bit Flash Converter, we would have 1023 Comparators. Picture each Comparator's voltage as being 1 mV (1 millivolt or 1/1,000 of a volt) higher than the lower Comparator. Thus, the comparators would be 1mV, 2mV, 3mV, ..., 1023mV. Thus for any voltage from 0 to 1.023 volts, any number of the Comparators will have an output of "1". The total number of Comparators at "1" represents the digital value for the voltage of the signal.

When Sony introduced the IMX017CQE CMOS Image Sensor, they broke with conventional design. The IMX017CQE does not use a Flash Converter, nor does it use a single A/D Converter.


COLUMN PARALLEL A/D CONVERSION:

The IMX017CQE contains 2,916 A/D Converters; one for each column and all the A/D Converters are working in parallel. Now Sony could have designed the IMX017CQE with Flash A/D Converters, but that would result in an excessive amount of chip space for all the converters and is really unnecessary.

A much smaller and simpler A/D Converter is the Ramp Converter. The Ramp Converter would start at 1mV and using a single Comparator per column, test each 1 mV step until it reaches the maximum count or the step voltage is greater than the voltage of the video signal. By counting the number of steps before we reach a step voltage greater than the voltage of the video, we know the digital value for the input voltage.

While Ramp Converters tend to be the slowest of A/D Converters, but because there are so many working in parallel, this makes up for the slow speed. In fact, the IMX017CQE can produce an 2,880h x 2,160v image 60 times per second. This is then converted to 1920h x 1080v.

The IMX017CQE is a 6 MP (Mega-Pixel) single chip image sensor that measures 1/1.8" (0.56 inches) and is intended for consumer applications. Professionals will be disappointed to hear that the chip generates this HD image with only a 10 bit resolution.

If the A/D conversation has 12 bits, the count takes 4 times longer and the HD image must be scanned at 60P / 4 = 15P. Since most profession video cameras use 14 bits for the A/D conversion, this would slow down the conversion by a total of 60P / 16 = 3.75P. Clearly these are unacceptable scan rates for professionals.

The problem with the Ramp A/D Converter is that as we try to increase the number of bits in the measurement, the count must double for each bit added. Thus, for 10 bits, the maximum count is 1024, but for 12 bits the count must be 4 times higher, 4096 and at 14 bits, the count becomes 16,384.

Sony is already pushing the limits with a 300 MHz internal clock for their Ramp A/D Converter. If Sony wants to use the Column Parallel A/D Converter Technology for a professional CMOS sensor, they have to use a faster conversion system. Once I realized that, I saw exactly what their solution for the next generation Column Parallel A/D Converter would be.


SUCCESSIVE APPROXIMATION CONVERTER:

For a 10 bit conversion, rather than go through every number (0 --> 1023), wouldn't it be better to start in the middle (512) and see if the value is greater than or less than the middle value? If greater than 512, try a value in between 512 and 1023, like 768. If less than 512, try a value in between 512 and 0, like 256. Notice that with each test, the search range drops down to 1/2 of what it was before the search.

This might sound complex, but with binary, this is easy. The following are the bit values for a 10 bit number: 512, 256, 128, 64, 32, 16, 8, 4, 2, and 1.

The first test begins with the highest value bit (512) set to 1, thus the number tested is 512. If the value is greater than this, the next test is 768 (512 + 256). If the value is less than this, the next number tested is 256. For each test, we will try 128, 64, 32, 16, 8, 4, 2, and finely 1.

With Successive Approximation, the number of steps is the number of bits plus 1. In our example, that's 10 bits + 1 step for the set-up. If we chose to use 14 bits, then there are 14+1 steps required.


The Bottom Line:

Assume that on the improved image sensor, rather than use a 300 MHz clock for each step, Sony chose to slow down the conversion by a factor of 3. I'll skip all the math, but it turns out that the rolling shutter would travel from top to bottom in less than 1/6,000 of a second. Now it's important to understand that on the chip nothing moves, but the rolling shutter acts as if a mechanical shutter opens and closes.

On a regular CMOS image sensor, the rolling shutter travels across the chip at 1/60 of a second, so our improved CMOS sensor would have a rolling shutter that's over 100 times faster. This would make a major difference in overcoming the problems with a rolling shutter.

Rapid pans with a rolling shutter can cause the vertical lines to lean, but in order to see the lean with a conventional CMOS sensor, the image must travel faster than 4 seconds from one edge of the screen to the other.

With our improved CMOS chip, this would need to be 100 times faster to have the same tilt, of 1/25 of a second from one edge to another. However at that speed, the image would be a blur. So in order to clearly see the line, the shutter speed would need to be 1/12,000 of a second or faster.

While it's possible to generate these conditions, it's not typical of real world conditions. So, the future next generation Column Parallel CMOS sensors will be mostly free of this artifact.

What about electronic flashes? With standard CMOS image sensors, part of the frame is exposed and the other part is not. The location is random, depending on when the flash is fired.

With a faster rolling shutter, it's less than 1 chance out of 50 that the flash will fire while the shutter is opening or closing. On the other hand, it's 49 chances out of 50 that the flash will not fire when the shutter is opening or closing. Even if the flash fires while the shutter is opening or closing, the rolling shutter is moving so fast, that 6 scan lines represent less than 1/1,000,000 of a second. That's so fast, that we might see a soft transition from dark to light.

Building the chip would be an impressive work of engineering design. A 1920h x 1080v image sensor would require 1920 Parallel Successive Approximation A/D Converters. Each Converter would require a D/A Converter, a Comparator, and the necessary control logic to drive the conversion.

In addition, in order to remove "fixed pattern noise" a 1920h x 1080v 14 bit word RAM would be required. The RAM would hold the reset levels for each cell and would need to have the ability to subtract the fixed pattern noise from the signal. If the result of the subtraction is placed back into the RAM, this could act as a buffer and allow for slower reads of the results.

While it would take a bit of work to design the chip and it will cost more than conventional CMOS image sensors due to all the extra space required, it will be worth it to profession videographers. Now let's hope that Sony is moving in that direction...


Bob Diaz

[Jack Zhang]

A global shutter 1920x1080 CMOS system in development? Sounds promising.

[Bob Diaz]

Remember I can say what is possible, but I can't be sure what Sony decides to do as far as development. IF I worked at Sony, I'd be pushing for this type of technology.

It's not a true global shutter, but it would be a very fast rolling shutter. The faster the speed of a rolling shutter, the less the impact of the rolling shutter.

I really hope I've guessed correctly about this happening...


Bob Diaz