Do We Really Need 22MP?

[Kevin Dooley]

So I was thinking the other day - having a camera that shoots both video and stills is great and can really cut down on gear needed on some of our shoots, but wouldn't the video image from the 5D be better (aliasing/moire wise) if the sensor was lower resolution?

Obviously the Full Frame size of the sensor is a huge component to getting the look that the 5D gives us, but what if the sensor remained the same size, but only had 4K resolution (or even 2K since we'd be down-rezing to 1080p anyway)? Would that not eliminate the aliasing/moire and give the camera even better low-light/high ISO performance (with pixels that large)?

Not that I have my own chip to throw in there, but it was a thought I had while futzing with wardrobe to get rid of aliasing...

[Justin Benn]

Though things are likely to change, I gave up my Nikon D3 to buy a couple of 5DIIs to use video. The D3's image is far superior, tonally, in my opinion and has about half the pixel count/density - and thus larger photosites. Colour rendition on the D3 is just dreamy.

Once the D3 (or D700) gets updated with full on video modes, I think it will perform some video DSLR magic - especially in low light. But by then, we may have a 5DIII...

[Dominik Seibold]

Originally Posted by Kevin Dooley

Obviously the Full Frame size of the sensor is a huge component to getting the look that the 5D gives us, but what if the sensor remained the same size, but only had 4K resolution (or even 2K since we'd be down-rezing to 1080p anyway)? Would that not eliminate the aliasing/moire and give the camera even better low-light/high ISO performance (with pixels that large)?

True, but if it would "just" have HD resolution (=2MP), it would be a very unattractive DSLR. To avoid the aliasing and noise problem, the way to go is to use a better down-conversion (which of course needs electronics with higher performance). I consider it possible that upcoming versions of the 5D (and similar DSLRs) will provide higher quality down-conversion or even the ability to record higher/full-resolution-video. If the latter happens, cameras like the red one will quickly become history.

[Dan Brockett]

This is the exact reason why Panasonic has developed the new AG-AF100. You will never have optimal HD output from an DSLR that needs to also shoot very sharp still images. The OLPF (optical low pass filter) on the DSLRs will always have to be optimized for still images (21MP), not video (roughly 2MP). You can work around this limitation on a DSLR or you can buy a video camera that is designed for video only, not stills, like the AG-AF100. Makes sense to me.

I love shooting video with my 5D MKII but I kind of wish that it was a video optimized model, something like a 5D MKII Vid model because it is true, there is a lot of aliasing and moire' that we have to deal with and it can be limiting.

Dan Brockett

[Daniel Browning]

Originally Posted by Kevin Dooley

wouldn't the video image from the 5D be better (aliasing/moire wise) if the sensor was lower resolution?

If they cut the resolution all the way down to HD (21 MP -> 2 MP), that would definitely solve the problem.

Originally Posted by Kevin Dooley

Obviously the Full Frame size of the sensor is a huge component to getting the look that the 5D gives us, but what if the sensor remained the same size, but only had 4K resolution?

Even reducing the resolution from 21MP to 9.4MP (4K) still would not have been enough to read out the full sensor at 30 FPS and downsample in real time.

Originally Posted by Kevin Dooley

Would that not give the camera even better low-light/high ISO performance (with pixels that large)?

Under most circumstances (AKA "photon shot noise limited"), the low-light performance is the same. That is because light collecting ability scales in perfect proportion with pixel size. In other words, when you resize four small pixels (say, 8 MP) down to 1 large pixel (say, 2 MP), you get the same noise level as if the sensor was designed as 2 MP in the first place.

In atypical circumstances (AKA "read-noise limited"), there can be cases where the large pixel sensor has less nosie than the small-pixel sensor, because read noise doesn't scale linearly with pixel diameter like photoelectron efficiency does.

Originally Posted by Dominik Seibold

To avoid the aliasing and noise problem, the way to go is to use a better down-conversion

Agreed.

Originally Posted by Dan Brockett

You will never have optimal HD output from an DSLR that needs to also shoot very sharp still images.

I kindly disagree.

Originally Posted by Dan Brockett

The OLPF (optical low pass filter) on the DSLRs will always have to be optimized for still images (21MP), not video (roughly 2MP).

If the 21 MP image is downsampled properly, it will tend to have *less* aliasing than the 2MP video camera, despite the difference in OLPF. On top of that, the video camera OLPF will cause a significant loss of contrast at 30% lower resolutions than required, whereas the 21MP image will have the highest contrast possible.

Of course, implementing a proper downsample in-camera is pretty tough. I'd prefer to have the camera shoot compressed raw and let me choose the downsample myself in post.

[Dominik Seibold]

Originally Posted by Daniel Browning

Under most circumstances (AKA "photon shot noise limited"), the low-light performance is the same. That is because light collecting ability scales in perfect proportion with pixel size.

Assuming that the gap between the pixels wouldn't change when enlarging the pixels, more of the sensors area would get "alive", so it would indeed capture more photons, if you decrease the mega pixels.
But I absolutely agree with you that a digital low-pass filter produces much better images than an optical one, because a digital filter can have an arbitrary high order and so approach the nyquist-shannon-limit as close as thinkable. I would also prefer a high-resolution sensor with good downsampling over one with native HD-resolution.

[Dan Brockett]

Okay Daniel, fair enough. But you are talking engineering, I am talking marketing.

But I will still say that you won't see a Canon 5D MKIII or other DSLR that will have optimal video output in the near future. The wind is blowing toward DSLRs going back to just being still cameras that can shoot video anyway, rather than the state of the art cheap HD camera. The writing is on the wall with the AF-100 and the inevitable Sony, Canon and JVC imitators that are coming in the near future.

Dan Brockett

[Daniel Browning]

Originally Posted by Dominik Seibold

Assuming that the gap between the pixels wouldn't change when enlarging the pixels, more of the sensors area would get "alive", so it would indeed capture more photons, if you decrease the mega pixels.

The difference in how the gap between microlenses scales is such that optical Q.E. remains the same among a huge variety of pixel sizes. For example, all of the following cameras have an optical QE between 35-40% at 550nm in the green pixel:

Canon 1D4, 7D, 50D, 40D
Nikon D3X, D5000, D700
Olympus E-30, E410
Panasonic DMC-G1
Sony NEX-5

There is a huge variety of pixel sizes and designs there. For an even more extreme example, compare the king of low light performance, the Nikon D3s, with a cheap, old, little digicam: Canon G11. Both are 55% QE -- despite a tremendous difference in pixel size: 8.4 micron vs. 2.1 micron. (That's 71 um^2 vs 4.4 um^2.) This proves that light collecting ability can easily scale accross huge differences in pixel size.

Originally Posted by Dan Brockett

But I will still say that you won't see a Canon 5D MKIII or other DSLR that will have optimal video output in the near future.

Agreed. I think we've got at least 4 more years to go.

[Dominik Seibold]

Originally Posted by Daniel Browning

The difference in how the gap between microlenses scales is such that optical Q.E. remains the same among a huge variety of pixel sizes.

According to wikipedia quantum efficiency is "the percentage of photons hitting the photoreactive surface that will produce an electron–hole pair". But the gaps between the pixels aren't included as "photoreactive surface" in this definition, are they? So the quantum efficiency term doesn't deal with gaps, but just gives information about the efficiency of the pixels itself. So there could be two sensors of the same size, the same pixel amount and the same Q.E. of their photoreactive surfaces, but with different low-light capabilities, because of different gap/pixel sizes (which leads to a different efficiency of photons reaching active surface per total incoming photons). So increasing the pixel-size (respectively lowering the pixel count) without changing the gap-size would give a sensor with more active surface and therefore higher efficiency regarding capturing incoming photons. Am I missing something?

Originally Posted by Daniel Browning

Agreed. I think we've got at least 4 more years to go.

I don't know when you would determine something as optimal, but do you think there won't be any improvement in video quality with a hypothetical 5D mark III at all?

[Daniel Browning]

Originally Posted by Dominik Seibold

According to wikipedia quantum efficiency is "the percentage of photons hitting the photoreactive surface that will produce an electron–hole pair". But the gaps between the pixels aren't included as "photoreactive surface" in this definition, are they?

QE always includes the gaps between pixels, otherwise every unfiltered sensor would have 100% QE, even without microlenses. The only thing that wikipedia means by that term is "surface area of the photographic film or image sensor".

Originally Posted by Dominik Seibold

I don't know when you would determine something as optimal, but do you think there won't be any improvement in video quality with a hypothetical 5D mark III at all?

I think there will be some improvement, but nowhere near optimal. The first problem is that they need to readout pixels much, much faster. This is probably hard, because readout speed in Mpix/second hasn't advanced much in the last 20 years. Then they need to process it into a quality 1080p, but at least this part benefits from Moore's Law. The still photographer's need for increased pixel count is going to make both of these even harder.

One possibility for a short-term solution is in-sensor binning combined with a variable switch anti-alias filter. For example, if you could flip a switch that enabled stronger optical anti-aliasing (e.g. through a circular sensor shift or by something in the actual optical path), then on-sensor binning could be used without aliasing -- then 33 MP could be readout as fast as 2 MP. You wouldn't have the contrast advantage of reading out the full 33 MP and downsampling, but it would be something.

[Bill Binder]

Don't even think about taking away my 21mp RAW files, thank you...

(Says the stills photographer in the minority on this site, heh...)

[Tim Polster]

I agree with Bill and Dan. Yes we need 22mp as this is a still camera and it is really nice to be able to crop and retain image integrity on a larger print. I think 22mp is enough going forward and would rather have the industry focus on dynamic range and even better noise/low light performance.

This does not help the video side of things but I think a dedicated video version would be best in the long run. Canon will probably take a while to get the 5DMKIII out as the MKII is still drawing a crowd. But they probably also know this will not last forever with cameras like the AF-100 finally appearing.

Canon's version will be something to watch for although who knows when it will surface.

[Dominik Seibold]

Originally Posted by Daniel Browning

QE always includes the gaps between pixels, otherwise every unfiltered sensor would have 100% QE, even without microlenses. The only thing that wikipedia means by that term is "surface area of the photographic film or image sensor".

I don't think that there's any material which has a QE of 100%. If that would exist, we would have the perfect solar cells, which would look really dark. ;) I'm not an expert in semiconductors, but I know that in this world there's nothing with an efficiency of 100%. There are always some imperfections and tradeoffs.
Another question is: If we assume, that the term quatum efficiency indeed descripes the efficiency of the whole sensor (including the gaps), why wouldn't it change if the gap-sizes change? If we would increase the gaps-sizes, the portion of the sensors area, which is photoreactive, would get smaller and more photons would hit the unresponsive gaps. So why should the QE stay the same?

Originally Posted by Daniel Browning

I think there will be some improvement, but nowhere near optimal.

What is optimal? As you know, even native HD-sensors can't have an optimal output, because of their optical low-pass filter, which is always a tradeoff between resolution and aliasing. So it would suffice to provide the sub-optimal quality of native sensors, to be an attractive alternative solution.

Originally Posted by Daniel Browning

(...)readout speed in Mpix/second hasn't advanced much in the last 20 years.

Could you provide a reference for this information? And what about cameras like the red one, which have CMOS-sensors and provide 12MP video?

Originally Posted by Daniel Browning

You wouldn't have the contrast advantage of reading out the full 33 MP and downsampling, but it would be something.

And you wouldn't have any reduction of noise, which is pretty high for my taste even when recording video at ISO100.

[Daniel Browning]

Originally Posted by Dominik Seibold

I don't think that there's any material which has a QE of 100%.

You're right. I should not have strayed from my main point which is that the definition of QE definitely includes the gaps between pixels.

Originally Posted by Dominik Seibold

If that would exist, we would have the perfect solar cells, which would look really dark.

Well, that QE is only for 550nm which I think I mentioned a few posts ago. Other wavelengths tend to be worse (in typical designs, the bluer part of the spectrum is low).

Originally Posted by Dominik Seibold

So why should the QE stay the same?

I have no idea why. All I know is that it does stay the same. Maybe it's because they are able to scale the microlens design with pixel size. Or maybe it's because they use slower microlens f-numbers with smaller acceptance angles. Or maybe it's because of factors that are beyond my comprehension. Might as well be pixie dust sprinkled on the microlenses, for all I know. :)

Whatever it is they are doing, it works, because I've measured and seen measurements of $5,000 DSLR with 8 micron pixels and the $500 digicam with 2 micron pixels. They both have 55% QE.

Originally Posted by Dominik Seibold

What is optimal?

To me, optimal is basically a RED ONE with a high quality demosaic and downsample (e.g. lanczos) to an in-camera 1080p format.

Originally Posted by Dominik Seibold

So it would suffice to provide the sub-optimal quality of native sensors, to be an attractive alternative solution.

Agreed.

Originally Posted by Dominik Seibold

Could you provide a reference for this information?

In a recent discussion with other image sensor technologists, the inventor of CMOS image sensors said "It is interesting to note that output data rates have hovered around 1 Gpix/sec since the late 1990's." A few exceptions were pointed out (e.g. Vision Research 7.5 Gpix/second):

Image Sensors World: Canon Announced 120MP APS-H-sized Sensor

Originally Posted by Dominik Seibold

And what about cameras like the red one, which have CMOS-sensors and provide 12MP video?

Lookaround is 11.5 MP, at the maximum rate is 30 FPS, comes to 344 Mpix/second. I seem to recall somewhere that the viewfinder can run at 60 FPS, though. If that's true and all the pixels are read out for that, then the Mysterium can do 689 Mpix/s.

Originally Posted by Dominik Seibold

And you wouldn't have any reduction of noise, which is pretty high for my taste even when recording video at ISO100.

I agree that ISO 100 on the 5D2 has a bad pattern noise problem. It limits me to about 5-7 stops of dynamic range, depending on the color temperature of the light. If it didn't have that problem, then I would easily be able to use 12 or more stops of dynamic range in raw files and small display sizes.

[John Wiley]

I think an APS-C sized video optimised chip would be amazing, but the problem is that the r & d expenses are so much higher than using current sensors and changing other things like the OLPF.

I have read a rumour in a few places that Canon are working on a 22mp sensor, not optimised for video but with a lot of consideration about video. While I don't necassarily believe the rumour or want to perpetuate it, their was some maths and reasoning behind 22mp being a magic number - I think it was something like crop the top and bottom to get a 16:9 image, then while reading out the data pixels are put into into groups of 9 and you end up with a 1920x1080 image.

I'm no engineer but I'd like to think that people much smarter than me are researching these sorts of options.