So I was thinking the other day - having a camera that shoots both video and stills is great and can really cut down on gear needed on some of our shoots, but wouldn't the video image from the 5D be better (aliasing/moire wise) if the sensor was lower resolution?

Obviously the Full Frame size of the sensor is a huge component to getting the look that the 5D gives us, but what if the sensor remained the same size, but only had 4K resolution (or even 2K since we'd be down-rezing to 1080p anyway)? Would that not eliminate the aliasing/moire and give the camera even better low-light/high ISO performance (with pixels that large)?

Not that I have my own chip to throw in there, but it was a thought I had while futzing with wardrobe to get rid of aliasing...

Though things are likely to change, I gave up my Nikon D3 to buy a couple of 5DIIs to use video. The D3's image is far superior, tonally, in my opinion and has about half the pixel count/density - and thus larger photosites. Colour rendition on the D3 is just dreamy.

Once the D3 (or D700) gets updated with full on video modes, I think it will perform some video DSLR magic - especially in low light. But by then, we may have a 5DIII...

Obviously the Full Frame size of the sensor is a huge component to getting the look that the 5D gives us, but what if the sensor remained the same size, but only had 4K resolution (or even 2K since we'd be down-rezing to 1080p anyway)? Would that not eliminate the aliasing/moire and give the camera even better low-light/high ISO performance (with pixels that large)?
True, but if it would "just" have HD resolution (=2MP), it would be a very unattractive DSLR. To avoid the aliasing and noise problem, the way to go is to use a better down-conversion (which of course needs electronics with higher performance). I consider it possible that upcoming versions of the 5D (and similar DSLRs) will provide higher quality down-conversion or even the ability to record higher/full-resolution-video. If the latter happens, cameras like the red one will quickly become history.

This is the exact reason why Panasonic has developed the new AG-AF100. You will never have optimal HD output from an DSLR that needs to also shoot very sharp still images. The OLPF (optical low pass filter) on the DSLRs will always have to be optimized for still images (21MP), not video (roughly 2MP). You can work around this limitation on a DSLR or you can buy a video camera that is designed for video only, not stills, like the AG-AF100. Makes sense to me.

I love shooting video with my 5D MKII but I kind of wish that it was a video optimized model, something like a 5D MKII Vid model because it is true, there is a lot of aliasing and moire' that we have to deal with and it can be limiting.

Dan Brockett

wouldn't the video image from the 5D be better (aliasing/moire wise) if the sensor was lower resolution?


If they cut the resolution all the way down to HD (21 MP -> 2 MP), that would definitely solve the problem.


Obviously the Full Frame size of the sensor is a huge component to getting the look that the 5D gives us, but what if the sensor remained the same size, but only had 4K resolution?


Even reducing the resolution from 21MP to 9.4MP (4K) still would not have been enough to read out the full sensor at 30 FPS and downsample in real time.


Would that not give the camera even better low-light/high ISO performance (with pixels that large)?


Under most circumstances (AKA "photon shot noise limited"), the low-light performance is the same. That is because light collecting ability scales in perfect proportion with pixel size. In other words, when you resize four small pixels (say, 8 MP) down to 1 large pixel (say, 2 MP), you get the same noise level as if the sensor was designed as 2 MP in the first place.

In atypical circumstances (AKA "read-noise limited"), there can be cases where the large pixel sensor has less nosie than the small-pixel sensor, because read noise doesn't scale linearly with pixel diameter like photoelectron efficiency does.


To avoid the aliasing and noise problem, the way to go is to use a better down-conversion

Agreed.

You will never have optimal HD output from an DSLR that needs to also shoot very sharp still images.


I kindly disagree.


The OLPF (optical low pass filter) on the DSLRs will always have to be optimized for still images (21MP), not video (roughly 2MP).


If the 21 MP image is downsampled properly, it will tend to have *less* aliasing than the 2MP video camera, despite the difference in OLPF. On top of that, the video camera OLPF will cause a significant loss of contrast at 30% lower resolutions than required, whereas the 21MP image will have the highest contrast possible.

Of course, implementing a proper downsample in-camera is pretty tough. I'd prefer to have the camera shoot compressed raw and let me choose the downsample myself in post.

Under most circumstances (AKA "photon shot noise limited"), the low-light performance is the same. That is because light collecting ability scales in perfect proportion with pixel size.
Assuming that the gap between the pixels wouldn't change when enlarging the pixels, more of the sensors area would get "alive", so it would indeed capture more photons, if you decrease the mega pixels.
But I absolutely agree with you that a digital low-pass filter produces much better images than an optical one, because a digital filter can have an arbitrary high order and so approach the nyquist-shannon-limit as close as thinkable. I would also prefer a high-resolution sensor with good downsampling over one with native HD-resolution.

Okay Daniel, fair enough. But you are talking engineering, I am talking marketing.

But I will still say that you won't see a Canon 5D MKIII or other DSLR that will have optimal video output in the near future. The wind is blowing toward DSLRs going back to just being still cameras that can shoot video anyway, rather than the state of the art cheap HD camera. The writing is on the wall with the AF-100 and the inevitable Sony, Canon and JVC imitators that are coming in the near future.

Dan Brockett

Assuming that the gap between the pixels wouldn't change when enlarging the pixels, more of the sensors area would get "alive", so it would indeed capture more photons, if you decrease the mega pixels.


The difference in how the gap between microlenses scales is such that optical Q.E. remains the same among a huge variety of pixel sizes. For example, all of the following cameras have an optical QE between 35-40% at 550nm in the green pixel:

Canon 1D4, 7D, 50D, 40D
Nikon D3X, D5000, D700
Olympus E-30, E410
Panasonic DMC-G1
Sony NEX-5

There is a huge variety of pixel sizes and designs there. For an even more extreme example, compare the king of low light performance, the Nikon D3s, with a cheap, old, little digicam: Canon G11. Both are 55% QE -- despite a tremendous difference in pixel size: 8.4 micron vs. 2.1 micron. (That's 71 um^2 vs 4.4 um^2.) This proves that light collecting ability can easily scale accross huge differences in pixel size.


But I will still say that you won't see a Canon 5D MKIII or other DSLR that will have optimal video output in the near future.


Agreed. I think we've got at least 4 more years to go.

The difference in how the gap between microlenses scales is such that optical Q.E. remains the same among a huge variety of pixel sizes.
According to wikipedia quantum efficiency is "the percentage of photons hitting the photoreactive surface that will produce an electron–hole pair". But the gaps between the pixels aren't included as "photoreactive surface" in this definition, are they? So the quantum efficiency term doesn't deal with gaps, but just gives information about the efficiency of the pixels itself. So there could be two sensors of the same size, the same pixel amount and the same Q.E. of their photoreactive surfaces, but with different low-light capabilities, because of different gap/pixel sizes (which leads to a different efficiency of photons reaching active surface per total incoming photons). So increasing the pixel-size (respectively lowering the pixel count) without changing the gap-size would give a sensor with more active surface and therefore higher efficiency regarding capturing incoming photons. Am I missing something?
Agreed. I think we've got at least 4 more years to go.
I don't know when you would determine something as optimal, but do you think there won't be any improvement in video quality with a hypothetical 5D mark III at all?

According to wikipedia quantum efficiency is "the percentage of photons hitting the photoreactive surface that will produce an electron–hole pair". But the gaps between the pixels aren't included as "photoreactive surface" in this definition, are they?


QE always includes the gaps between pixels, otherwise every unfiltered sensor would have 100% QE, even without microlenses. The only thing that wikipedia means by that term is "surface area of the photographic film or image sensor".


I don't know when you would determine something as optimal, but do you think there won't be any improvement in video quality with a hypothetical 5D mark III at all?

I think there will be some improvement, but nowhere near optimal. The first problem is that they need to readout pixels much, much faster. This is probably hard, because readout speed in Mpix/second hasn't advanced much in the last 20 years. Then they need to process it into a quality 1080p, but at least this part benefits from Moore's Law. The still photographer's need for increased pixel count is going to make both of these even harder.

One possibility for a short-term solution is in-sensor binning combined with a variable switch anti-alias filter. For example, if you could flip a switch that enabled stronger optical anti-aliasing (e.g. through a circular sensor shift or by something in the actual optical path), then on-sensor binning could be used without aliasing -- then 33 MP could be readout as fast as 2 MP. You wouldn't have the contrast advantage of reading out the full 33 MP and downsampling, but it would be something.

Don't even think about taking away my 21mp RAW files, thank you...

(Says the stills photographer in the minority on this site, heh...)

I agree with Bill and Dan. Yes we need 22mp as this is a still camera and it is really nice to be able to crop and retain image integrity on a larger print. I think 22mp is enough going forward and would rather have the industry focus on dynamic range and even better noise/low light performance.

This does not help the video side of things but I think a dedicated video version would be best in the long run. Canon will probably take a while to get the 5DMKIII out as the MKII is still drawing a crowd. But they probably also know this will not last forever with cameras like the AF-100 finally appearing.

Canon's version will be something to watch for although who knows when it will surface.

QE always includes the gaps between pixels, otherwise every unfiltered sensor would have 100% QE, even without microlenses. The only thing that wikipedia means by that term is "surface area of the photographic film or image sensor".
I don't think that there's any material which has a QE of 100%. If that would exist, we would have the perfect solar cells, which would look really dark. ;) I'm not an expert in semiconductors, but I know that in this world there's nothing with an efficiency of 100%. There are always some imperfections and tradeoffs.
Another question is: If we assume, that the term quatum efficiency indeed descripes the efficiency of the whole sensor (including the gaps), why wouldn't it change if the gap-sizes change? If we would increase the gaps-sizes, the portion of the sensors area, which is photoreactive, would get smaller and more photons would hit the unresponsive gaps. So why should the QE stay the same?
I think there will be some improvement, but nowhere near optimal.
What is optimal? As you know, even native HD-sensors can't have an optimal output, because of their optical low-pass filter, which is always a tradeoff between resolution and aliasing. So it would suffice to provide the sub-optimal quality of native sensors, to be an attractive alternative solution.
(...)readout speed in Mpix/second hasn't advanced much in the last 20 years.
Could you provide a reference for this information? And what about cameras like the red one, which have CMOS-sensors and provide 12MP video?
You wouldn't have the contrast advantage of reading out the full 33 MP and downsampling, but it would be something.
And you wouldn't have any reduction of noise, which is pretty high for my taste even when recording video at ISO100.

I don't think that there's any material which has a QE of 100%.

You're right. I should not have strayed from my main point which is that the definition of QE definitely includes the gaps between pixels.

If that would exist, we would have the perfect solar cells, which would look really dark.

Well, that QE is only for 550nm which I think I mentioned a few posts ago. Other wavelengths tend to be worse (in typical designs, the bluer part of the spectrum is low).


So why should the QE stay the same?


I have no idea why. All I know is that it does stay the same. Maybe it's because they are able to scale the microlens design with pixel size. Or maybe it's because they use slower microlens f-numbers with smaller acceptance angles. Or maybe it's because of factors that are beyond my comprehension. Might as well be pixie dust sprinkled on the microlenses, for all I know. :)

Whatever it is they are doing, it works, because I've measured and seen measurements of $5,000 DSLR with 8 micron pixels and the $500 digicam with 2 micron pixels. They both have 55% QE.


What is optimal?


To me, optimal is basically a RED ONE with a high quality demosaic and downsample (e.g. lanczos) to an in-camera 1080p format.


So it would suffice to provide the sub-optimal quality of native sensors, to be an attractive alternative solution.


Agreed.


Could you provide a reference for this information?


In a recent discussion with other image sensor technologists, the inventor of CMOS image sensors said "It is interesting to note that output data rates have hovered around 1 Gpix/sec since the late 1990's." A few exceptions were pointed out (e.g. Vision Research 7.5 Gpix/second):

Image Sensors World: Canon Announced 120MP APS-H-sized Sensor (http://image-sensors-world.blogspot.com/2010/08/canon-announced-120mp-aps-h-sized.html)

And what about cameras like the red one, which have CMOS-sensors and provide 12MP video?

Lookaround is 11.5 MP, at the maximum rate is 30 FPS, comes to 344 Mpix/second. I seem to recall somewhere that the viewfinder can run at 60 FPS, though. If that's true and all the pixels are read out for that, then the Mysterium can do 689 Mpix/s.


And you wouldn't have any reduction of noise, which is pretty high for my taste even when recording video at ISO100.

I agree that ISO 100 on the 5D2 has a bad pattern noise problem. It limits me to about 5-7 stops of dynamic range, depending on the color temperature of the light. If it didn't have that problem, then I would easily be able to use 12 or more stops of dynamic range in raw files and small display sizes.

I think an APS-C sized video optimised chip would be amazing, but the problem is that the r & d expenses are so much higher than using current sensors and changing other things like the OLPF.

I have read a rumour in a few places that Canon are working on a 22mp sensor, not optimised for video but with a lot of consideration about video. While I don't necassarily believe the rumour or want to perpetuate it, their was some maths and reasoning behind 22mp being a magic number - I think it was something like crop the top and bottom to get a 16:9 image, then while reading out the data pixels are put into into groups of 9 and you end up with a 1920x1080 image.

I'm no engineer but I'd like to think that people much smarter than me are researching these sorts of options.

I have no idea why. All I know is that it does stay the same.
Take a look at this review page of the 5D mark II:
Canon EOS 5D Mark II Review: 3. What's New: Digital Photography Review (http://www.dpreview.com/reviews/CanonEOS5DMarkII/page3.asp)
They state, that the gaps became smaller compared to the original 5D. Also they mention, that the 50D has a gap-less design. So the gap-sizes seem indeed to vary across different camera models. But if all kinds of sensors have a QE of 55% at a certain wavelength, I interpret this as an indication that often the same semiconductor material is used and the QE is just a property of this material and not the whole sensor with all its microlenses and complex engineering-stuff. But I may be wrong.
Whatever it is they are doing, it works, because I've measured and seen measurements of $5,000 DSLR with 8 micron pixels and the $500 digicam with 2 micron pixels. They both have 55% QE.
How did you measure that?
In a recent discussion with other image sensor technologists, the inventor of CMOS image sensors said "It is interesting to note that output data rates have hovered around 1 Gpix/sec since the late 1990's."
1 GPixel/s would be enough to read 22MP 30 times per second. So do you think that it is possible but too expensive, to build it into a $3000 camera within the next years?

They state, that the gaps became smaller compared to the original 5D.


Yes, and that is one reason why smaller pixels tend to perform better than larger pixels if you don't control for technology level (as I'm sure you already know).


Also they mention, that the 50D has a gap-less design. So the gap-sizes seem indeed to vary across different camera models. But if all kinds of sensors have a QE of 55% at a certain wavelength,


The original 5D had a QE of 25%, the 5D2 improved this to 33%. The move to gapless microlenses in the 50D improved QE from 33% in the 40D to 38% in the 50D.


How did you measure that?


Take several raw files of a colorchecker chart, plot the values of the gray patches, and the slope gives the inverse gain (ADU per electron). From there all that's needed to calculate the QE is the ISO calibration of the camera (one camera's "ISO 200" may be another camera's "ISO 500"), but figuring that out requires some sort of calibrated lighting system and highly accurate light meter. So I just use measured ISO numbers from DxOMark.


1 GPixel/s would be enough to read 22MP 30 times per second. So do you think that it is possible but too expensive, to build it into a $3000 camera within the next years?

Yes, that's my guess.

Couldn't you just use larger pixels while in video mode? It seems like a straightforward way to optimize it. If you increased the pixel size so that the data was being read as a 2 or 4 megapixel video image it seems like it would a. increase light sensitivity and b. severely reduce the aliasing common with high pixel images.

Or did I just get off at the wrong train station?

Couldn't you just use larger pixels while in video mode?
How do you increase the pixel-size without physically replacing the whole sensor?

My understanding is that you don't literally have to read every point of data on the sensor as a pixel. You can choose instead to use an area with several physical points of sensor data as a single pixel. For example, you could read every 10x10 points on the sensor as 1 pixel for your image. Generally the bigger the pixel the better low light performance. Additionally, lowering the image size to 2-4 megapixels can also significantly lower aliasing artifacts.

Well it sounds like a win-win anyway, but camera manufacturers have been shy to go down this route, so there may be a good reason it hasn't been implemented. But it still sounds logical to me, or am I nuts?

For example, you could read every 10x10 points on the sensor as 1 pixel for your image.
So you want to read out the average value of multiple pixels with just one read-out operation? I don't think that's possible, without designing a whole new sensor. But I think it sounds like an interesting idea.

I thought this was being done in most cameras and camcorders anyway. There's a single read operation from the sensor and then compression is applied to raw sensor data to give us the final image. The larger pixel idea would be applied after the raw sensor data was read.

My understanding is that you don't literally have to read every point of data on the sensor as a pixel. You can choose instead to use an area with several physical points of sensor data as a single pixel. For example, you could read every 10x10 points on the sensor as 1 pixel for your image. Generally the bigger the pixel the better low light performance. Additionally, lowering the image size to 2-4 megapixels can also significantly lower aliasing artifacts.

Well it sounds like a win-win anyway, but camera manufacturers have been shy to go down this route, so there may be a good reason it hasn't been implemented. But it still sounds logical to me, or am I nuts?

That's pretty much what I was talking about. The rumour (again, I don't want to perpetuate or confirm it) is that Canon is working on sensor that does just that in video mode. I don't know anything about any future cameras but the logic behind the rumour seems to make sense.

Although a 10x10 pixel area might be a bit large. That would mean that to get a 1920x1080 image, you'd be working with roughly a 220mp sensor!

I thought this was being done in most cameras and camcorders anyway. There's a single read operation from the sensor and then compression is applied to raw sensor data to give us the final image. The larger pixel idea would be applied after the raw sensor data was read.
There's a read-out operation for each individual pixel, not just for the one frame! That's why 5DmIIs video has a lot of aliasing, because only every third row and column gets read out to reduce the number of read-out operations per frame. If you want to make a virtual larger pixel, you would have to read out each native one within it first. So you wouldn't cope with the problem that the number of pixels which can get read out per second is limited. Btw. calculating the average out of a block of pixels and building a new frame with smaller dimensions out of these blocks is equivalent to apply a box-low-pass-filter and then downsampling to the raw data.
I initially understood your idea, that the sensor gets designed in a way, that multiple pixels can become connected to additional operational amplifier circuits, which generate the average values in an analog way, and only the output of these additional circuits gets read out.