Does anyone have any concrete information about how the camera is using an 18 megapixel sensor to record a 2 megapixel video stream? Does it record a full resolution (whole sensor) image which the processor resizes down during compression, or is only part of the sensor used for video recording?

We read that the camera has a bigger sensor than most video cameras, but I wonder how much of the sensor is actually being used for the video?

On the 5D, the whole width is used. It's cropped top and bottom to make a 16:9 image. Horizontally, the image is filtered to balance aliasing and resolution. Vertically, the photosites are "binned" (added together), which causes aliasing.

Earlier reports claimed that one of every third line was used, but I recall an interview with a Canon engineer where he said that they were binned. That makes sense, as the noise will be lower with binning than with skipping two of three rows.

From what I've seen, the 7D almost certainly takes the same approach for 1080p, though it's not confirmed. I have no idea what it does to achieve 720p, though I assume that it still uses the whole sensor width.

Earlier reports claimed that one of every third line was used, but I recall an interview with a Canon engineer where he said that they were binned. That makes sense, as the noise will be lower with binning than with skipping two of three rows.
Can you please point to any authoritative information that discusses how these cameras work? I haven't found any official statements from anyone.

Binning makes no sense on a Bayer sensor, no sense at all. Now, if it was an RGB Stripe sensor, vertical binning would make perfect sense, but on Bayer it just plain wouldn't work.

The AA filter strength is determined by the small pixels used in still photos. A stronger AA would help video, but hurt still.
So does red just throw out pixels? That would explain reds worse noise but less aliasing output. When the 5DII was released, Red said something about their superior ability to read/reset the chip. I have difficulty reconciling that comment to Canon reading every pixel every frame.
Also, both Red and the 7D overcrank at reduced resolution (double speed means half the pixels). Is this done by skipping lines?

Does anyone have any concrete information about how the camera is using an 18 megapixel sensor to record a 2 megapixel video stream? Does it record a full resolution (whole sensor) image which the processor resizes down during compression, or is only part of the sensor used for video recording?
The speculation is that it skips pixels. The theory being that it can't read all 18 megapixels at 60 frames per second, that's an enormous task -- and it certainly isn't likely to read all 18 megapixels and perform demosaic'ing and scaling down to 2mpix (1080p mode) or 1mpix (720p mode) in realtime. Therefore the speculation is that it skips pixels horizontally and vertically to reduce the amount of chip that it has to read.

It's not "windowing" the sensor. It's not using the central 1920x1080 out of the chip or anything like that. It uses the full size of the chip, but perhaps not all the pixels on the chip.

Such an approach would lead to significant aliasing and moire issues, and indeed those have been observed in the video, so it seems like a likely explanation for how the chips are working. I just haven't found any official confirmation that that is indeed how they work. Alan Roberts' BBC assessment showed the effects, but he didn't speculate on whether it was row-skipping, he just concluded that it had a poor downscaling algorithm. You can see the results here: http://www.dvinfo.net/forum/canon-eos-5d-mk-ii-hd/357960-assessment-5d2-bbc-alan-roberts.html

Unquestionably the 5D delivers a gorgeous image, with some aliasing and moire. The 7D will probably perform comparably. The BBC chart looks scary, but unless you're shooting for the BBC you probably don't need to stress about it too much; instead spend your time looking at actual footage coming from the cameras and see if they meet your expectations and requirements.

Can you please point to any authoritative information that discusses how these cameras work? I haven't found any official statements from anyone.

Binning makes no sense on a Bayer sensor, no sense at all. Now, if it was an RGB Stripe sensor, vertical binning would make perfect sense, but on Bayer it just plain wouldn't work.

It wouldn't be a straight bayer algorithm. But certainly you can derive a pixel value based on many surrounding RGB values.

I recall discussions about binning, but no direct quote. Binning certainly could make"jello" worse.

I don't know how canon video works.

............

It's not "windowing" the sensor. It's not using the central 1920x1080 out of the chip or anything like that. It uses the full size of the chip, but perhaps not all the pixels on the chip.



Which Red does, yes? Crops for a given resolution?

It is possible to bin pixels on chip, as Foveon does.

The AA filter strength is determined by the small pixels used in still photos. A stronger AA would help video, but hurt still.
Maybe, maybe not. If it was using the entire sensor, yes. But if it's actually skipping pixels, then a stronger AA filter might just result in lower overall resolution. An AA filter is basically a "blur" filter that masks out the highest frequencies and tiniest details, but if the video frame is made using pixel-skipping, then it's dependent on those same tiniest details being reassembled into a smaller-sized full frame, so a stronger AA filter might just result in ruining the overall sharpness of the video...

So does red just throw out pixels?
Well, no, not in the same terms that Canon is suspected of doing; Red reads the entire sensor for its full-frame 4K imagery. Their sensor is purpose-designed for video performance, and is designed to be read at full resolution at up to 30fps.

Also, both Red and the 7D overcrank at reduced resolution (double speed means half the pixels). Is this done by skipping lines?
When going into faster frame rates, Red requires smaller frame sizes, because they window into the sensor. You can get faster frame rates in 3K mode, and even faster frame rates in 2K mode, because the 3K and 2K modes don't read the entire chip. However, they don't use the reported pixel-skipping technique either; they just read a smaller "window" off of the chip. The field of view shrinks in 3K mode, and shrinks further in 2K mode, as it's reading a smaller window out of the chip.

Windowing lets them retain the Bayer pattern of the chip and deliver proper color accuracy, etc.

Binning, would be a perhaps ideal way to accomplish lower resolution and higher frame rates, but it can't be done on a single Bayer sensor. Again, these chips are all monochrome; there aren't color sensors. Now, if you had three chips, you could bin them and get faster frame rates, less aliasing, more sensitivity, and less noise... but with a single chip you can't bin red and blue and green all together into one pixel, because the chips don't record color information, they only record brightness information.

It wouldn't be a straight bayer algorithm. But certainly you can derive a pixel value based on many surrounding RGB values.
Only if you've read all those values. Binning doesn't read them individually, it throws the green in with the red and the blue into a big soup, losing the color value for each pixel. Hence why binning would make no sense on a Bayer sensor.

The speculation about Canon using pixel-skipping, and specifically skipping to every 3rd pixel, is that doing so retains the R/G/B/G nature of the Bayer pattern.

Which Red does, yes? Crops for a given resolution?
Yep.

It is possible to bin pixels on chip, as Foveon does.
Yes, but Foveon is, in essence, a three-chip system. It's just three chips stacked in one place, instead of split by a prism. Foveon is not a Bayer pattern. The restriction on binning is specifically because of the Bayer pattern.

RGB Stripe would also be inappropriate for binning horizontally, but could be done vertically.

This has been posted here before:

Canon : Canon Technology | The Minds Behind the Magic (http://www.canon.com/technology/interview/hv10/index.html)

and it is long out of date, since it talks only about the 1G consumer camcorder, the HV-10. But if you go a few pages into the interview, it gives an example of how an EOS DSLR captures 8 fps @ 8 Mpixels. When you combine that information with the rest of skimpy info that Canon has online regarding it's CMOS sensors, you can get an idea as to the viability of the two "theories" espoused in this thread, as to how the 5d2 and the 7d turn large numbers of Mpixels into 1920x1080 (every third row used vs. binning, etc.).

Canon is not too likely to give out its trade secrets, so this is about all we "consumers" are going to find out, at least for now. What other people have tried is searching for the public information regarding the patents of the various Canon R&D employees, such as:

IMAGE PICKUP APPARATUS AND METHOD FOR CONTROLLING THE SAME - Patent (http://www.faqs.org/patents/app/20080297613)

So far, though, we're still stuck with educated guesses as to how the two Canon HDSLRs actually.

If I can find the source that claimed binning, I'll post it. I saw it about two months ago, and was somewhat surprised.

One clue that leads me to believe that we are seeing "creative binning" is the strong color errors on black and white zone plates. If we were to take every third line and filter it properly, color errors would be minimal. Show every third line, and we would see monochrome aliasing. Instead we see strange color artifacts. Maybe this is due to poor filtering in the horizontal direction, but I can't help but think that there is something more going on.

Straight binning would mean adding values together. It's possible that only common color photosites are added, and that they are appropriately scaled - in the case that you add two sites together, divide the result by two with resistors on the sensor itself. Certainly, skipping two of three rows would be faster and simpler.

Anyway, back to the original poster's question, It's pretty clear that (at least, nearly) the entire width of the sensor is being read, and the vertical image is cropped to provide a 16x9 image. The exact reading structure is not known for sure.

Earlier reports claimed that one of every third line was used, but I recall an interview with a Canon engineer where he said that they were binned. That makes sense, as the noise will be lower with binning than with skipping two of three rows.


But the noise isn't lower. In my (unscientific) analysis, the noise looks about the same as taking a JPEG and downsampling it with a nearest neighbor algorithm. It is much noisier than taking a JPEG and downsampling with sinc or any good algorithm.

It's worse in photon shot noise dominated image tones (midtones at low ISO and highlights at high ISO) as well as read noise dominated image toines (midtones at high ISO and extreme shadows at low ISO). If it was binning, the photon-shot-noise-dominated tones would have the same noise and the read noise would be lower than just resampling.

Binning a Bayer CFA sensor would likely result in certain artifacts, but not nearly as bad as what the Canon video has. There are some known techniques for reducing the artifacts in real time, Phase One is probably using them in the binning mode they added through a firmware update a few months ago.

Canon's sRAW2 is more in line what I would like to see with Canon video: the noise and aliasing are worse than a downsampled still, but not nearly as bad as the video currently is. But sRAW only needs to do 4 FPS, not 30.

Here's a credible source: Tim Smith, Product Manager of Digital Imaging at Canon.

Digital Cinema Society - New Streaming (http://digitalcinemasociety.org/content.php?page=Lighting%20Workshop)
(Go partway down the page. There are two QT files of Tim Smith. It's from July 23rd.)

In part 1, about half way through, Tim Smith strongly recommends UDMA cards. He also confirms that they don't pixel skip. They bin them in groups of six to pull down to 1080. He also says Keeping the camera level helps.

He talks about the need to re-clock the chip to get to 24p, and says its unlikely, but not impossible that there could be a firmware update. Then again, he would have bet money that they wouldn't release a manual control firmware update.

Tramm got Canon's attention.

Canon is thinking about cine-style lenses. Some of the development group silos are falling down.

Part 2 shows a Ghost Whisperer webisode, shot back in the auto-only era. He talks about his experience on that shoot. He mentions the 50L, 85L and 16-35L.

Tim recommends leaving HTP on. (I don't, unless the dynamic range of the scene is extreme.)

He's seen the 5D with a PL adapter. He hasn't yet seen a film transfer. When projected large, he expects that you will see lens flaws with cheaper glass.

The best news is that Tim Smith "gets" the needs of videographers. It's not like he's a photo guy who is just now being exposed to video and film.

Anyway, the videos are worth watching for anybody interested in the development of HDSLRs.

Jon:

Very interesting video... First acknowledgment by Canon that I have seen that Tramm's work has caught their attention.

I am not sure how technically competent he is, as he seems to believe file size can be increased by a hack. My understanding is that file size (at least in HiDef) is limited by the 4 gig maximum size. I think the reference to the guy in the guy is Seattle not having hacked that is kind o f interesting, as they seem to be mixing your location, somebody in Europe and Tramm as an all in one deal.... maybe he is referring to linked files when size limit is over reached.

He is pretty clear that 24p won't come out of chip without clock change, but then again, something that technical isn't his forte. Most interesting is the acknowledgment that Canon never saw this camera gettting anywhere near film making...

He's a marketing guy, rather than an engineer. Still, it's good when a marketing person understands use cases and user needs.

Also, as a marketing person, he probably didn't realize that the engineers would probably not want any information about their scanning structure to be hinted at in public. ;)

Anyway, that video is definitely worth watching...

I am not sure how technically competent he is, as he seems to believe file size can be increased by a hack. My understanding is that file size (at least in HiDef) is limited by the 4 gig maximum size.

The problem is that the CF card standard requires that the cards have been formatted with either FAT16 or FAT32 (which the 5d2 uses) file systems. These days, most CF cards use FAT32 and this file system limits individual files to a maximum of 4 GB (HD or SD, there's no difference). The newer versions of Windows use the NTFS files system for its hard drives and they don't have this file size limitation.

But a file system is software. You theoretically can format a CF Card in a number of other ways (NTFS for example) or alternatively, you could produces a proprietary card that fits in a CF slot. But the camera would have to be reprogrammed to read and write to the CF Card using an alternate file system. And once you go this route, then you have to worry whether the CF card would work on a Mac (which can usually read NTFS formatted devices, but not write to them without using some tricks). Another problem is that small capacity CF cards use a lot of overhead (which shows up as unusable storage) when formatted as NTFS. But on bigger capacity cards, this problem is negligible.

BTW, those two Tim Smith videos really formalize the "legend" of the 5d2. Just about everything we know about the camera and its development appears to be correct. I'm also now convinced that the camera bins 6 pixels to get 1920 x 1080 video.

[...] He also confirms that they don't pixel skip. They bin them in groups of six to pull down to 1080.

It looks like a duck, walks like a duck, and quacks like a duck. But Canon Marketing says it is NOT a duck. If Marketing really is telling the truth, and they really are binning all the pixels on the sensor, then they are doing a stupendously terrible job of it, because the noise and moire is as bad as if they were reading only 1/3 rows.

Binning will also cause moire. It will also cause the chroma artifacts that we see. If they just read lines and filtered them properly, there would be no chroma artifacts per line on B&W images, and line skipping wouldn't add any chroma artifacts. As far as the noise goes, I'll take your word for it. I've never done the photo/video noise comparison.

Anyway, mysteries remain...

I expect that a future 1D-series cam will have the read speed and processing to be able to read the photo-sites and filter them properly. Or at least, for $5k+ or whatever they'll charge, it should...

It looks like a duck, walks like a duck, and quacks like a duck. But Canon Marketing says it is NOT a duck. If Marketing really is telling the truth, and they really are binning all the pixels on the sensor, then they are doing a stupendously terrible job of it, because the noise and moire is as bad as if they were reading only 1/3 rows.

There has been more analysis that shows Canon did not tell the truth. They are indeed skipping 2/3 rows:

Re: Update considering Canon's claim of 6 pixel binning: Open Talk Forum: Digital Photography Review (http://forums.dpreview.com/forums/read.asp?forum=1018&message=33697972)

video from DSLRs: every third line used? - Luminous Landscape Forum (http://luminous-landscape.com/forum/index.php?showtopic=38713&view=findpost&p=320968)

I wonder if the marketing guy spoke out of turn. It's unlikely that a marketing person would come up with that off the top of his head.

Given the low noise of the 1D4, maybe that's the binning camera...

Well, it sounded like he'd asked the engineers...

DSPographer is perhaps saying something along a compromise between both camps? He says the magnitude of the horizontal aliasing is consistent with binning three alternate same-color pixels per row (which would mean three out of six, perhaps where the "six pixel bin" idea comes from?)

However, on the vertical, he says the magnitude and the frequencies are consistent with 1-out-of-3 skipping.

But Tim Smith did flatly deny any pixel-skipping...

So maybe it bins three (out of six) together on the horizontal, and only takes one out of three vertically. That would make it effectively reading 1/9 of the sensor, or 2 megapixels out of 18. Which should give it the speed necessary to do 720/60p as well...

EDIT: Now that I go back through that thread, I see that's exactly what DSPographer was suggesting three weeks ago -- binning three consecutive same-color (out of six) on the horz, and skipping 2 out of three on the vert, to create an 1872x1053 image. Then demosaic'ing and scaling to 1920x1080. Which, if it's an 1872x1053 image before demosaic, that would explain its basically 720p res after demosaic (assuming about a 30% res loss due to the demosaic process).

Given the low noise of the 1D4, maybe that's the binning camera...

Yes, that could be it. It will be interesting to see it pointed to a zone plate after it hits the streets.

DSPographer just posted his test images and analysis (http://forums.dpreview.com/forums/read.asp?forum=1019&message=33711632) of the horizontal and vertical frequency sweeps.