FS700 sensor read out?

[Bob Prichard]

How does Sony get an HD image out of its 4K sensor?

Crop? Line skipping? Pixel binning?

Thanks.

[Chris Medico]

The lead money is on pixel binning. It does not appear to line skip. Except at high frame rates. Then it line skips. You can see the aliasing increase between the two readout modes.

For sure does not crop the sensor. Except at the highest frame rates. Then it crops.

[Tim Dashwood]

Sony has said it is pixel binning.

[Noah Yuan-Vogel]

From the look of it, scaling for 1-60fps, binning 120-240fps, cropping+binning 480-960fps

[Alister Chapman]

As this is a 4K camera the optical filter must be able to pass 4K detail.The camera uses pixel binning for 2K almost certainly, hence the aliasing issues. If you simply pixel binned or skipped for HD you would get a lot of aliasing. I believe the sensor must be read at 4K and then down sampled to HD electronically with an electronic low pass filter.

[David Heath]

Let's ask what we know about the camera for sure.

Firstly, for HD, it has a 3840x2160 effective sensor. Given that, then surely the simplest and best way to derive an HD output is to do something very similar to what the Canon C100/300 do - derive R, G, and B values for a pixel directly from 2x2 blocks of photosites. You get a 4:4:4 true 1920x1080 image with minimal effort in terms of processing.

What would be the point of Sony doing a full conventional deBayer, only to then have to do a 4K-HD downscale? I can't believe it would give a better image (the above should give full 1080 resolution) yet would need much more complex processing and likely use much higher power doing it. Why bother?

But the Canon cameras give the same results horizontally and vertically, around 1000 lpph as would be expected. The other thing we know about the FS700 is it's chart resolution performance, and the best examples I've seen are from the tests Adam Wilt did - see top of page High Speed and Low Light with the NEX-FS700 by Adam Wilt

A FS700 fact that has never been fully explained is why the horizontal resolution is so much less (about 800 lpph) than the vertical (about 1000 lpph). The best theory I've heard is that whilst it does do some sort of direct read as the C100/300, it forms two output pixels from a block of photosites 5x2, (effectively wasting 20% of the photosites), then upscales horizontally to get 1920.

That would answer many of the observed facts. Horizontal resolution of about 80% of what may be expected, but more interestingly, colour fringing has been observed on high contrast vertical edges, noticeable with movement. Not enough to normally prove a problem with everyday shooting, and normally only visible via pixel-peeping - but could it back up the theory about sensor readout?

The "how" is one thing. Even if the theory above is correct, the question of "why" remains. Why aren't Sony just doing the same as Canon and making the most of what the sensor should be capable of - even with a simple form of readout?

[Noah Yuan-Vogel]

The method the C300 uses is surprisingly effective given it's extreme simplicity, but it is hardly the best way to get an HD output from QuadHD. Combining bayer quad pixels into one RGB superpixel is undoubtedly going to result in a lot more aliasing than debayering in quadHD and downscaling properly, especially in a camera with an OLPF designed for 4K and not HD.

[Alister Chapman]

Noah is right. Because the FS700 is designed for 4K readout the camera has a 4K OLPF. So at HD if Sony just did a readout like the Canon C300 you would have a lot of unwanted aliasing and moire. If you read the sensor at anything other than 4K there will be aliasing. So for HD Sony need to read out the sensor at 4K and then down convert electronically adding a lowpass filter in the down convert process (which may explain the odd resolution imbalance).

At 2K and up to 240fps the sensor appears to be read by pixel binning, pixel skipping would result in a sensitivity drop which we don't see. The extra aliasing a result of now treating the sensor as a 2K/HD sensor while the OLPF is 4K. Above 240fps then vertical line skipping appears to come into play and above 240fps the read out area is reduced.

[Paul Curtis]

The sensor is 4096 but the 1080p is a 3840 crop out of that. When we get the full 4K signal we can get the full 4096 width.

The 1080p is IMHO a simple scale from the quad HD debayer, you can see this because the scale attempts to sharpen the edges (regardless of detail settings). There are also cases of black pixels near high contrast edges, which can be an artefact from certain scaling algorithms. Plus all the other reasons mentioned already.

cheers
Paul

[David Heath]

Originally Posted by Alister Chapman

Noah is right. Because the FS700 is designed for 4K readout the camera has a 4K OLPF. So at HD if Sony just did a readout like the Canon C300 you would have a lot of unwanted aliasing and moire.

I agree it has a 4K matched OLPF, most likely, but the aliasing will be governed by the sensor characteristics and count - not the output resolution. This is also the case with the C500. It's 4K capable - but will also do HD like the C300, same readout system. It likewise should have an OLPF matched to 4K resolution - but it doesn't therefore mean high levels of aliasing/moire at 1080.

Think of it this way. If the input is lines at 1080 lpph, then for this sensor each white line will cover a 2x2 Bayer block exactly, the following black line will give a full black output for the next Bayer block - no aliasing. Double the frequency and the situation radically changes - a white/black pair to every block, hence the white line covering a red/green photosite pair, the black a blue/green pair - so expect coloured aliasing. Only the red/green photosites will be stimulated.

Now what would be the difference if you did a full debayer? Effectively, as far as aliasing is concerned, not a lot. No blue photosites are receiving light, so as far as the sensor is concerned, it may just as well be illuminated by orange light. Aliasing patterns have far more to do with the relationship between the light pattern and it's frequency and the physical geometry of the chip. A 4K chip will tend to give aliasing most noticeably when the input frequency gets more than about 2000 lpph - when it's 1000 lpph, each line will be so (relatively) coarse as to cover a number of photosites.

If you read the sensor at anything other than 4K there will be aliasing. So for HD Sony need to read out the sensor at 4K and then down convert electronically adding a lowpass filter in the down convert process (which may explain the odd resolution imbalance).

The assumption is that the sensor is fully read - otherwise, yes, there will be aliasing. By the reckoning above it would be equivalent (sort of) to changing the physical geometry of the chip. But deriving the pixel values from the photosite readouts, and how that is done, will have far less effect on aliasing appearance. So I'd maintain that the most likely method of 1080 derivation is not via full deBayering, but rather a more simple method such as the C300 uses.

As far as the readout goes, then I understand that values are read out line by line, so for a given line, it will be a series of values corresponding to R,G,R,G etc for one line, then B,G,B,G etc for the next - a series of discrete values representing site by site. To do "low pass filtration" on such is equivalent to saying that adjacent values are averaged to some extent, after allowing for the colour interleaving, that or dropping one in so many samples.

And that is comparable to what I said before - that on average, two output pixels are formed from a Bayer block measuring 5 horizontally, 2 vertically. Exactly how, I don't know. It could be as simple as dropping every 5th read value (hence every 5th column). It could be that pixels are formed from R,G,R in one row, and G,B in the lower row, then the next from G,R and G,B,G - averaging all values corresponding to the same colour. But physically, that is roughly what "low pass filtration" means in practical terms. More likely is that the averaging (or low pass filtering) is carried out on a more rolling basis.

The evidence for the "two pixels from a 5x2 block" is that the horizontal resolution alias null seems to be around 850lpph. Knowing the sensor geometry, it is highly likely that it bears a mathematical relationship, so 4/5 of 1080 - 864 - is reasonable.

But as I said before, the question is not really "how" (whether you call it low pass filtration, or look at it geometrically), but "why"? Why not just do as Canon do and form a pixel from each 2x2 block? If you're starting from a 3840x2160 Bayer sensor, that's the simple way to do it and still get full 1080 resolution. It seems that Sony are adding complexity and getting a worse end result. Why?

[Wacharapong Chiowanich]

Is the tech employed by Canon in their C300/500 patented? If it's not then I think maybe because Sony at that time already had the chip available that could process the 4K/UHD readout data and downconvert them to targeted outputs so they just decided to utilize the design and availability out of cost considerations or the timing needed to put the camera on the market. Or they just wanted to send some kind of message that their method and implementation, not their competitor's, was the way to do it despite the results. I can't think of any other reasons.

[David Heath]

Originally Posted by David Heath

Now what would be the difference if you did a full debayer? Effectively, as far as aliasing is concerned, not a lot. No blue photosites are receiving light, so as far as the sensor is concerned, it may just as well be illuminated by orange light.

If further evidence of that claim is needed, it's worth looking at tests of the F3 - which we know to have a Bayer sensor of around 2456x1372, and which is fully deBayered.

Look at another of Adam Wilts test charts - Review: Sony NEX-FS100 “Super35” LSS AVCHD Camcorder by Adam Wilt , 3rd one down - for the F3 and on the horizontal and vertical line wedges coloured aliasing can be seen between the 1250 and 1500 lpph circles - in other words, centred on 1372. Interestingly, it's orange/cyan in colour.

This is being caused by the situation I hypothesized above - white/black lines falling on single rows/columns alternately. So, according to phase, the situation will arrive where all white lines must fall on only either R and G photosites - or R and B. In the former case you'll get net orange output, in the latter cyan. As observed. This is with a readout which gets full deBayering, and well illustrates the point that in this respect the aliasing is solely dependent on the geometry of the chip and the input line spacing. You'd get the same effect, at the same place on the chart, if the Canon style decoding was used - though in that case lower luminance resolution, a max of around 685lpph. (Half of 1372)

I'd suggest the main reason we don't see anything similar on the FS700 charts - regardless of deBayering/decoding method - is that it would be predicted to be between 2000 and 2250 lpph - and the charts don't go that fine.

Originally Posted by Wacharapong Chiowanich

Or they just wanted to send some kind of message that their method and implementation, not their competitor's, was the way to do it despite the results. I can't think of any other reasons.

Or it could be as simple as they didn't want a relatively low cost product to equal or outperform higher end models such as the F3..........? :-)

And doing this is simple, keeps the performance at 1080 below the higher end models, but still gets pretty impressive results - way better than any DSLR video, for example? And coupled with the high frame rate modes, and the RAW ability, it's still a very impressive camera.

[Alister Chapman]

But the only method that fits with the way the FS700 works is to read the entire sensor at 4K because the sensitivity and aliasing on the HDMI output and internal HD recordings are the same whether the camera is outputting 4K or HD over the HDSDI. This strongly suggests that the sensor read is the same in both HD and 4K and in order to get a 4K output, you need to read the sensor at 4K.

[David Heath]

Originally Posted by Alister Chapman

But the only method that fits with the way the FS700 works is to read the entire sensor at 4K because the sensitivity and aliasing on the HDMI output and internal HD recordings are the same whether the camera is outputting 4K or HD over the HDSDI.

Well, yes - but that's as I said above: "the aliasing will be governed by the sensor characteristics and count - not the output resolution" - that implies all the photosites are indeed read every frame! The whole point above about theorising about a line pattern falling on alternate rows assumes a full 4K readout.

Consequently, I'd expect to see cyan/orange coloured aliasing on horizontal and vertical lines around the 2000 to 2250 lpph - and to see such in both 4K and HD modes.

This strongly suggests that the sensor read is the same in both HD and 4K and in order to get a 4K output, you need to read the sensor at 4K.

OK. I've reread the thread and think I see where the confusion comes in.

Firstly, I fully agree that at normal frame rates, it's most likely the whole frame is read out - all 3840x2160 photosites every frame. The question is what happens next. In 4K mode, it's my understanding that that all 8 + million data values are just output without further processing to form a RAW 4K recording.

What I *DO* think happens is that to derive the HD output, this 4K data is processed directly to get the HD signal - but it's in a similar manner to how the C300 does it. Forming output values more or less directly from individual colour photosite values.

What I *DON'T* think happens is that this data ever gets conventionally deBayered as such to form a 4K video signal in camera - either in it's own right, or as an intermediate step to produce HD by downconversion.

In an earlier post, you said that "I believe the sensor must be read at 4K and then down sampled to HD electronically with an electronic low pass filter. " A lot depends how to interpret "down sampled to HD electronically". If it means forming a deBayered 4K raster, then downconverting that to HD, then I disagree. For one thing it would mean the designer is making a rod for his own back, it would involve a lot of complicated processing and ....... why bother when 2x2 direct read ("the C300 way") will give full HD resolution much more simply?

But whichever way you look at it, why is this low pass filter applied to the horizontal processing? If the camera was to do a full 4K deBayer, then electronically downconvert, the intermediate (4K) raster should give detail up to about 80% of 2160lpph - about 1728 lpph. So if that is what is being started with, then why does the horizontal resolution end up so low, at about 850lpph? But even "the C300 way", then the simpler processing should still give over 1000lpph (as with the C300) - so why put in this filtration, and why on one axis, not the other?

Finally, look very, very, very closely at the Adam Wilt chart I earlier mentioned - High Speed and Low Light with the NEX-FS700 by Adam Wilt - top of page. And at the vertical lines trumpet. Moving towards the centre, and up to about 850 lpph, the lines are converging, (as on the real chart) - but then they begin to DIVERGE - they are aliases of the true pattern, (which is obviously still converging!) If the mechanism really did do a full 4K deBayer and downconvert, that would not be expected. It implies a Nyquist mechanism at around 850 lpph horizontally - or exactly every 2.5 photosites if you take the figure to be 864.

[Alister Chapman]

The read out of the high speed modes is very different to the standard speed modes whether raw or otherwise. In the high speed modes the camera combines pixels or pixel skips depending on the frame rate, hence the aliasing.

In 4K the camera outputs 4096x2160, not that this makes much difference to the read out I suspect. In raw mode the signal from the sensor is passed to the raw recorder, but there is some slightly odd behaviour as camera gain or changing the gamma curve effects the raw recording level. For maximum dynamic range the camera must be set to 0db and you must use the Slog2 gamma curve.

Aliasing is dependant on the sensor resolution. For the 4K to work the OLPF must pass detail at near 4K, so if you read the sensor either directly or via the DSP by combining pixels 2x2 then in effect the sensor resolution is now 2K so strong aliasing would be expected at 2K and HD, but we don't see strong aliasing in the HD modes. Indeed the 2K raw where the pixels are combined is full of aliasing as you would expect, but the HD is not. You can't just combine pixels without getting a lot of extra aliasing.