Rolling-Shutter Artifacts on Large-Sensored Cameras

[J. Stephen McDonald]

I've been studying the various problems that CMOS cameras have when shooting video, such as skew, blur, wobble and flutter. The rolling shutter system of exposure, capture and scanning causes these glitches, but I can't find or resolve for myself the answer to one question: Why are these problems so much worse with cameras with large sensors? The number of pixels may be the same as those with small sensors and the frame-rate can be the same.

My Sony video-shooting cameras with small CMOS sensors (.27-inch diagonal) and a 60p frame-rate, show none of these artifacts, but these problems can be very bad with D-SLRs, when they're panned, tilted or have subjects with fast motion. The new little Sony RX100 photo camera, that shoots 60p video, shows profound skew, wobble and blur when the camera wiggles or is swept around a scene, but its sensor is just .57-inch in actual diagonal measurement. My four-year old Sony HC9 camcorder, with a 1/3-inch type CMOS and a 60i frame-rate, shows only a small bit of occasional flutter with moving subjects, but no skew, blur or wobble.

This specific subject, regarding sensor size and CMOS artifacts, seems completely neglected in the many tutorials I've read on the general subject. I'm hoping that someone here may be able to provide an answer.

[Dave Blackhurst]

I was noticing that on the A65 (could just be delays due to the 24Mp sensor?), it is more prone to RS type issues... got to more tightly control the camera.

You pose an interesting question about the "physics" involved... I'm a bit curious myself as to the answer. Perhaps there's a factor of the smaller consumer type cams having "higher expectations", and so more time/effort goes into minimizing the RS defects? IOW, a "pro" would realize camera and scene control are part of the equation, where "soccer mom" wouldn't and would complain more?

I'll admit that I'm finding the DSC-TX100 to have a very good image quality for such a "tiny" camera and sensor - I've gotten excellent results (as long as the minimal zoom is adequate for the subject... it runs out pretty fast at 4x/100mm-ish equiv...). Oddly, it seems like it was optimized rather differently than the other 2011 cameras, which don't perform as well, although the HX100 was close. Still fiddling with the HX200, so far it seems good for video, but the .jpeg engine is a bit messy, IMO... last year's model almost seems "cleaner" somehow.

The smaller chips don't have the DoF, but there is something "better" about RS handling...

[J. Stephen McDonald]

Thanks for replying, Dave. I can usually count on you to have an open mind about small-sensored as well as expensive large-sensored cameras. Most of the subscribers here don't bother to consider anything that isn't professional-level equipment and lose the useful options that some of the low-priced models present.

I also picked-out a TX100V as a backup camera this year and have found its image quality to be very good. Try setting the still-image size to 16:9 at 2-MP and you'll get 230mm. The quality is surprisingly good, compared to the 2-MP mode on the larger HX-Series cameras. Like the HX200V, it doesn't show any of the rolling-shutter artifacts on its videos. Shooting at 60p, it produces very dependable HD video. I often use it with a steadying-rod attached to the tripod jack.

There must be something other than sensor size involved with the skew and other glitches you see on CMOS video. You'd think it would be minimized on the new Sony RX100 photo camera, with a .57-inch sensor, but it's just as bad as any APS-C or full-frame D-SLR in that regard.

There is a traffic video on my Vimeo album that I shot with a Sony TX100V, just before sunset. Unfortunately, the playback on Vimeo has a lot of stuttering due to the heavy compression. Those who have a free subscription to Vimeo can download the MTS full-quality version and play it with a good program, to see how well this little camera can do with 60p. It was handheld, with a straight steadying-rod mounted in the tripod jack. The link to my Vimeo album is in my Signature line and this video is currently in the first position.

[John McCully]

Steve, I can’t provide an answer regarding sensor size and CMOS artefacts. Skew, wobble and blur have only been problems for me when I over-indulged, and that was some time ago as now I’m too old to get into that kind of mischief anymore. The only time I make large pirouette-like moves is when I take a panoramic (still) shot, a handy feature on most of the recent Sony consumer camera models, which never ceases to amaze and delight; the ultimate wide angle if you will.

In all fairness to the high end pro video shooter I suppose he or she would not be seen dead with a consumer-type camera; imagine the looks you would get if you showed up to shoot a Nike add, or a high society wedding using a Sony HX 200v never mind that this model shoots video that is streets ahead of what was de rigour just 10 years ago. The thing is Aunt Agatha might pull such a camera out of her purse and elbow her way in alongside, and how silly do you look now, especially when you are charging an arm and a leg.

OK, that’s an extreme, but my point is that to be professional I suppose you need to look professional and that means bells and whistles (that will hardly make a whole lot of difference to the DVD you give the client).

I don’t want to hijack your thread but I will mention that the question of large sensor vs small sensor (and 24p vs 60p but we won’t get into that) and the shallow depth of field craze is a fashion thing (right now it’s in the early mini-skirt era where lust rules the day) and often overdone however like any other tool/technique it absolutely has a place in the production of interesting moving pictures, and stills of course.

My impression, and I must admit I have not viewed a lot of footage, is that the new Sony RX100 might be just the ticket to take along in my shirt pocket when I climb Mount Mania come springtime. But I will not do the Highland Fling while filming even though the image stabilization capability is said to be up there with the HX200 (which is remarkably good in my opinion) and if these weary bones are up for it I shall bring along a small tripod (and a small bottle of water - not beer).

Pardon me for being so long-winded, Steve, but the underlying questions you raise about ‘gear’ is of interest and I venture to say that while Mr Hurd’s excellent forum is undoubtedly frequented by lots of extremely over-paid high-end gear-heads, professionals and wanna-bees, I suggest a lot of visitors here are just ordinary people who love to shoot video, like you and me, and learn. I think it’s helpful to make the point, for instance, that in this day and age a $400.00 camera in the hands of an ‘artist’ is capable of shooting exquisite moving pictures better than a camera 10 or 20x that price might deliver in the hands of a klutz (or Aunt Agatha).

Cheers...

[Chris Medico]

The problem is caused by data rate. The pixel density can have a big affect on the problem but sensor size will not. How quickly the data is read off the chip determines how severe those artifacts are. The faster you read the data off the imager the less prominent those artifacts are. High pixel count imagers will demand more data capacity than lower pixel count imagers. Tricks like line skipping can reduce the data demand but that leads to other issues such as ailaising which we see plenty of with the dSLRs.

Moving data around faster requires electronics that are clocked at higher frequencies and/or more capable processors. High speed electronics and powerful processors may not be in the design due to power constraints (current demand and heat generated) or cost constraints.

As customers we need to continue to voice our wants for these artifacts to be reduced. The Canon C300 has shown that even with a rolling shutter skew can be minimized.

[J. Stephen McDonald]

Chris, I found a good C300 short on Vimeo and downloaded it. It used 10.5 Mbps with an MOV file. That's a much smaller bit-rate than the 35 or 50 Mbps in which it was probably shot, but played on Splash Lite, it showed only a minimum of CMOS artifacts. Just a bit of skew at the corners with the fastest pan and an occasional flutter in background objects. You'd never have noticed them if you were paying attention to the woman walking through the scene, as was obviously intended. But, this camera is optimized for video and the body costs $16 G. I wonder if the improvements for producing smooth video will make their way into CMOS cameras of much lower cost?

This article makes an interesting read about its features. It's got 8.3-MP and I'll bet it delivers very good still-captures from its video. What's special is the way it samples pixels, that is the equivalent of using a 3-sensor imaging block.
http://www.dvinfo.net/article/acquis...l-cameras.html

Chris, thanks very much for your explanation. So how do our sub-$500. Sony cameras we've discussed, that have 18-MP, shoot unflawed 60p? Obviously, they don't use all the pixels for video, but enough to cause problems, you'd think. Is each line scanned only every fourth frame or some such method? Based on the equivalent focal-length you get in the Active EIS mode (1,250mm), it would appear that about a 10-MP inner sector of the sensor is used for video at full zoom and about 13-MP in the standard OIS mode.

John, as usual I am in agreement with your position on skill versus expensive equipment. The fact that the C300 can pack so many features and options into a fairly small size is encouraging. If I had a good manual and access to a user's forum about it, I might be able to make it work on the basic level in a couple of months. I wonder how it would look and work on my shoulder-mount? Just daydreaming.

[Dave Blackhurst]

It's not data rate - I shoot the highest rate AVCHD (28Mbps by Sony nomenclature, their consumer software chokes on it, but Vegas and Windows seem to play it back fine...), 1080/60p on everything from the TX100 to the A65... IF it were only data rate, then you'd expect the same issues. BUT you definitely must handle the bigger A65 with a bit more stable shooting technique...

I'll venture a WAG at one possible cause - perhaps it's a matter of "mass"? The sensor block and stabilization mechanisms MUST be larger, and maybe because of that are more prone to accelleration/decelleration issues? I doubt it's "electronic", but maybe the physics also dictate a longer time as the bits must travel further on the "larger" sensor... longer to "read" the whole sensor? Seems like it shouldn't matter, but the bigger APS-C sensor is many times larger than the little DSC chips.


And for the insane cheap price, the TX100 is a great "pocket" cam... originally around $400, they blew 'em out at around $250, and can be had for less on eBay in new condition... great bang/$ ratio, I had a couple already to encourage photo and video skills in the kids (got one young guy who is a rock when shooting video I WISH I could hold a frame the way he can!!). Just added a couple more like new ones for slightly more than a song, just because they make a handy "extra angle" cam as long as the action is not too far from the shooter! I've experimented with "digital" zoom, and you can get almost acceptable results up to around "8x" (4x optical, with "digital doubling") Sony is now marketing this as a "feature" and calling it "Clear Image Zoom"...

[Chris Medico]

Originally Posted by Dave Blackhurst

It's not data rate - I shoot the highest rate AVCHD (28Mbps by Sony nomenclature, their consumer software chokes on it, but Vegas and Windows seem to play it back fine...), 1080/60p on everything from the TX100 to the A65... IF it were only data rate, then you'd expect the same issues. BUT you definitely must handle the bigger A65 with a bit more stable shooting technique...

Let me be more specific. It is not the data rate of the recording medium but the data rate at which the imager data is read into the frame buffer. This is the issue with skew. The faster the data can be read from the imager itself the less rolling shutter artifacts you will see.

[Steve Game]

Ultimately, CMOS sensors fitted to better cameras will have a 'global read', where sufficient parallelism will enable the whole image to be read to the video processing circuits in much less time than vertical scan, i.e. in a similar time to the CCD chip capture period. That's why CCD cameras do not normally suffer from RS artifacts.

[Dave Blackhurst]

Originally Posted by Chris Medico

Let me be more specific. It is not the data rate of the recording medium but the data rate at which the imager data is read into the frame buffer. This is the issue with skew. The faster the data can be read from the imager itself the less rolling shutter artifacts you will see.

OK, so that still leaves a bit of a "mystery" - Steve is reporting the same thing I noticed myself... that the "CMOS artfacting" present in a smaller camera from Sony with a small sensor is far less noticeable than in a higher end camera from the same manufacturer with a bigger sensor. I need to experiment with it further to gauge exactly how big a problem it might be, as I'd like to shoot the A65 more.

I understand that it's the delay time between the read from the "top" of the sensor and the bottom that causes all the "fun" - I shot in high winds the other day (on tripods), and there's more jiggles than a jello festival... although the cameras managed to compensate surprisingly well in the key center zone.

Ideally a global read is the obvious "answer", but I gather CMOS doesn't accomodate that for some reason with current processing methods.

[J. Stephen McDonald]

I'm glad to see some discussion has developed about the rolling-shutter artifacts here. I'm repeating a few things to emphasize a point: My Sony HC9, an HDV camcorder introduced more than four years ago, has a 1/3-inch type CMOS. According to size-type charts, the actual diagonal sensor size is just .15-inch-----less than 1/6th-inch. It scans at 60i in a 16:9 sector of a 4:3 sensor, which is even smaller. With fast panning, it may occasionally show a hint of flutter, but no other rolling-shutter artifacts at all. The new Sony RX100 photo camera, with a 1-inch type CMOS (.57-inch actual diagonal size on a 3:2 sensor), has skew that is profound when panning or tilting. When jiggled by an unsteady shooting hand, the blur and wobble is bad.

As mentioned by others, the newer Sony cameras with .27-inch diagonal sensors also show no RS artifacts when in 60p mode, but do flutter a bit in 60i. So if the data rate and the delay of it reaching the signal buffer is the cause of skew and other artifacts, why do these little, inexpensive sensors and processors do so well? The problem seems to be something related to sensor size. This is my experience with many different types of CMOS cameras and camcorders. But if data-rate inadequacy actually is the culprit in RS artifacts, at what point could this be improved?

The expensive Canon C300 has a large sensor and has eliminated most of the rolling-shutter artifacts, but not all, as I said previously. It has a different scanning method of grabbing two green, one red and one blue pixels in groups, as though it was using a 3-sensor imaging block. Does this have something to do with its reduction of RS problems? What is different about the scanning and registry time-element in the small-sensored cameras? Analyzing these things might produce the answer I'm seeking.

Note to Dave: I got my TX100V direct from Sony for $200. in March. A solid little model-specific bag came free for purchasing a 3-year extended warranty for $35.

[Dave Blackhurst]

Yep, the TX100's on closeout might be the best bargain ever from Sony... I've got a couple of the other TX series, but they aren't as well optimized (TX10 for "worst case" shooting, and a 55 I picked up for dirt cheap and decided it was the first thing that ever fit in that stupid jeans pocket that's good for nothing else, meaning it's easy to have along all the time!).

Still wondering what they were thinking with the TX200 though... might be some deals on those when they discover "camera bling" doesn't sell well! Collected all FOUR of the 100's for less than the price of the 200 by patient buying... cheap extra angle cams and for the kids to learn on!

I'm sure there is some physics affecting the bigger chips, just not sure how or in what way - in theory you wouldn't expect the longer distances that signals must travel to have an effect, but then again, we do know that the big sensors have heat related issues fairly often as well, and that's due to scurrying electrical signals creating heat... for now, I'm going with there's a bit more of a delay introduced in the larger area...

[Chris Medico]

Lets look at a quick comparison with your HC9 and a Canon 7d.

The HC9 has a 3.2mp imager. It doesn't use all of it but for comparison sake lets say it does. That means if you are shooting 24fps the camera needs to offload the data from about 77 million sample sites every second. It then needs to convert all that luminance data into a full color video stream. All of the processors in that camera are purpose built for the exact jobs they are doing.

If we look at the same info from a Canon 7d each row of photosites from the imager will produce about 5k samples. Multiply that by 1080 lines and you get 5.4mp for each frame. Times 24 fps and it comes out to 130 million samples to data crunch each second. Right there you can see the 7d has a harder job to do due to the sheer difference in data being generated by the imager. This is what I'm talking about when I say that the problem comes from data rate from the imager.

The HC9 has processors that are purpose built for video. The 7d does not. In the 7d it is processing video with silicon that was designed to take still images and turn them into jpegs at 8fps or less. The processors in the 7d are not at all efficient when processing video at 24 or more fps. That is why the compression method on those cameras is relatively poor. They just don't have the processor horsepower to handle video well.

They are remarkable for what they do and at the price point they can do it at. Expecting them to have the same imager performance as a purpose built video camera is more than a little unfair. Maybe the new 4k EOS camera will have the processor needed to get a low latency video image. I don't know.

As a second item of worthy mention is that I have a Sony V1U. It has 1/4" imagers. Its rolling shutter artifacts are equal if not a little bit more severe than the Sony F3 I also have. That suggests that its not sensor size alone that is the problem.

One additional interesting thing to note. I just received a Sony FS700. When you put that camera in high speed mode it seems to have zero rolling shutter artifacts but it does have some when shooting regular rate 1080p video. I am planning to do some tests to compare the F3 to the FS700. That should be interesting to see if they are different in the rolling shutter department.

[J. Stephen McDonald]

Maybe the high-speed frame-rate is part of the reason the RS artifacts don't show. The capture and registry functions may go into afterburner mode for high-speed and change everything.

The 60p mode eliminates flutter on the TX100V and HX100V/HX200V models.The RX100 video I saw that showed bad skew and wobble was labeled as 60p, but maybe it was actually shot in 60i. There's no way to tell on Vimeo as they convert everything to 24p for playback. There are very few examples of video available from the RX100 and maybe when I can download something I know for sure was shot in 60p, I may see different RS effects.

[J. Stephen McDonald]

I've watched numerous 60p videos from the new RX100 in the last few days and many of them don't seem to show rolling-shutter artifacts nearly as bad as I saw in some clips last week. There's a hint of a little warping at the sides during panning, but it's not so bad that you'd notice it if you weren't looking for it. If handled properly, this tiny camera might work fairly well for video, even though it has a maximum of 120mm focal-length.