Noise comparison: 35/4:2:0 vs. 180/4:2:2

[Dan Keaton]

Dear Eric,

Based on the Video Clarity Test alone, the I-Frame Only mode beats Long-GOP, but please read the following.

We tested the nanoFlash with a Video Clarity System. I will provide the raw numbers below.

In our testing, we used a representive, high-quality video stream, with lots of motion and detail, video from a football game. This video is feed into the nanoFlash, then recorded, then played back into Video Clarity, which then analyzes the video.

A numeric score is then obtained. Lower numeric scores are better.


While the Video Clarity system is very good, it can not duplicate your actual shooting conditions. For example, not everyone shoots football games. In order to keep out testing time to a reaonable level, we did not test with each and very one of the various video test streams that the Video Clarity system provides.

For example, you use an entirely different camera than Piotr and the noise level in your images would be different.

With the nanoFlash, you can easily run a test using the two modes, and determine what is best for your shooting conditions.

We do feel that if one is providing excessively detailed, or very noisy images, to the nanoFlash, that I-Frame Only will provide a better image than Long-GOP at 140 Mbps or higher. At 100 Mbps is it very close, and one should run their own tests.


Here are the raw numbers from Mike Schell, for the Football video test. Mike ran the tests.

Here’s the I-Frame vs Long-GOP comparison for the Luma data
Note: the chroma is always so much better than luma in the video clarity system.
This may seem counter-intuitive, but the Video Clarity system is based on Human Visual Perception and our eyes are better with luma than with chroma, so we are showing the results of Luma.

Lower numbers = better quality:

50I = 3.40
50L = 2.97 Long-GOP Better

100I = 1.953 I-Frame slightly better
100L = 2.043

140I = 1.536 I-Frame slightly better
140L = 1.686

180I = 1.277 I-Frame better
180L = 1.448

For the absolute best results, for your camera, for your type of shooting, we recommend running a quick test for yourself, using the bit-rate that you want to use.

Since you will probably only use a few bit rates, then testing the difference between the two modes is not a lengthy process.

[Jeff Silverman]

Dan,

Based on your numbers why do you recommend L-GOP at 100Mbs? If I-frame is a more compatible format (in MXF for example at 100) and slightly better quality why would you use L-GOP.

Also this information seems contrary to what CD has long advised. I had been led to believe that 100 L-GOP was definitely better quality than I-frame at that bit rate. I-frame was supposedly a less efficient approach, and maybe it is at 50.

Mike Schell said this last year:

"Long-GOP is typically 2X to 3X more efficient than I-Frame only. So 100 Mbps Long-GOP = 200 to 300 Mbps I-Frame only. There is a significant improvement with Long-GOP recording."

Can you clarify this information?

Jeff

[Dan Keaton]

Dear Jeff,

This is the first time I had actually seen the raw numbers.

For 100 Mbps, the difference is 0.09, which is a very small difference.

Our earlier testing had shown Long-GOP to be better.

I have never recommended I-Frame Only at 100 Mbps. I have recommended 140 Mbps I-Frame Only before.

I was also surprised.

Before I would switch from 100 Mbps Long-GOP to 100 Mbps I-Frame Only, I would run some tests.

It could be that if we had run all of the different video test suites (different videos), then we may have found some more differences.

I think it would be great if some of our users, run some tests and show some real world examples.

They may either confirm the Video Clarity tests, or not. The results may be very subjective.

[Gints Klimanis]

I collected a number of bit-rate/MPEG format clips of natural subjects such as water streams and evergreen branches which are generally lower in contrast than football games. 220 MBps I-Frame compared to LongGOP140 in noise levels, and there was a visible reduction in macroblock artifacts between 100 and 140 LongGOP. I-Frame140 was visibly noisier (fine grain) than LongGP140. That is not to say that there is anything wrong with the Nanoflash - only that the bitrate, high as it may seem, is inadequate for the detail in the scene.

SxS is crunchy in high motion areas, and I don't think that increase in noise at higher bitrates for Nanos I-Frame and LongGOP is worse than the decrease in blocky artifacts. For my tests, LongGOP140 had about the same level of noise as I-Frame100. This noise/detail modulation is easy to see on a 24" 1920x1200 Apple Cinema display.

I just hooked up my Nano to my 52" Sony Bravia and looped the files until I became familiar with them. Also, I watched frame grabs from each. It is difficult to see the difference between the files or frame grabs in these examples on an Apple 24", but the Bravia easily reveals the blocky artifact. I'm viewing the 24" display from 6" and the 52" display at about 24" .

If anyone is interested in a set of 5-10 second clips and frame grabs, I'll post them.

[Gints Klimanis]

Originally Posted by Dan Keaton

Before I would switch from 100 Mbps Long-GOP to 100 Mbps I-Frame Only, I would run some tests.

Here are a few of my tests shot at f/5.6 with shutter around 1/125 and gain=0dB with Sony EX1. Bark was chosen because of the undulating sun reflected from nearby water. Evergreen was chosen for complexity under slight movement. Wide fountain was a compression buster with moving tall grass, several water patterns and background grass. The compression type is in the file name as well as in the (crappy) audio of most files. My Nanoflash started recording with a latency of a few seconds with the latest firmware.

Clips (Nano IFrame100, I-Frame140, I-Frame220, LongGOP100, LongGOP140, SxS ) and Frame grabs here :
Index of /Videos/Nanoflash/Test_July11

Although it is difficult to see the difference on a 24" 1920x1200 monitor, a ~52" 1080p TV will make the differences very obvious. I use the Playlist feature of VLC media player for fast switching between clips.

[Billy Steinberg]

As long as this thread has sort of morphed into evaluating the Sony MPEG-2 hardware codec used in the nano, I wanted to add my 2¢ again. (Even though my earlier comments were not received well...).

First, I think this is really where the discussion belongs, not where it started, and I'm happy about that. With the exception of reliability, the effects of starving the codec used are probably the most important decision about whether the nano is suitable for ones needs. Personally, I find the Sony MPEG-2 hardware codec truly outstanding, particularly due to CD allowing us to set whatever bit rate and compression mode we feel appropriate.

Instead of comparing nano clips to pristine uncompressed clips, (almost) everyone is comparing the nano clips to the native Sony EX1 clips. While this test is very meaningful to those of you who only shoot with Sony EX1 cameras, it barely begins to tell the story about the good and bad parts of the Sony MPEG-2 hardware codec used in the nano. While I believe that the Sony EX1 gives a tremendous bang for the buck, and would recommend it (when used with a nano!) without hesitation to those that ask me about a camera in that price range, the camera section of the EX1 is by no means a source of pristine video. I've wondered all along if those making comparison tests are mostly coming to wrong conclusions, and making the wrong tests. It seems to me that most of the testing just reveals that the EX1's built in codec sucks. It would be really useful (and revealing) if someone with an EX1 could record a little video to an SRW or uncompressed to a computer/drive, and compare THAT to the nano's recording of the same thing.

If all you want is to figure out what nano codec settings to use when you're shooting with a Sony EX1, then fine; your observations are relevant to your needs. If you want to figure out what the strengths and limitations of the nano and its codec are, then you really need to (1) evaluate the nano output compared to an uncompressed signal (or at least a signal from a top of the line Sony SRW video tape recorder), and (2) you need to use source material from a MUCH better camera than the Sony EX1. I'm not saying you shouldn't evaluate the nano using an EX1, just that you shouldn't evaluate the nano using ONLY an EX1.

[begin part that always pisses everyone off]
I wish I could upload a few clips of stuff I've shot and recorded with the nano (at 100Mb-Long GOP mostly), to show how utterly transparent the recording is, but they're not mine to upload. The video comes out of $100,000+ cameras, via fiber, as HD-SDI, and each camera has been extensively setup before and during the shooting, including gamma tables and noise reduction, to make the best pictures possible under the shooting conditions. The monitoring is done in a dark room, at a 12" viewing distance, using the very best 20" Sony CRT broadcast monitors, which are also calibrated before the shoot. (At home, I watch the files using an Apple 30" Cinema Display, which is pretty good, but not in the same league as the CRT monitor I use at work). You also need to make sure the monitor you're using to evaluate the pictures is not negatively effecting what you see. Is that 50" LCD displaying pixel for pixel, for example, or is it scaling the picture before it displays it? Is it adding edge detail to the picture, and the noise in the picture, which won't show up in the mushy 35/4:2:0 EX1 codec, but will show up in the nano codec or uncompressed video?
[end part that pisses everyone off]

So I reiterate; if you really want to evaluate the nano codec, you have to compare it's output to its input, and you have to use more than just a Sony EX1 as the source. Comparing the nano codec and the Sony EX1's (relatively crappy) built in codec is a misleading and fruitless endeavor, unless all you're looking for is to decide whether to buy a nano for your EX1.

(And lest you feel otherwise, I have no illusions that the nano set to 100Mb-Long GOP puts out the same picture as I'm feeding it. It is VERY close though. I have yet to see a real macroblock, just a bit more noise (probably due to the 8-bit reduction). I have some concert footage with strobes going off and saturated lights shining through smoke that holds up just fine. There is no tearing or macroblocking when the strobes go off, and the moving light beams show full gradation of color and intensity with no banding at all. If I get a chance, I'll make some 180Mb-I Frame recordings and see how much apparent difference there is when shooting with ridiculously expensive cameras, but I'm pretty sure that any improvements I see won't be worth the reduction in recording time.)

Billy

[Gints Klimanis]

Billy Steinberg wrote "unless all you're looking for is to decide whether to buy a nano for your EX1."

That, and which Nano settings improve my video footage that is submitted for broadcast. CD is probably very happy about their low-end market and may comment on the percentage of their sales to EX1-class users. I bought the Nanoflash on faith-in-concept because there were nearly no clips or frame grabs available to view. To my knowledge, there are *zero* clips or framegrabs comparing uncompressed data from any camera made available to the Nano/XDR community. If you have the gear, please consider uploading some of that video or at least a pair of grabs of the same frame for our education.

" just a bit more noise (probably due to the 8-bit reduction)"

No, the noise is more significant than the least significant bit. Surely, you will be convinced if you step through your footage frame by frame and notice spatial detail modulation in the mid-tones.

You make a good point about monitor display capabilities, but the answer is simple in this case : it is very easy to see clusters of 16x16 pixel macroblocking artifacts in 1080p video or stills on the 52" 1080p LCD TV and harder to see them on a smaller LCD unless they cover large screen areas. These blocky artifacts are much easier to find if a frame is viewed at 200% magnification on the smaller 24" Apple Cinema monitor. The changes in noise levels among clips of different bitrates are also obvious and seem to scale inversely to the bitrate. This noise appears to be mostly detail modulation rather than sensor noise in most cases.

"I'll make some 180Mb-I Frame recordings"

This thread, Piotr made a point that additional noise was noticed for Long GOP at 140 MBps and higher. 100MBps didn't show a great fluctuation of detail between the I-Frame and the successive P-frame. Please let us know what you see.

[Eric Liner]

"I have yet to see a real macroblock, just a bit more noise (probably due to the 8-bit reduction)."

I'm inclined to agree with Billy on this one...I've not yet seen a macro-block artifact with the nano, even at 100mbps. Even when I've accidentally left the unit recording while a swing the camera around or walk to a different set up--I get totally clean garbage :-).

I do think there might be an increase in noise, but I can't confirm without further testing...and I'll stick to opinion that 220-I frame is noticeably better than any bit rate beneath it (assuming the material calls for it).

In response to Dan's reply...thanks for providing those numbers. While I understand the confusion surrounding whether or not to use Long-GOP or I-frame, I'm glad to see that there are advantages to using increasingly higher bit rate Long-GOP modes (i.e. 180 is better than 100). Your early posts read as though you were suggesting otherwise.

Bottom line (IMHO) is you can't get something for nothing. If you want the highest possible quality from the Nano you need to be prepared to invest in larger, faster cards and larger storage so that you can record 220 and above. If you want to have the highest quality from your camera, you need to invest in a better camera and lenses.

Cheers,

[Julio Veas P.]

I have been capturing uncompressed 8 bit 422 with a blacmagic decklink hd extreme, from the EX1, if certainly there is a difference in quality I am not impressed with the results, now for sure it´s not the capturing card, but the camera, the EX´s are great cameras for the price, but they are noisy cameras. This weekend will be using my recently arrived nano, so I can make comparision with uncompressed from the same camera. but I am sure that the achilles heel is the camera head and not the codec.

[Gints Klimanis]

Originally Posted by Eric Liner

While I understand the confusion surrounding whether or not to use Long-GOP or I-frame, I'm glad to see that there are advantages to using increasingly higher bit rate Long-GOP modes (i.e. 180 is better than 100).

There is additional confusion as to whether detail modulation due to I-Frame and P-Frame quality differences worsens as the bitrate for LongGOP increases above 100 MBps. When played at the correct speed, this noise appears to be at the mid-tone level but the cause is not obvious until time is spent stepping through the material frame by frame. I can pick out better frames usually every 6 frames (sometimes 7) and am wondering why I see this for group length = 15 for an NTSC system. Camera is Sony EX1. Frame Grabber is Sony Vegas 9e .

[Tom Roper]

Originally Posted by Gints Klimanis

...it is very easy to see clusters of 16x16 pixel macroblocking artifacts in 1080p video or stills on the 52" 1080p LCD TV and harder to see them on a smaller LCD unless they cover large screen areas. These blocky artifacts are much easier to find if a frame is viewed at 200% magnification on the smaller 24" Apple Cinema monitor. The changes in noise levels among clips of different bitrates are also obvious and seem to scale inversely to the bitrate. This noise appears to be mostly detail modulation rather than sensor noise in most cases.

"I'll make some 180Mb-I Frame recordings"

This thread, Piotr made a point that additional noise was noticed for Long GOP at 140 MBps and higher. 100MBps didn't show a great fluctuation of detail between the I-Frame and the successive P-frame. Please let us know what you see.

1.) When it's said that noise levels seem to scale inversely to the bitrate, that means as bitrate goes up, noise goes down. That's what I would expect but Piotr's point is the opposite.

2.) Kind of hate to jump into the fire with this, but I don't trust Vegas for viewing noise levels on any size monitor. Has it been tried to view stills on the 52 inch monitor just connected to the Nanoflash itself via it's HDMI connector?

3.) I understand my comments at this stage may be unwelcome, particularly since I don't have the EX1 anymore since I use the PMW350 instead. But my recollections from the EX1 were that it was a lot less noisy than is being assumed now. And I can't offer any more than that, since I have used 100 mbps Nanoflash setting exclusively, the videos have been very clean and noiseless, but I don't think it's just because I don't use the EX1. Again, I'm suspicious of Vegas. $0.02

[Gints Klimanis]

Tom, we could suspect Vegas. However, I see the same type of artifacts when playing the clips in VLC Media Player and the Nano to the large TV via HDMI. I'll spend some time with Sony ClipBrowser.

[Piotr Wozniacki]

Tom, Gints, Dan at al.

Thank you all very much for your input to this thread.

I have not posted for quite a long time, but believe me I have taken every opportunity between my jobs (that couldn't afford experimenting) to see into the issue even deeper, and have come to the conclusion that:

- as far as the noise is considered (and especially those noise fluctuations between frames), the Long-GoP format is inferior to I-frame at anything above 100 Mbps.

At 140 Mbps and up, I-frame is much quieter than Long-GoP.

And believe me, I made sure that neither Vegas, nor my particularly monitoring device, influence this conclusion.

Disclaimer: all this thread of mine, as well as this (hopefully final) conclusion, applies to the 1/2" EX series cameras which unfortunately suffer from quite an excessive noise in the first place. Anyone using the nanoFlash on cameras with less inherent noise (read: larger S/N ratio, usually achieved with larger imagers), may find their results differ.

[Gints Klimanis]

Thank you for starting this thread, Piotr. As a result of your instigation and my successive anecdotal experiments, I'm upgrading from 140 MBps LongGOP to 220 MBps I-Frame only. It is a shock that nearly 7x the native Sony EX1 35 MBps data rate is needed for adequate results in mostly handheld shooting, but so be it.

For tripod shooting, I was surprised that 140 MBps LongGOP showed a different and higher noise modulation than 100 MBps LongGOP. Now that I'm over the data rate shock, the Nano has proven to be a great path for ~$6000 camera owners to significantly improve video and frame grabs.

Also, I have since learned that my consumer TV (Sony Bravia 52" , 1st gen) unfairly amplifies macro-block edges even with all menu sharpening/noise reduction set to neutral or off. Still, that has been a a valuable viewing tool to quickly which parts of the video are compressed inadequately. Thanks to Eric Liner for asking that I pay attention to that.

Piotr wrote "1/2" EX series cameras which unfortunately suffer from quite an excessive noise in the first place. "

We need to examine uncompressed video from the EX1 HD-SDI stream to conclude that this is the root of the problem.

[Piotr Wozniacki]

You will probably think I'm out of my mind, but I cannot just hide from the public what I have just find out. I only hope I will receive a constructive answer from Convergent Design.

So far, what I have been looking for in the images generated by my nanoFlash at various settings has been the noise. Typically, on the EX cameras it's most prominent in low light, or on shaded areas (generally, on large uniform color areas exposed below some 40 IRE) - this is what I was inspecting.

After settling with the conclusion the 140 Mbps I-Frame format is the best compromise for my camera, typical usage etc. from the viewpoint of noise, today I discovered - much to my shock - that the WHOLE picture is buzzing with somewhat I would not call noise (as it occurs even on very bright areas), but a sort of grain. When this shimmering grain's magnitude is comparable with very fine detail (like tree foliage in a distance), the grain is spoiling the detail definition and is very disturbing.

This is true of all my I-frame clips of up to 140 Mbps, and makes them much lower quality than my EX1's internal encoding (L-GoP, 35 Mbps).

Dan and Mike, I really am shocked and disappointed. I'm back in the point where I have to question my nanoFlash and/or my camera's HD-SDI working up to specs!

Please explain the nature of what I'm seeing. If this is how recording to the nanoFlash is actually supposed to look like, I must say - with a deep and sincere disappointment - that this $3,000 device is reduced now to just a backup device, the simultaneous recording to the broadcast-accepted, 50 Mbps 4:2:2 L-GoP being the only format where the advantages outweigh the shortcomings, when compared to my EX1's internal recording...

Tomorrow I will run another series of tests to compare the camera codec with the nanoFlash I-frame at higher bitrates. Hopefully, the grain is not as disturbing as it is at 140 Mbps.