Black Gamma and Noise.

[Wayne Zebzda]

I will be shooting a reflective suit made of 3m products (ths very same substance on traffic signs) . The hope is for the suit to look like a 2 dimentional object running down the middle of the road, like flat pixels, tiny squares flatly moving like computer animation in the dark . I would like the lowest noise possible in the darks and have one evening to shoot with my film permit. The reflective material shoots back all the light to the source which is a camera and car head lights.
I will post the results but would appreciate any picture tweaking you think might help
Thanks (Mahalo),
Wayne Z

[Alister Chapman]

I have been looking very closely at the EX noise levels and what can be done with the PP to minimise the noise.

My first and perhaps most significant discovery is that the different STD and Gamma curves all have very different noise levels. The worst is STD1 which is much, much noisier than any of the other Gammas. I found CineGamma 1 to have the least noise.

From what I can tell the camera applies gain after the gamma curve. This has a bearing on the peak white levels and black levels that are recorded. As the gain is increased so is the recorded white clip level, so with gain above 0db it is very easy to record illegal levels.

The Black level setting effects the pedestal. I have found on both my EX1 and EX3 setting black to -4 at 0db gives no pedestal (measured with a scope). You might want to use -3 to be sure of not clipping any blacks.

Crisping sets the level at which detail correction is applied to the image. My trials indicate that increasing the crispening to a positive value makes the camera apply less detail correction to low contrast areas of the image. From my experiments I have decided that a value of +20 to +30 helps control noise in darker areas. It's not a massive difference but it does help.

Detail Frequency changes the width of the detail correction that is applied to the image. Remember that the camera is not clever enough to differentiate between noise and true image detail so detail correction gets applied to both. Personally I prefer to increase the detail frequency to make the edges thinner, this make the noise less obvious and gives a less processed looking image. However you have to be careful as go to high (thin) and you will get twitter, aliasing and stress the codec. I use Frequency at +30 to +40.

Now with the above settings the picture is already looking a little softer, but it is still just a little too sharp for me so I run the Detail at -8 to -10 just to take the edge off.

I don't use black stretch, but Increasing it to a positive value can help if you have large areas shadows but this also brings up the noise levels in dark areas.

Further improvements can be gain by using the Low Key sat control to reduce the chroma saturation in dark parts of the image. Again it's not a big difference but it helps if you run it at -15 it helps control saturation noise in dark images.

While I was looking at all these setting I also noticed that if you use the CineGammas with the STD or HISAT matrix you get a slightly odd colour response curve, this may be contributing to the IR problems, but needs further investigation to confirm. Whatever it seems that using the STD or HISAT matrix with the CineGammas gives some very uneven colour response with big peaks at certain wavelenghths, in particular reds. I found that by using the Cinema Matrix I get a much flatter and more natural response, however the image is much less saturated. So for a natural look with CineGammas I am using Cinema Matrix at +35.

So if you set up a profile using
CineGamma 1
Black -4
Detail -8
Crispening +20
Frequency +30
Cinema Matrix at +35

and then compare that to Standard setting 1 you will see a massive difference in noise. Additionally as CineGamma 1 records to about 109IRE if you use -3db gain you will get a further reduction in noise and also now only have levels going up to 104IRE.

The trade off of this PP is that the overall sensitivity of the camera is reduced by about 1.5 stops over standard. Even if you then apply 6db gain you still have a clean picture. If you need a setup for low light try the above settings but with STD Gamma 4 or CineGamma 4.

Please not that these are just my findings and are not necessarily based on science! But they do seem to work.

[Paul Kellett]

Originally Posted by Alister Chapman

I don't use black stretch, but Increasing it to a positive value can help if you have large areas shadows but this also brings up the noise levels in dark areas.

.

Very informative Alistair, thanks for posting.
When you say black stretch, i presume you mean black gamma yes ?

Thanks.
Paul.

[Alister Chapman]

Sorry, yes black gamma. It has the same effect as black stretch.

[Piotr Wozniacki]

Finally, Alister's confirmed with his insightful post that what I've been doing since the beginning isn't wrong - namely, using Cinema color matrix, rather then Hisat, especially in conjunction with Cine gammas.

Cranked up a little (like Alister's +35), it's always looked better to me than using Hisat color matrix with individual pairs convoluted to fight the dominating Red channel (and I have seen people using PP's with R-G as low as -75 or as high as +75). Sorry, Bill ;-)

[Alister Chapman]

I spent quite a while with a really good 52" monitor (proper monitor, not a TV) and a waveform monitor looking at various settings. I had not really realised before just how uneven and inaccurate chroma response with the CineGammas and both Standard and HiSat matrices is. There is a really big and quite narrow peak in the reds and I do wonder if this is what makes the IR issue so pronounced. As I said this needs to be investigated further. I've never really used the Cinema Matrix as it has always looked very under saturated, but it was almost out of desperation, trying to get a flat chroma response with the CineGammas that I tried it once again. The difference is quite dramatic to say the least. I'm really pleased with these settings as I now have what I regard as a good starting point for any "looks" that I may want to brew using the individual matrix settings.
It started out as an investigation in to reducing noise, which it does quite dramatically compared to the factory default, but I have also found a really pleasing and I think very natural look.

I would be interested to see the results of the BBC's noise evaluation with these settings. The other thing to remember is less noise means less stress on the codec which is always a good thing.

Again I'm more than happy to share my findings, just don't take this as the only way to do things, just a suggestion based on a little (well a lot) of experimentation.

[Wayne Zebzda]

Many thanks Alister!

Wayne Z

[Juan McFarland]

Alister, thank you for taking the time to post your findings and explain them as well. I'm looking forward to trying them out.

In regard to gain structure and recording latitude I have questions and observations related to your findings:
You reported less noise with the minus 3db setting. Based on theory (correct me if I'm wrong -it's usually the case!), signal to noise ratio of the sensors is not improved by lowering the sensor output. A gain reduction will reduce noise, but you also get less signal, the end result is the same amount of noise relative to the signal (although some of that noise may now be crushed into the blacks, so that noise is gone since that part of the image is now black, so in a way I guess this does reduce noise!). You found that the using Cine 1 and -3db had the camera clipping at about 104 IRE*. IF the noise reduction improvement using this scenario is resulting from the noise in the dark areas of the image being hidden in the blacks would it be better to shoot at 0db to maintain maximum (109 IRE) latitude and either compress the black gamma or crush the blacks? This method of preserving maximum latitude up to 109 IRE seems to make more sense to me.

That said:
Do you think the EX1/3 sensors are calibrated to zero db in relation to the 0db gain display indicator, Or did the Sony engineers cheat a bit to make the camera seem faster than it actually is by amplifying the sensor output and claiming the amplified level as "zero" db? This would explain improved s/n ratios at negative gain settings.


And finally, some interesting observations of clip levels relative to combinations of camera gain and gamma settings.

*When minus 3db gain is selected I noticed that only Cine 1 makes the camera clip at 104 IRE.


Observed clipping levels using Cine and Standard gamma settings when -3db is used as follows:

Cine gamma 1 clips at 104 ire (based on Alister's finding using a waveform, and my observations of the lowered histogram graph)
Cine gamma 2 clips at 100 ire (based on this being the 'broadcast safe setting' and my observation of an even lower histogram graph)
Cine gamma 3 clips at 101-103 ire (I'm estimating because I'm using the histogram, no waveform scope available. Histogram graph is maxing out between the 100 and 104 readings previously established.)
Cine gamma 4 clips at 101-103 ire (I'm estimating because I'm using the histogram, no waveform scope available. Histogram graph is maxing out between the 100 and 104 readings previously established.)
Standard gamma settings 1 through 4, (interesting!) all the Standard Gamma settings clip at 109 ire when the camera is -3db!!

[Alister Chapman]

Just quickly as I'm between shoots..

-3d gives less noise and you are correct also less output, but you can compensate for less output by opening up the iris, which does not add noise, that improves the signal to noise.

Cinegamma 2 at -3db limits at 96IRE so you loose headroom.

[Keith Moreau]

Alister, thanks for the great information you've presented here, I'm going to try to digest it and implement it when I have a chance.

I've found that the -3db gain setting reduces the headroom as is indicated on the histogram for most picture profiles. There is a 'brick wall' on the right side with -3db set.

If I can't increase the output level beyond the brick wall it's hard for me to understand how making the EX1 clip at lower levels will would decrease the noise as well, it seems it would have the opposite effect. I've been using the 0db setting for shooting for a while since I realized this and though I haven't done any scientific tests I'm pretty happy with the footage and the increased latitude I get.

[Alister Chapman]

The headroom brick wall exists with any of the Gammas, standard or Cine. You have to remember that anything above 100IRE is not "legal". That is to say that when video that goes above 100% is broadcast, encoded or put on a DVD it may simply get clipped at 100% so any extra information may be lost. Now, how bad that is will depend on the software you encode with, how it behaves above 100IRE, the broadcast chain, the codec, the DVD player, the monitor and many many other things. You might get away with it, or you might not. Best practice is to keep your peak white level at or below 100IRE. So while at first glance you may have more latitude it really depends on how you will work with the material in the post production chain.

Any noise is going to be most apparent in the darker parts of the image. If you can swamp the noise by bring the video brightness above the noise floor then your picture will be cleaner. By reducing the gain to -3db you reduce the noise level and thus make it easier to swamp the noise by opening up the iris to increase the amount of light coming in relative to the noise signal. The small loss of latitude or headroom is not going to make much difference as the top end of the Gamma curve is compressed anyway, either through the use of a knee or Cinegamma. Most of your noise is at lower levels which have higher gain due to the way the gammas work.

If as you say you can't see how reducing the gain can give a reduction in noise look at it the other way. As you increase gain you will see an increase in noise. A like for like correctly exposed shot at +9db gain will be very noisy while a 0db gain shot will not.

[Mark Twittey]

G'day Alister,
I have just ried your PP settings on a small shoot this week, It is very nice very natural looking pictures, not over saturated.
Could you confirm with that PP, what the other settings are?
Colour Correction, Offset White, Black Gamma e.t.c.
Thanks

[Keith Moreau]

Originally Posted by Alister Chapman

The headroom brick wall exists with any of the Gammas, standard or Cine. You have to remember that anything above 100IRE is not "legal". That is to say that when video that goes above 100% is broadcast, encoded or put on a DVD it may simply get clipped at 100% so any extra information may be lost. Now, how bad that is will depend on the software you encode with, how it behaves above 100IRE, the broadcast chain, the codec, the DVD player, the monitor and many many other things. You might get away with it, or you might not. Best practice is to keep your peak white level at or below 100IRE. So while at first glance you may have more latitude it really depends on how you will work with the material in the post production chain.

Any noise is going to be most apparent in the darker parts of the image. If you can swamp the noise by bring the video brightness above the noise floor then your picture will be cleaner. By reducing the gain to -3db you reduce the noise level and thus make it easier to swamp the noise by opening up the iris to increase the amount of light coming in relative to the noise signal. The small loss of latitude or headroom is not going to make much difference as the top end of the Gamma curve is compressed anyway, either through the use of a knee or Cinegamma. Most of your noise is at lower levels which have higher gain due to the way the gammas work.

If as you say you can't see how reducing the gain can give a reduction in noise look at it the other way. As you increase gain you will see an increase in noise. A like for like correctly exposed shot at +9db gain will be very noisy while a 0db gain shot will not.

Thanks for taking the time to reply here. I've observed the brick wall in the histogram to not be present for some PP settings with -3db gain, present in others. I'm actually an electronics engineer so I'm pretty well versed in gain vs signal to noise ratio and I believe your statement about additional positive gain also raising the noise floor (as well as the signal level) is correct, although I'm debating whether negative gain (-3db) will reduce the noise relative to the signal level. My intuition and training suggests that negative gain will reduce the noise floor but also the desired low level signals along with it, for example by adding attenuation to a signal prior to being processed further, which doesn't help reduce the noise floor relative to desired signal.

I also agree that the extra headroom afforded by the 0db setting adds headroom in the non-legal areas of the signal, however since I'm color correcting the signal if necessary to reduce non-legal whites I feel the 0db gain gives me more latitude and choices.

Using the camera's histogram as an example there is also a 'brick wall' on the left side, similar to the right side when using -3b gain. That brick wall doesn't seem to go away for any reason with different PP or gain settings, but the right brick wall does go away with 0db gain settings.

Of course all my theory and instinct becomes irrelevant if the EX's terminology or use of the term 'gain' has a different meaning. Like is '0db' gain labels in the camera are in fact electrically +3db and -3db gain is in fact 'no gain' electrically. I guess actual scientific tests of the noise levels at -3db vs 0db would end the debate and I apologize if this has already taken place previously for the EX camcorders.

Thanks again for your time Alister, this is all very helpful and interesting.

[Bill Ravens]

Good post, Keith. I agree. Reducing gain, typically, drops the signal level without dropping the noise level, which is really a function of the fixed "dark current" of the detector.

I've observed that I CAN move the left hand "brick wall" in the histogram by adjusting the black level.

[Alister Chapman]

Lets look at it a different way. As Bill says the bulk of the noise comes from the sensors dark current . The gain setting controls the amplifiers that amplify the output of the sensors. They are set with a nominal 0db point where the best balance between noise, linear output and dynamic range is thought to be. Assuming 0db actually means no amplification then -3db means attenuation of the sensors output by 3db. If we take the output from the sensor and attenuate it by 3db we will reduce everything, reduce the noise by 3db and reduce the output signal by 3db. However we can increase the output signal of the sensor by increasing the amount of light falling on the sensor simply by opening the iris. Opening the iris is a linear function so even the amount of light in dark areas will be increased, the entire image will be made brighter by opening the iris. Thus the signal to noise is improved as for the same output we are attenuating the noise by 3db. So where you have enough light in hand to reach the desired exposure you can get a better S/N ratio by using negative gain or attenuation of the sensors output. Even if -3db gain is not actually an attenuation, but simply 3db less gain than there is at 0db the same applies, there will be 3db less noise.

I would agree that 0db gives the best dynamic range, but you do have some serious over shoots into illegal levels to deal with in the edit. Given the slightly reduce headroom (but still over 100%) levels with -3db gain and Cinegammas 1,3 and 4 I don't think that the tiny bit of extra lattitude (quarter of a stop perhaps) will always be preferable to less noise. Noise is difficult to get rid of once it's there and a "clean" image will grade better and withstand post production better than a noisy signal. As I have said there is no "right way" or "wrong way" to apply the gammas and gain and different settings will work for different situations.

You can adjust the black level, but you are not adjusting the gain or sensitivity, just changing the level at which the blacks are clipped. Raising the black level will simply raise the raise the brightness of the darkest level that can be recorded. Other than for setting the pedestal correctly I can't really see a great deal of use for the Black Level setting.

Mark: all the other PP settings are at factory default. Glad you like the look.