abrupt highlights clipping

Bob Grant · February 6th, 2008, 09:55 PM

I just tried a simple test with the EX1, hasn't been out of it's box today :)

Lamp with a diffused shade again a dark wall. Camera in manual, lamp filling around half the frame. EX1 histogram shows a nice lump in the middle, nothing anywhere near clipping.
Zoom in so the light is now full frame, the lump moves to the right of the histogram, looks like I'm getting pretty close to clipping. Zoom all the way into the slightly hotter middle of the lamp and on comes the Too Bright warning. Note I have not changed exposure while zooming in.

This is very different to how the scopes in Vegas work. The waveform display is the sum of all lines. If I'd recorded what I shot above and checked it with the Vegas scopes I'd see a lump in the middle of the waveform that was clipped. As I zoomed in the lump would get broader.

I just simulated the same kind of shot using Vegas generated media. Using Vegas's Histogram metering I get much the same thing. It's almost impossible to detect clipping with a histogram. In fact from what I'm seeing with histograms the more clipping there is in the shot the less noticeable it is in the histogram. With a whole frame clipped you just see a tiny line on the RH side of the display.

This article here:
http://www.luminous-landscape.com/tu...stograms.shtml
gives a detailed explaination of what a histogram shows. About half way down the page there's an example of a scene very similar to the one causing all this grief.

Michael H. Stevens · February 7th, 2008, 01:00 AM

I started a thread on this in the Vegas forum. An EX1 histogram that looks like it goes from 10 to 90 put into Vegas clips like it wants to go to 120. By this I mean extrapolate the shape of the histogram into the clip area that Vegas scope cut of. AND the Vegas scopes clip at the BOTTOM END TO! Like the blacks want to go to -20!

Also Piort I DID give my opinion as to what is causing this much earlier in this thread and no one commented on it so maybe everyone thinks I am wrong, but here is my take again. The EX1 gammas are not straight line or even a partial exponential of a log of a straight line, I think it is programmed to have the Gamma changes with luminosity. The area behind the trees has a lower luminosity (because the trees block the light) and the gamma drops in that area which brings the blown white back into the blue. I think this how the camera gets its enormous exposure latitude and why the Vegas scopes get stretched. JMO as they say.

Piotr Wozniacki · February 7th, 2008, 02:47 AM

Quote:

Originally Posted by Michael H. Stevens

The area behind the trees has a lower luminosity (because the trees block the light) and the gamma drops in that area which brings the blown white back into the blue. I think this how the camera gets its enormous exposure latitude and why the Vegas scopes get stretched. JMO as they say.

This explanation is the only one that can be thought of, Michael - and probably this is the reason nobody commented on it when you first suggested it. Assuming now that this is a byproduct of the "enormous EX1 latitude", one should simply be aware of it and, in backlit situations, either use cine gammas, or modify the Knee point/curvature of a standard gamma. I'd like this to be true, as the solution is so easy.

However, even on the scopes I posted here, the V1 also went well over 100 in Vegas, yet they sky is consistently blown to white, including the parts behind tree twiggies. This looks much more natural.

On top of that, please take another look at the picture Christopher linked to earlier in this thread:

http://www.siliconcine.net/temp/Blew_Skies.jpg

Judging from the overall look, I'd say a cine gamma curve was used - and yet do you see what's happening to the trees at the right upper horizon part?

So, I will be completely reassured only after considerable time of trouble-free shooting with the above recipe in mind, i.e. when I'm 100% sure that with cine gammas (or standard curves with modified knee), nothing like this can happen to my picture.

Christopher Barry · February 7th, 2008, 04:19 AM

Piotr, factory default settings used on the EX1/HVX comparision, however, I can not be certain.
http://www.siliconcine.net/temp/Blew_Skies.jpg

Looking at the standard curve for XDCAM cam, IMO, why would you bother?
http://www.siliconcine.net/temp/XDCAM_Gamna_Curves.jpg

I understand your quest to solve this. Myself, I do not foreshadow dialing out of the four cine gamma options.

Piotr Wozniacki · February 7th, 2008, 04:37 AM

Quote:

Originally Posted by Christopher Barry

Piotr, factory default settings used on the EX1/HVX comparision, however, I can not be certain.
http://www.siliconcine.net/temp/Blew_Skies.jpg

Looking at the standard curve for XDCAM cam, IMO, why would you bother?
http://www.siliconcine.net/temp/XDCAM_Gamna_Curves.jpg

I understand your quest to solve this. Myself, I do not foreshadow dialing out of the four cine gamma options.

Christopher, if this was indeed shot with the same, standard curve, it'd mean it's not just my camera - and the solution being cine gammas, or standard ones but with modified knee. Thanks.

You're right in seeing this thread as a quest to solve/explain/understand an issue, and not another EX1 bashing. The fact is I only use standard curves for indoor, lowlight shooting when I usually crush blacks with their noise and try to catch as much mids and highs as possible. For outdoor shooting, the cine curves are so much better...

Dennis Schmitz · February 7th, 2008, 04:59 AM

After filming a few shots I think Cine2 looks best.
I tried to get the same dynamic range with the STD3 Gamma, but I used too much knee as you can see.

The STD3-Picture looks much softer und muddy (maybe because I used too much Knee? 70)

http://img163.imageshack.us/img163/3...99cine2kf6.jpg
http://img352.imageshack.us/img352/4...3knee70ex9.jpg

regards Dennis

Alexander Ibrahim · February 7th, 2008, 05:00 AM

The trees are acting as nets or scrims in the "problem" images. They are dropping the exposure of the sky... and hence your results.

As you zoom in, or even better walk closer, to the branches this should "magically" go away. This is because eventually the branches are too far apart to operate effectively as a scrim/net.

You will observe the same effect on diffuse lights, and especially lights with nets or scrims on them.

You will observe this with any camera. The exact circumstances will vary a bit based mostly on the sensor resolution, lens resolution and to some degree on the exposure. The main point is that regardless of what camera you are using you will find some combination of fine elements that will trigger this perception- that includes your eyes.

That is just the way light works.

All the talk of gamma levels and matrices is off the mark. No really - all of it. Go shoot that same scene with an HVX200, RED, a Viper or 35mm and you'll get similar results. (Again accounting for the sensor and lens resolution.) That isn't to say anyone's been wrong- just wrong that it has anything to do with the "problem."

The one note is that the issue is made more prominent by the XDCAM codec. Capture that as ProRes and while you'll still see the problem, but it will be less objectionable.

The case presented here can be "solved" in one of two ways:

1) drop exposure so that the entire sky has a valid exposure level. The differences in exposure in the sky will thus be less objectionable.

2) increase exposure levels so the sky well and truly blows out, then do a sky replacement in post.

3) use a filter or grip kit to bring levels down in the sky. I'd try a graduated ND filter first.

A circular polarizer might help a great deal. It would bring exposures down a bit and may have some effect on the fringing.

If it was a film set with a still camera I'd drop a net very close to the lens (and out of focus) to bring the level down in the sky. The net will also give that portion of the image a little softening. The problem with nets is that the edges are sometimes too thick.

For a lot of ENG/EFP video work I'd just live with it during acquisition and maybe throw a vignette and a soft blur on the image to drop the levels and sharpness there a bit. That would soften the effect a bit. There is no post solution that will really fix that sort of thing though once you shoot it.

Piotr Wozniacki · February 7th, 2008, 06:32 AM

OK guys - you got me convinced that this is not actually a flaw, but a feature. I did some backlit shooting today with all the factory setting, but the Knee point of the STD3 curve dialed down from 90 to 75.

The result is that I can no expose the darker foreground much better before they sky even starts to overblow (which was expected). However, once it does, the trees act exactly the same i.e. before the exposure is too high (and whole sky is simply white), it goes though the same stage where the sky areas right behind the trees become blue-gray (ugly and unnatural, as the rest of the sky is already white).

Now, my question is this: apart from changing the knee point, should I also change the slope and/or sauration in order to minimize this effect? Please give me some theory background to this; let me stress that I'm no longer looking for "flaws", but investigating "features" :)

Bob Grant · February 7th, 2008, 08:00 AM

The steeper the top slope of the curve the less risk of banding in gradient highlights e.g. sky. If you look at the curve you'll see that a quite large change in light (the X axis) produces a very small change in the output (the Y axis).Unfortunately the sky can as you've seen fall right along that slope. Part of it ends up clipped to white, the next darkest region has some color in it but two channels are clipped still so the color shifts.
Drop the exposure and the sky slides down the shallow slope and out of clipping. Increase the exposure and it all clips to white. Except you are maybe overloading the sensor. Look at the curve and see just how much light you're pushing into the camera to hit clipping. With that amount of light the sensor itself starts to do and see wierd things, like lens artifacts, light bouncing around between lens elements and inside the prism.
My suggestion would be not to use that curve in that shooting scenario. If you want to shoot that scene with natural looking blue sky and still preserve detail in the shadows you either need to add light to the shadows or reduce the brightness of the sky. I think someone previously had mentioned how to knock the sky down with a net, this is standard fare, either that or else adding fill lighting to bring the dark parts of the scene up.
Or even simpler, shoot with the sun in a different position or at a different time of day.

One thing that doesn't help is Sony haven't published any real gamma curves. The ones they have have no scales, they give a hint as to what's happening but clearly for example the std curve cannot go to 200% or if it does then the cine curves only go to 50%. Maybe what they mean is the top of the std curve is 100% for that curve and the top end of the cine curves ins 100% for that curve. Putting them all on the one chart like that might be graphically appealing but very confusing.

Bill Ravens · February 7th, 2008, 08:25 AM

If you look at the black wedge WFM pictures I posted, you'll see that the top end of the gamma curves in CINE mode are a fraction of the top end point for the STD curves, reinforcing the XDCAM HD curves you showed, Christopher.
This surprised me when I first saw them, which is why I, initially, normalized them to 100 IRE.

Christopher Barry · February 7th, 2008, 09:02 AM

Bob and Bill, the F900 Standard REC709 setting is up at around 150 IRE, likely the same in the XDCAM graph.
http://www.siliconcine.net/temp/XDCAM_Gamna_Curves.jpg

Alfonso Parra, DoP, has performed F900 tests on the Digital Praxis custom curves. If you look at this image/graph, at 400%, the F900 Standard curve runs up to 150 IRE. The other curves are the Digital Praxis customs, somewhat similar to the EX1's cine gamma, except closer to Cineon log and some ideal for intense post grading and film out. This provides some perspective of what the STD curve values may be, and IMO, can be applied to the EX1 STD curve, as approximations of the X and Y values. The EX1 is after all part of the CineAlta family.
http://www2.alfonsoparra.com/php/ima...nt/167/049.jpg

Bill Ravens · February 7th, 2008, 09:10 AM

This might be a stupid question, but, isn't the implication that the cam is more light sensitive in STD mode?
Furthermore, highlights will appear to clip in any NLE that can't handle IRE values over 100-110%?
Or conversely, crush the blacks if the highlights are exposed properly?

Christopher Barry · February 7th, 2008, 09:27 AM

Bill, I looked at your WFM wedges, thanks. When looking at Alfonso's wedges, they appear exposed 18% grey wedge to 50 IRE on the WFM, and then observe how the different curves responded to shadow and highlight detail.
http://www2.alfonsoparra.com/php/ver...&id=134&ln=eng

The image recorded is not always appealing, however, in post there should be the most amount of detail preserved to extract, however, an F900 and the EX1 have different codecs and data rates, and one will be limited by how robust the EX1, in HQ mode, holds up, and how far your can exploit it in post, if one makes extreme presets, similar to Cineon log curve.

My EX1 arrives next week, so lots of testing to follow, in due course.

Bill Ravens · February 7th, 2008, 09:32 AM

Exactly!! ;o)

Piotr Wozniacki · February 7th, 2008, 09:38 AM

This is becoming funny... I know you guys don't regard me high as a camea operator, but I have just finished my next couple of tests, and have a couple more surprizes for you:

1. Even with cine curves, at the moment EX1 starts clipping (and you cannot totally avoid it), the resulting colour patches are awful to watch (see the left grab: as if the blue colour resosution was insufficient). In the right grab, I was those tiny fractions of a stop on the safe side, and the sky exposed better.

2. I'm not sure, but I guess I have found the vignetting problem source; do you recognize the upper right corner thingy in the right grab? - well, it is the ND filter not quite snapped in...What a silly construction, and why no knob to rotate it quick and precisely?

Anyway, even though I really appreciate your experience and knowledge, I can now state it with 100% certainty: the highs compression algorhitm on the EX1 is faulty. You're simply wrong saying that all cameras do that; my V1E doesn't do that - period.

February 7th, 2008, 05:00 AM	#97
Alexander Ibrahim Major Player Join Date: Oct 2001 Location: Washington D.C. Metro Area Posts: 384	You have all over thought this one The trees are acting as nets or scrims in the "problem" images. They are dropping the exposure of the sky... and hence your results. As you zoom in, or even better walk closer, to the branches this should "magically" go away. This is because eventually the branches are too far apart to operate effectively as a scrim/net. You will observe the same effect on diffuse lights, and especially lights with nets or scrims on them. You will observe this with any camera. The exact circumstances will vary a bit based mostly on the sensor resolution, lens resolution and to some degree on the exposure. The main point is that regardless of what camera you are using you will find some combination of fine elements that will trigger this perception- that includes your eyes. That is just the way light works. All the talk of gamma levels and matrices is off the mark. No really - all of it. Go shoot that same scene with an HVX200, RED, a Viper or 35mm and you'll get similar results. (Again accounting for the sensor and lens resolution.) That isn't to say anyone's been wrong- just wrong that it has anything to do with the "problem." The one note is that the issue is made more prominent by the XDCAM codec. Capture that as ProRes and while you'll still see the problem, but it will be less objectionable. The case presented here can be "solved" in one of two ways: 1) drop exposure so that the entire sky has a valid exposure level. The differences in exposure in the sky will thus be less objectionable. 2) increase exposure levels so the sky well and truly blows out, then do a sky replacement in post. 3) use a filter or grip kit to bring levels down in the sky. I'd try a graduated ND filter first. A circular polarizer might help a great deal. It would bring exposures down a bit and may have some effect on the fringing. If it was a film set with a still camera I'd drop a net very close to the lens (and out of focus) to bring the level down in the sky. The net will also give that portion of the image a little softening. The problem with nets is that the edges are sometimes too thick. For a lot of ENG/EFP video work I'd just live with it during acquisition and maybe throw a vignette and a soft blur on the image to drop the levels and sharpness there a bit. That would soften the effect a bit. There is no post solution that will really fix that sort of thing though once you shoot it. __________________ Alexander Ibrahim http://www.alexanderibrahim.net

February 7th, 2008, 06:32 AM	#98
Piotr Wozniacki Inner Circle Join Date: Aug 2006 Location: Poland Posts: 4,086	OK guys - you got me convinced that this is not actually a flaw, but a feature. I did some backlit shooting today with all the factory setting, but the Knee point of the STD3 curve dialed down from 90 to 75. The result is that I can no expose the darker foreground much better before they sky even starts to overblow (which was expected). However, once it does, the trees act exactly the same i.e. before the exposure is too high (and whole sky is simply white), it goes though the same stage where the sky areas right behind the trees become blue-gray (ugly and unnatural, as the rest of the sky is already white). Now, my question is this: apart from changing the knee point, should I also change the slope and/or sauration in order to minimize this effect? Please give me some theory background to this; let me stress that I'm no longer looking for "flaws", but investigating "features" :) __________________ Sony PXW-FS7 \| DaVinci Resolve Studio; Magix Vegas Pro; i7-5960X CPU; 64 GB RAM; 2x GTX 1080 8GB GPU; Decklink 4K Extreme 12G; 4x 3TB WD Black in RAID 0; 1TB M.2 NVMe cache drive

February 7th, 2008, 09:38 AM	#105
Piotr Wozniacki Inner Circle Join Date: Aug 2006 Location: Poland Posts: 4,086	This is becoming funny... I know you guys don't regard me high as a camea operator, but I have just finished my next couple of tests, and have a couple more surprizes for you: 1. Even with cine curves, at the moment EX1 starts clipping (and you cannot totally avoid it), the resulting colour patches are awful to watch (see the left grab: as if the blue colour resosution was insufficient). In the right grab, I was those tiny fractions of a stop on the safe side, and the sky exposed better. 2. I'm not sure, but I guess I have found the vignetting problem source; do you recognize the upper right corner thingy in the right grab? - well, it is the ND filter not quite snapped in...What a silly construction, and why no knob to rotate it quick and precisely? Anyway, even though I really appreciate your experience and knowledge, I can now state it with 100% certainty: the highs compression algorhitm on the EX1 is faulty. You're simply wrong saying that all cameras do that; my V1E doesn't do that - period. Attached Thumbnails __________________ Sony PXW-FS7 \| DaVinci Resolve Studio; Magix Vegas Pro; i7-5960X CPU; 64 GB RAM; 2x GTX 1080 8GB GPU; Decklink 4K Extreme 12G; 4x 3TB WD Black in RAID 0; 1TB M.2 NVMe cache drive

February 6th, 2008, 09:55 PM	#91
Bob Grant Trustee Join Date: Nov 2005 Location: Sydney Australia Posts: 1,570	I just tried a simple test with the EX1, hasn't been out of it's box today :) Lamp with a diffused shade again a dark wall. Camera in manual, lamp filling around half the frame. EX1 histogram shows a nice lump in the middle, nothing anywhere near clipping. Zoom in so the light is now full frame, the lump moves to the right of the histogram, looks like I'm getting pretty close to clipping. Zoom all the way into the slightly hotter middle of the lamp and on comes the Too Bright warning. Note I have not changed exposure while zooming in. This is very different to how the scopes in Vegas work. The waveform display is the sum of all lines. If I'd recorded what I shot above and checked it with the Vegas scopes I'd see a lump in the middle of the waveform that was clipped. As I zoomed in the lump would get broader. I just simulated the same kind of shot using Vegas generated media. Using Vegas's Histogram metering I get much the same thing. It's almost impossible to detect clipping with a histogram. In fact from what I'm seeing with histograms the more clipping there is in the shot the less noticeable it is in the histogram. With a whole frame clipped you just see a tiny line on the RH side of the display. This article here: http://www.luminous-landscape.com/tu...stograms.shtml gives a detailed explaination of what a histogram shows. About half way down the page there's an example of a scene very similar to the one causing all this grief.

February 7th, 2008, 01:00 AM	#92
Michael H. Stevens Major Player Join Date: Sep 2007 Location: Palm Desert, California Posts: 311	I started a thread on this in the Vegas forum. An EX1 histogram that looks like it goes from 10 to 90 put into Vegas clips like it wants to go to 120. By this I mean extrapolate the shape of the histogram into the clip area that Vegas scope cut of. AND the Vegas scopes clip at the BOTTOM END TO! Like the blacks want to go to -20! Also Piort I DID give my opinion as to what is causing this much earlier in this thread and no one commented on it so maybe everyone thinks I am wrong, but here is my take again. The EX1 gammas are not straight line or even a partial exponential of a log of a straight line, I think it is programmed to have the Gamma changes with luminosity. The area behind the trees has a lower luminosity (because the trees block the light) and the gamma drops in that area which brings the blown white back into the blue. I think this how the camera gets its enormous exposure latitude and why the Vegas scopes get stretched. JMO as they say.

February 7th, 2008, 04:19 AM	#94
Christopher Barry Regular Crew Join Date: Jan 2007 Location: Australia Posts: 107	Piotr, factory default settings used on the EX1/HVX comparision, however, I can not be certain. http://www.siliconcine.net/temp/Blew_Skies.jpg Looking at the standard curve for XDCAM cam, IMO, why would you bother? http://www.siliconcine.net/temp/XDCAM_Gamna_Curves.jpg I understand your quest to solve this. Myself, I do not foreshadow dialing out of the four cine gamma options.

February 7th, 2008, 04:59 AM	#96
Dennis Schmitz Regular Crew Join Date: Dec 2007 Location: Viersen, Germany Posts: 120	After filming a few shots I think Cine2 looks best. I tried to get the same dynamic range with the STD3 Gamma, but I used too much knee as you can see. The STD3-Picture looks much softer und muddy (maybe because I used too much Knee? 70) http://img163.imageshack.us/img163/3...99cine2kf6.jpg http://img352.imageshack.us/img352/4...3knee70ex9.jpg regards Dennis

February 7th, 2008, 08:00 AM	#99
Bob Grant Trustee Join Date: Nov 2005 Location: Sydney Australia Posts: 1,570	The steeper the top slope of the curve the less risk of banding in gradient highlights e.g. sky. If you look at the curve you'll see that a quite large change in light (the X axis) produces a very small change in the output (the Y axis).Unfortunately the sky can as you've seen fall right along that slope. Part of it ends up clipped to white, the next darkest region has some color in it but two channels are clipped still so the color shifts. Drop the exposure and the sky slides down the shallow slope and out of clipping. Increase the exposure and it all clips to white. Except you are maybe overloading the sensor. Look at the curve and see just how much light you're pushing into the camera to hit clipping. With that amount of light the sensor itself starts to do and see wierd things, like lens artifacts, light bouncing around between lens elements and inside the prism. My suggestion would be not to use that curve in that shooting scenario. If you want to shoot that scene with natural looking blue sky and still preserve detail in the shadows you either need to add light to the shadows or reduce the brightness of the sky. I think someone previously had mentioned how to knock the sky down with a net, this is standard fare, either that or else adding fill lighting to bring the dark parts of the scene up. Or even simpler, shoot with the sun in a different position or at a different time of day. One thing that doesn't help is Sony haven't published any real gamma curves. The ones they have have no scales, they give a hint as to what's happening but clearly for example the std curve cannot go to 200% or if it does then the cine curves only go to 50%. Maybe what they mean is the top of the std curve is 100% for that curve and the top end of the cine curves ins 100% for that curve. Putting them all on the one chart like that might be graphically appealing but very confusing.

February 7th, 2008, 08:25 AM	#100
Bill Ravens Join Date: Nov 2001 Location: santa fe, nm Posts: 3,264 Images: 10	If you look at the black wedge WFM pictures I posted, you'll see that the top end of the gamma curves in CINE mode are a fraction of the top end point for the STD curves, reinforcing the XDCAM HD curves you showed, Christopher. This surprised me when I first saw them, which is why I, initially, normalized them to 100 IRE.

February 7th, 2008, 09:02 AM	#101
Christopher Barry Regular Crew Join Date: Jan 2007 Location: Australia Posts: 107	Bob and Bill, the F900 Standard REC709 setting is up at around 150 IRE, likely the same in the XDCAM graph. http://www.siliconcine.net/temp/XDCAM_Gamna_Curves.jpg Alfonso Parra, DoP, has performed F900 tests on the Digital Praxis custom curves. If you look at this image/graph, at 400%, the F900 Standard curve runs up to 150 IRE. The other curves are the Digital Praxis customs, somewhat similar to the EX1's cine gamma, except closer to Cineon log and some ideal for intense post grading and film out. This provides some perspective of what the STD curve values may be, and IMO, can be applied to the EX1 STD curve, as approximations of the X and Y values. The EX1 is after all part of the CineAlta family. http://www2.alfonsoparra.com/php/ima...nt/167/049.jpg

February 7th, 2008, 09:10 AM	#102
Bill Ravens Join Date: Nov 2001 Location: santa fe, nm Posts: 3,264 Images: 10	This might be a stupid question, but, isn't the implication that the cam is more light sensitive in STD mode? Furthermore, highlights will appear to clip in any NLE that can't handle IRE values over 100-110%? Or conversely, crush the blacks if the highlights are exposed properly?

February 7th, 2008, 09:27 AM	#103
Christopher Barry Regular Crew Join Date: Jan 2007 Location: Australia Posts: 107	Bill, I looked at your WFM wedges, thanks. When looking at Alfonso's wedges, they appear exposed 18% grey wedge to 50 IRE on the WFM, and then observe how the different curves responded to shadow and highlight detail. http://www2.alfonsoparra.com/php/ver...&id=134&ln=eng The image recorded is not always appealing, however, in post there should be the most amount of detail preserved to extract, however, an F900 and the EX1 have different codecs and data rates, and one will be limited by how robust the EX1, in HQ mode, holds up, and how far your can exploit it in post, if one makes extreme presets, similar to Cineon log curve. My EX1 arrives next week, so lots of testing to follow, in due course.

February 7th, 2008, 09:32 AM	#104
Bill Ravens Join Date: Nov 2001 Location: santa fe, nm Posts: 3,264 Images: 10	Exactly!! ;o)