Quality expected from the EX3 and Flash XDR

[Steve Phillipps]

Mike,
Not sure I agree that it's difficutl to compare different cameras. Put a top-notch lens on a Varicam, the standard (I assume pretty decent) lens on the EX3, with Flash XDR and shoot a couple of different, challenging scenes. Put 'em on a decent sized monitor and decide if they look pretty close or poles apart.
What I would like is some fairly clear evidence that the combination is up there with Varicam/Sony 750 and would cut with them well. These 2 cams are pretty much the standard and the benchmark for HD TV in the UK, and most producers would flinch a little at using a 1/2" "semi-pro" camera (I know that definition is getting a bit blurred these days) unless one could show them some evidence that it'll do what they need it to.
Steve

[David Heath]

Originally Posted by Dan Keaton

Our goal is to have the best possible comparisons.

When we shoot the motion video footage, we will be posting the footage on our website so that others can perform their own tests, in addition to the tests that we will be performing. Of course, the real problem is posting uncompressed footage due to the size of the files.

It does occur to me that one of the biggest tests of interframe codecs (at least on the evidence of bitrate limited broadcast TV!) is quick mixes or fades.

Since you are interested in codec comparisons (as opposed to actual cameras) then bearing in mind what you say above, is there any merit in not using actual video for the test, but rather mixing between high quality 1920x1080 still images? Using that as a source of the uncompressed reference? As well as precision and repeatability, that may have an advantage in that whilst the scene is changing rapidly frame to frame, (and is hence a good test of long-GOP compression), nothing is actually moving in the frame, so it may be easier to draw conclusions via the Photoshop difference method and filmstrips.

It would also make it very easy to compare the differences between (say) 8, 16, and 30 frame mixes.

Posting the original images, for people to download and derive their own reference (as per your instructions), would also vastly reduce download file sizes compared to motion video.

[Dan Keaton]

Dear David,

Your suggestions are very interesting.

Yes, it would make the task easier and the downloads quicker since we could control the images better. Of course,we would have to obtain or create the still images.

But, I wonder if everyone would feel that this is a valid test? Would they think it is "kind of artifical" and then they would still want to see actual full motion video as created by a known camera.

[Mike Schell]

Originally Posted by Steve Phillipps

Mike,
Not sure I agree that it's difficutl to compare different cameras. Put a top-notch lens on a Varicam, the standard (I assume pretty decent) lens on the EX3, with Flash XDR and shoot a couple of different, challenging scenes. Put 'em on a decent sized monitor and decide if they look pretty close or poles apart.
What I would like is some fairly clear evidence that the combination is up there with Varicam/Sony 750 and would cut with them well. These 2 cams are pretty much the standard and the benchmark for HD TV in the UK, and most producers would flinch a little at using a 1/2" "semi-pro" camera (I know that definition is getting a bit blurred these days) unless one could show them some evidence that it'll do what they need it to.
Steve

Hi Steve-
I only meant that it's difficult to compare various cameras from the point of view of determining whether the CODEC or the sensor controls the ultimate quality. Since the HD-SDI output bypasses the built-in recorder, we can easily compare the front-end of all these cameras.

I have seen resolution comparison charts of the EX1 vs cameras in the $25K price range. The EX1 clearly outperforms cameras with 2/3inch sensors on this resolution chart. I think the use of full-raster 1920x1080 sensor vs 1440x1080 (or 1280x1080) makes all the difference.

Since the MPEG2 CODEC on Flash XDR is also full-raster, we can keep all this quality through the compression process. DVCProHD CODEC uses a 1280x1080 raster, which results in a considerable loss in resolution.

I hope to post comparision video in the next week or so.

[David Heath]

Originally Posted by Dan Keaton

But, I wonder if everyone would feel that this is a valid test? Would they think it is "kind of artifical" and then they would still want to see actual full motion video as created by a known camera.

I certainly wouldn't propose it as the only test, and yes, I agree most people would want to be aware of real moving video performance. Normally, it's cameras that are being compared and hence front ends etc as well as codecs, so such an approach isn't possible.

But too often, I find it difficult to draw accurate conclusions from some tests along these lines - the motion can all too often cause such blurring etc that it's difficult to tell what degradation is due to any codec failure, and what is due to characteristics of the image itself.

Mixing between two still images shouldn't have the same problem, especially if the difference approach is used to compare the compressed and uncompressed versions - whilst still being able to tax inter-frame performance. "Kind of artificial"? Well, maybe, but sometimes "artificial" means are necessary to quantify video performance in a meaningful way. Hence the variety of test signals from analogue days - pulse and bar, colour bars etc.

But as you imply, all this is academic if the majority opinion is that the test is meaningless. Anyone care to give their views?

[Dan Keaton]

Dear David,

For discussion purposes:

(Please forgive me if all of this is completely obvious.)

Let's say we have 15 distinct images, each completely different from the 14 others. Then if we use a Long GOP Codec to encode them, then this would be an extreme test of the Long GOP Codec. If we use an I-Frame Only codec, it would make no difference that all were different.

In typical motion video, there are some similarities from one frame to another. Of course, when the scene changes, there can be a drastic change in which there are no similarities. But, typically these drastic changes usually do not occur more than once or twice in a group of 15 images.

So, for typical motion video, a Long GOP codec takes advantage of these similarities. The areas that do not have to be encoded, since they are similar to the previous frame, allow more bits to be used to encode the image, thus providing a high quality encoding.

In your proposed test, will each image be completely different?

[Mike Schell]

Hi Dan, David-
Just wanted to mention that we are working on a program to stripe 160Mbps I-Frame only video across two of the low-cost ($150) 32GB Transcend cards and then re-assemble in post. Assuming this can be accomplished, I think we'll have an very cost effective solution for either 100 Mbps Long-GOP or 160 Mbps I-Frame only. Four of the 32GB cards would support 105 minutes of 160 Mbps record time w/o any card swaps.

So depending on the situation, you can either go with Long-GOP or I-Frame mode. Even if our tests do show good motion performance with Long-GOP (all indications are positive), you will still need to make a judgment call for any given shoot. We would simply like to provide you with the capability and let you make the choice.

[David Heath]

Originally Posted by Dan Keaton

In your proposed test, will each image be completely different?

You raise some good points!

Yes, I agree that a "test" where 15 totally different images are presented is hardly a fair test of a long GOP codec - though it may be interesting to see the results!

As far as my suggestion of using a mix, then I'd say that whilst each frame will be different from the ones before and after, it would be wrong to say it would be completely different. Imagine a 8x8 section of the image, starting as (say) left half white, right half black and mixing over 15 frames to a uniform red. One frame into the mix, the left side will still be quite high luminance, the right quite low luminance, and the whole still quite low chrominance - in other words, still with quite a lot of similarities to the previous frame.

Obviously, the slower the mix, the more similar adjacent frames will be, hence my suggestion to do the test at several different speeds. Effectively, varying the speed of the mix simulates varying the amount of motion in an action sequence, but does it in a controllable and repeatable fashion.

Whilst a camera can't mix from one image to another within itself, I understand the XDR is marketed as a recorder for applications other than purely single camera recording, so encoding mixes is far from academic. Even in single camera use, there's the situation where still camera flashes may be going off and causing rapid frame by frame changes, even more so than a uniform mix.

[John Mitchell]

While others argue about the technical merits of this camera versus that camera and this codec versus that codec, I think Mike has hit the nail on the head - it is much more interesting to see some typical footage. People moving about, colourful images of a beach scene perhaps, some test chroma key footage etc. Photoshop difference maps are just so one company can claim their codec is better than anothers and I've seen plenty of instances where they mean nothing with real world images. Believe your eyes my friends - they are the ultimate QC...

From all accounts the EX3 (like the EX1) will be stunning HD full raster quality - if you want to whack a 1/2" broadcast HD lens on the front it's likely to get even better. Combine that with the FlashXDR and you have a technically sound recording device with cheaper alternative media.

The other stuff is just a distraction (an interesting distraction) - come on Mike - give us some real world footage to look at in HD. I'd prefer to see it off the EX1, but even the XL-H1 will do.

[Dan Keaton]

Dear David,

I agree completely.

Multiple Flashes going off in multiple frames are a torture test for a codec.

Mike's news is very nice to hear. What has been obvious in our discussion is that one codec has an advantage over another under specific conditions. So, the choice of which codec to use is provided by the Flash XDR.

I would like to perform your recommended tests as soon as we can fit it into our schedule.

Dear John,

Tests with a Sony EX1 are in our plans. I do not know the exact date, but it should not be a long wait. As Mike says: "Stay Tuned".

Dear Mike,

Supporting 160 Mb I-Frame only, in the very cost effective Transcend 32 GB cards is a very nice feature!

[Nikol Manning]

Comparing cameras and codecs is it really hard to believe that a EX3 and Flash XDR combo could out perform a varicam recording to tape? The EX3 combo should be noticeable better. The varicam is a 720p camera recording to a 40Mbps dvcprohd codec. Are you serious! The varicam is a great camera but it is old technology. The EX3 and XDR is future technology. It is the nature of the business. New stuff comes out old stuff gets replaced with cheaper options. Now a varicam recording to XDR is a different story. :)

[David Heath]

Originally Posted by Dan Keaton

I agree completely.

Multiple Flashes going off in multiple frames are a torture test for a codec.

Which brings a thought to mind. I've read quite a few posts where samples of EX footage have been examined and commented on with regard to rolling shutter effects and camera flashes. Yet in none of these posts has any comment been made about codec breakup with this footage (AFAIK). Which leads me to think that even the basic 35Mbs codec is far more robust than some may think. So..... going up to 50 or 100 Mbs should indeed be very promising from a quality point of view?

Originally Posted by John Mitchell

Photoshop difference maps are just so one company can claim their codec is better than anothers and I've seen plenty of instances where they mean nothing with real world images. Believe your eyes my friends - they are the ultimate QC....

Sorry John, I disagree. Except at the low end (which is not what we're talking about here) I'd argue that all the decent codecs look pretty good with real world images for straightforward record/playback. Full raster codecs are obviously going to look sharper than subsampled codecs - if the source material is up to it - but any differences the eyes will see at this level are largely due to camera front ends.

Where codec differences start to become more important is when decoding/recoding happens, and that can happen several times cascaded in a broadcast chain. Some codecs can also stand up far better during the post production process, during FX, grading etc work. And difference maps not just show up, but actually help quantify differences between images, differences that are just not discernable in straight forward viewing. But differences which do become important cascading down the production/broadcast chain.

http://www.photoshopessentials.com/e...g-compression/ explains the basic principles of difference mapping for still images (with examples) if anyone is interested. The process can be taken further by separating the images into Y,U,V and comparing these individually.

A company doesn't need difference maps to CLAIM codec A is superior to codec B, but do need them (or something equivalent) to PROVE the claim if challenged. And if another company has a vested interest in codec B, and does seriously dispute the claim, that proof may be very valuable - far from "just a distraction ".

[Steve Phillipps]

Nikol, not saying I disagree with you. I just to be able to convince producers that it's the case.
Steve

[John Mitchell]

Originally Posted by David Heath

Sorry John, I disagree. Except at the low end (which is not what we're talking about here) I'd argue that all the decent codecs look pretty good with real world images for straightforward record/playback. Full raster codecs are obviously going to look sharper than subsampled codecs - if the source material is up to it - but any differences the eyes will see at this level are largely due to camera front ends.

David - I'm sorry if my previous post seemed a little blase. It wasn't my intention. I just see no point in comparing MPEG2 @ x data rate with DVCProHD @ x data rate, because the Flash XDR doesn't record to DVCProHD. It will however blow HDV and and 35Mb/s MPEG2 away on cheaper recording media, and because of it's full raster nature will be preferred over DVCProHD. I don't need tests to confirm that. But I would like to see some vision first :)

Originally Posted by David Heath

http://www.photoshopessentials.com/e...g-compression/ explains the basic principles of difference mapping for still images (with examples) if anyone is interested. The process can be taken further by separating the images into Y,U,V and comparing these individually.

A company doesn't need difference maps to CLAIM codec A is superior to codec B, but do need them (or something equivalent) to PROVE the claim if challenged. And if another company has a vested interest in codec B, and does seriously dispute the claim, that proof may be very valuable - far from "just a distraction ".

I understand the way difference maps work, but applying a technique best used on still images is less than ideal for any intraframe codec. That's why I don't think it's relevant, because an interframe codec like DnxHD may "appear" to outperform an intraframe codec like MPEG2 for any particular frame (except for an iframe) because the MPEG2 codec has to recreate the image from earlier and later frames, in reality when you watch the motion footage those differences are irrelevant to the human perception system. That's why I say trust your eyes... not Photoshop.

And some things like repeated camera flashes will "break" an MPEG2 or AVCHD codec because that is not what the codec was designed to do - it's designed to compress motion as well as data by using predictive algorithms, and by it's very nature you cannot predict a camera flash. Of course the quality of images produced under a camera flash may depend more on the front end of the camera than the recording method.

Originally Posted by David Heath

Where codec differences start to become more important is when decoding/recoding happens, and that can happen several times cascaded in a broadcast chain. Some codecs can also stand up far better during the post production process, during FX, grading etc work. And difference maps not just show up, but actually help quantify differences between images, differences that are just not discernable in straight forward viewing. But differences which do become important cascading down the production/broadcast chain.

No argument here. Multi generation work is where Photoshop difference mattes are valuable. But I don't think you'd argue that MPEG2 or AVCHD were "good" for multi-generation work. Ideally if you anticipate doing a lot of post manipulation you will not use Mpeg 2 to work in - it is an acquisition and delivery format (and originally it was designed purely as a delivery format, it's just that manufacturers saw the value of compressing the data more at the acquisition end to improve quality at any given bit rate). You would use ProRes, DnxHD, Cineform or a totally uncompressed system like tiff sequences etc.

Much more important (to me anyway) is the real world stuff - can I pull a beautiful chroma key off vision recorded on the FlashXDR? How far can I colour correct and push things like exposure before it starts to break up at any given data rate? Can it capture fast moving action effectively without macroblocking? These are things I can only evaluate with real world footage from a decent front end.

[David Heath]

Originally Posted by John Mitchell

I just see no point in comparing MPEG2 @ x data rate with DVCProHD @ x data rate, because the Flash XDR doesn't record to DVCProHD. It will however blow HDV and and 35Mb/s MPEG2 away on cheaper recording media, and because of it's full raster nature will be preferred over DVCProHD. I don't need tests to confirm that.

I suspect the reason that Convergent Design included DVCProHD in their initial tests was to act as a form of "control" sample. It's an established professional codec, and makes a useful benchmark against which to compare MPEG2 at different bitrates - it would be interesting to see how HDCAM also compared.

Of the codecs you mention, I would also bet on 100Mbs MPEG2 to outperform the others, no argument, but feel that PROVING it would be valuable for the reasons mentioned before. To that end, difference maps are the best way I can think of of making valid measurements. I would strongly suspect that DVCProHD would show substantially more differences in still images (partly due to subsampling), but the level to remain constant during any mix. I would also expect the difference levels for the MPEG2 samples to increase during the mix relative to the still images, but if the levels were still lower than for DVCProHD at the equivalent point, this would prove fairly unequivocally that it was a superior codec for quality, even in motion scenes.

Originally Posted by John Mitchell

I understand the way difference maps work, but applying a technique best used on still images is less than ideal for any intraframe codec. That's why I don't think it's relevant, .........., in reality when you watch the motion footage those differences are irrelevant to the human perception system. That's why I say trust your eyes... not Photoshop.

I have much agreement there. I've seen certain images put forward from very rapid motion to show the effects of interframe compression, and yes, they show something, but totally lost admidst the blur of motion footage in normal viewing.

But in a way it's why I say don't just trust your eyes. Test signals for analogue video were designed to show up imperfections that just weren't discernable on normal pictures, but could become a problem in a cascade. Hence of use for lining up one link in a broadcast chain in isolation to the rest, and it's a similar principle here. Small imperfections don't neccessarily matter in real life, but they do allow quantification and comparison.

Originally Posted by John Mitchell

And some things like repeated camera flashes will "break" an MPEG2 or AVCHD codec because that is not what the codec was designed to do - ....... by it's very nature you cannot predict a camera flash.

Yes, but unfortunately camera flashes going off, strobes etc is a fact of real life and a codec does need to be able to cope, at least if it's claiming an upmarket status. But even 35Mbs seems to do quite a good job in practice, and presumably 50 and 100Mbs will be better still. Flashes and mixes in this case may well not break any codec - but they will stretch and test it.

Originally Posted by John Mitchell

Much more important (to me anyway) is the real world stuff - can I pull a beautiful chroma key off vision recorded on the FlashXDR? How far can I colour correct and push things like exposure before it starts to break up at any given data rate? Can it capture fast moving action effectively without macroblocking? These are things I can only evaluate with real world footage from a decent front end.

I'm sure the conclusion will be "it's very, very good". Trouble is it may be so good that no difference is seen between it and rival codecs, or even uncompressed. Difference maps are more likely to show up subtle differences which may prove more of an issue on other images, or when codecs are cascaded.

Exposure and colour correction are more likely to be influenced by bit depth than data rate, and if there's anything about this that leaves me thinking "if only" is that it's still an 8 bit codec.