Independent Codec Comparisons

[Dan Keaton]

Dear Friends,

One of our friends, Mr. Michael Condon of Clairmont Camera has performed some in-depth codec analysis.

He forwarded these to me with permission to publish his results.

Here is his "Read Me" for these files:

HD Codec Evaluation/Test

Submitted by:
Michael Condon SOC, VP Digital Division
Clairmont Camera
March 16, 2011

This test was done because I’m often asked “Which codec is better?” by many of our clients. Since I haven’t seen any results of tests like this, I decided to do it myself. Included are stills taken from the following codecs:

HDCAM-SR 444SQ (10-bit MPEG4 Studio Profile)
HDCAM-SR 422SQ (10-bit MPEG4 Studio Profile)
ProRes 422HQ (10-bit)
ProRes Proxy (10-bit)
XDCAM 280Mbs I-Frame only (8-bit 4:2:2)
XDCAM 50Mbs Long-GOP (8-bit 4:2:2)
XDCAM 35Mbs Long-GOP (8-bit 4:2:0)

The original material was recorded from Sony F35 cameras directly to HDCAM-SR 444SQ. This is a still frame from a short film written by Sam Nicholson ASC and produced by his company Stargate Studios entitled “Thoroughbred”. I did not have an uncompressed master to work with. All of the stills were evaluated or compared to 444SQ as the benchmark. The process of gathering these stills were as follows:

The SR 444SQ master was played out and recorded on the KiPro Mini utilizing it’s ProRes codec and to the nanoFlash utilizing it’s XDCAM codec in the codec bit-rate flavors mentioned above. All material was then played back out of each device and captured uncompressed to a CODEX Onboard Recorder. These files were then output as 10-bit dpx. They were then transcoded to uncompressed 16-bit tiff files to maintain the integrity of some of the higher 10-bit sampled codecs.

A difference matte was created with each still against the benchmark SR 444SQ still. These are included as well. After all was done, I created 8-bit jpg files for the ease of emailing size.

I suggest opening the SR 444SQ still and any others you want to compare with it in separate windows. Then you can enlarge them by a large amount to see where they break down against the SR still.

The difference matte stills are easy to see how the codecs slowly fall apart from the SR 444HQ still.

I’ll be interested to get responses from people reading this regarding the order in which they believe the codecs should be from highest to lowest.


The following are my comments.

These are difference masks.

The less detail that one can see in a difference mask, the closer the image is to the comparision file.

Thus, the more detail that one can see in these images, the less fidelity to the comparison image.

I will post the actual images (not difference masks) in my next post.

[Dan Keaton]

Dear Friends,

Here are the normal images, not difference masks.

[Billy Steinberg]

Very interesting and useful information, thanks for sharing it. I just wish Michael had tested the Nano 100Mb Long GOP codec along with the rest.

Billy

ps For those who are wondering which pictures are which, just mouse over one and read the filename in the browser, and it will indicate which codec corresponds to each picture. (Though you might want to evaluate what you see before you look, in case you don't trust yourself to be honest). :)

[Buck Forester]

This stuff is all over my head... not only did I not understand much of what you wrote, I can't see a spec of difference in any of the photos. Maybe I'm getting into the wrong business and should stick with still photography, ha!

[Dan Keaton]

Dear Buck,

Please look at the images in the first post.

A perfect match, between the source comparison file and the image under test, would be completely void of any image.

The more of an image you can see, then the more the image differs from the comparison image.

In these tests, the comparison image is a very high-quality 4:4:4 image. All of the images under test are 4:2:2 (or 4:2:0) so you will see a difference in all.

[Adam Reuter]

Big thanks to Michael Condon for running these tests. I am evaluating these external recorders at the moment and am still on the fence. My issues (which may not be issue, just a lay person thinking) are:

1. Why record to the CODEX? Why not just open the captured files up in a 10-bit editor and export TIFF files?

2. JPEG files :-( I know, I know...email convenience.

3. XDCAM 280Mbs I-Frame vs. ProRes 422 HQ is 220Mbs. Not an exact comparison from a datarate standpoint.


I never was a big fan of Apple's DV codec. Compared to the Canopus (now Grass Valley) DV Codec it always looked "soft" to me. And I'm wondering if the ProRes codecs still have that "stink" on them. This plus the inability to record over/undercranked footage to the Ki is holding me back.

[Cees van Kempen]

[QUOTE=Dan Keaton;1633422]
The less detail that one can see in a difference mask, the closer the image is to the comparision file.

Thus, the more detail that one can see in these images, the less fidelity to the comparison image.
QUOTE]

I like to get some more explanation on this. If I look at the difference mask of the proress 422hq I see a lot of 'noisy detail' which I do not see at the difference mask of the xdcam 280 I-Fo. However the 280 I-Fo clearly shows more 'clean' detail of the picture (here you can see that it is a hores race, which is hardly to discover from the prores 422hq picture). So both pictures show detail, but in a completely different way. What does this tell?

[Dan Keaton]

Dear Friends,

The first image, in the first post is a comparison of HDCam SR 4:4:4 versus HDCam SR 4:2:2.

This should be your reference image.

In other words, in my opinion, in evaluating each of the other "difference mask" images, the one that is visually closest to the HDCam SR 4:4:4 versus HDCam SR 4:2:2 would be best.

Note: All of the images under test are 4:2:2 (except for the 4:2:0 ones).

Thus, none of these will give you a perfect difference mask, since the master images is 4:4:4 (meaning more detail) and all of images being compared are either 4:2:2 or 4:2:0.

I welcome comments from others and maybe someone else can explain this better.

[Piotr Wozniacki]

Not to be a bore reminding "remember what I said?" but -

- omission of any bitrate higher than 50 Mbps in L-GoP confirms my findings.

It's either 50 in L-GoP, or 280 in I-Fo.

Piotr

[Dan Keaton]

Dear Piotr,

These were independent tests by Mr. Michael Condon.

We did not specify, in any way which flavors and bit-rates that were tested.

Nor did we omit any comparisons. I posted what I received.

For Michael's very high-end clients, their normal choice would be 280 Mbps, and we have publicly stated many times that this offers the best quality available out of the nanoFlash. Of course, we also feel that 220 Mbps is a close second. (But this was not tested).

In my opinion, no other conclusions should be drawn about other flavors and bit-rates that were not tested.

[Piotr Wozniacki]

Just for the records, Dan:

- I'm not a "very high-end client", and yet my normal choice is also 280 Mbps I-Fo. Other than the highest quality jobs, the EX's 35 Mbps still offers the best efficiency per overall cost involved (with the exception perhaps of material requiring acceptance by some HD broadcasters like BBC, when 50 Mbps 422 is the best choice).

Last but not least: as you're aware, the above are conclusions I came to after more than 2 years of experimenting and testing, and is not intended as any kind of bashing the nanoFlash. Nevertheless, I truly believe that - 3 years after its debut - it'd be a good time to think about all new approach towards this marvellous device firmware. I realize some fundamental re-code would imply paid options, but I am sure those AF100 or FS100 users would appreciate it and probably pay for it happily, rather than jump on the new and shiny 4:4:4 recorder which is just an overkill with this type of cameras (with the exception of the F3, of course).

Because - no matter if the FS100 really outputs 10bit (it's not been 100% confirmed yet) - the incoming large sensor cameras will have much less noise, and those extra 2 bit of color quantization will become very desirable.

Best,

Piotr

[Buck Forester]

"The less detail that one can see in a difference mask, the closer the image is to the comparision file."

"Thus, the more detail that one can see in these images, the less fidelity to the comparison image."

When I read those statements above, the grey image with the most 'detail' that I see, as in the horses and riders are the most clearly defined, is the last one, 280 mbps, which would seem would suggest it is the worst of them all. But 280 mbps is supposed to be the best. Clearly I'm not grasping this, but that's okay, it takes me a while, ha ha! Don't mind me. Carry on.

[Simon Wood]

Originally Posted by Buck Forester

"The less detail that one can see in a difference mask, the closer the image is to the comparision file."

"Thus, the more detail that one can see in these images, the less fidelity to the comparison image."

When I read those statements above, the grey image with the most 'detail' that I see, as in the horses and riders are the most clearly defined, is the last one, 280 mbps, which would seem would suggest it is the worst of them all. But 280 mbps is supposed to be the best. Clearly I'm not grasping this, but that's okay, it takes me a while, ha ha! Don't mind me. Carry on.

I thought it was the other way round?

The top left image is the reference picture (hdcam-sr-444-vs-hdcam-sr-422). So the closest image to that should be the best?

In which case the best would be the image in the bottom right (hdcam-sr-444-vs-xdcam-280mbs-i-frame-only).

Second would be the image at the top right (hdcam-sr-444-vs-prores-422hq).

Third would be bottom left (hdcam-sr-444-vs-xdcam-50mbs-long-gop).

Fourth middle left (hdcam-sr-444-vs-prores-proxy).

Last middle right (hdcam-sr-444-vs-xdcam-35mbs-long-gop)

???

[Dan Keaton]

Dear Simon,

I agree with you.

[Dan Keaton]

Dear Buck,

It is hard to put this concept into words.

Yes, one can see the outline of the horses and riders in the 280 Mbps image.

But on many others, there is more "noise" or "clutter" which one has to take into consideration.

This noise and clutter is represents the difference from the ideal or comparison image.

If the comparion image was 4:2:2 and not 4:4:4, then one might expect the results to be such that one might not be able to make out much actual detail at all.