I think it is long overdue to do a comparison of HD codecs, in the same vein as video camera shoot-outs have been done. I would propose taking a high-end camera in the $60,000 range and feeding its HD-SDI output to various free-standing recorders using different codecs, then playing the recordings back and measuring resolution, artifacts and double-blind viewer preference. Panasonic makes the HPG20 outboard AVC-intra recorder and the HMR10 24 Mbs AVCCAM recorder, Sony makes the PMW-EX30 XDCAM HD and HDCAM recorder, Blackmagic makes numerous ProRes422 Quicktime / AVID DNxHD MXF / recorders, and I'm sure that there are a number of HD MPEG-2 recorders out there as well. I believe that someone also makes a motion JPEG 2000 video recorder. B & H Photovideo would be a great candidate to coordinate such a test, as they probably have most if not all of this equipment.

It's a lot more complicated than it may at first seem. Firstly, all the best codecs are likely to be virtually transparent at the first generation - you'd be hard pressed to see any differences at all on a first generation recording. Secondly, what pictures do you make your judgement on? You may well find that codec A is better than B on a scene with a lot of movement, but B is better than A on a static scene with a lot of fine detail. Other scene differences can similarly make one codec look good, another bad. It's vital to have a consistent and controlled variety of tightly controlled test scenes.

When such as the EBU have done such tests they have compared multi-generation performance to make any differences more obvious, but even that's not simple. You have to make sure that then it's not just a case of simply decompressing/recompressing with no other changes. Ensuring that pixel shifts are put in at each stage, and making sure that in the case of an interframe system the GOP structure gets staggered from stage to stage (so that the same frame is not always an I-frame) is the sort of thing that is neccessary.

Finally, all coders are not the same - even when compressing to the same codec at the same bitrate. A codec defines the DECODER - not the coder. And not all hardware coders are the same performance by any means, so the test in the form you describe may be testing individual bits of hardware - not the underlying codec.

Good points, David. Thus the difference between a professionally coded DVD that appears pristine from frame to frame, versus a DVD using the same Codec but using an inexpensive DVD encoding program. I do however still think that such a test would be interesting, and might perhaps show whether there are significant advantages to recording high bit-rate modes such as AVC-intra as opposed to 24 Mbs AVCHD.

I do however still think that such a test would be interesting

While AVC-intra, DNxHD, ProRes422, I-frame only MPEG2, MJPEG and JPEG2000 are all intraframe codecs, even at high bitrates there will be differences. See, for example,

Eugenia's Rants and Thoughts Blog Archive Intermediate Codecs: the face-off (http://eugenia.queru.com/2008/09/15/intermediate-codecs-the-face-off/)

Comparing the hardware implementations present in current recording devices would be interesting and may give surprising differences. Would you use some sort of HD-SDI splitter to feed an identical signal to multiple recording devices simultaneously?

I would also be interested in how the recorders handle noisy video produced by 1/3" chip camcorders. Noise can cause surprising artifacts. I wonder how the different recorders manage the compromise between filtering the noise and faithfully recording it.