HD Camera Shootout: "Let the games begin" : In this corner we have.....

[Shannon Rawls]

The festivities begin Wednesday Jan 11th @ 9am. Barry Green called in some HUGE favors and set everything up.

So LETS GET READY TO RUUUUUMMMMBLLLLLLLLLLLEEEEEEE!!!

In the red trunks weighing in @ 960 pounds we have the Sony HVR-Z1U (provided by shannon rawls)
In the green trunks weighing in @ 720 pounds we have the JVC HD100 (provided by adam wilt and/or nate weaver)
In the blue trunks weighing in @ 1080 pounds we have the Canon XL-H1 (provided by shannon rawls)
In the vari-colored trunks weighing in @ various weights we have the Panasonic HVX200 (provided by barry green)

Guest appearnaces by the two heavy-weight champions of the world:
Sony CineAlta (provided by big vision studios)
Panasonic Varicam (provided by big vision studios)

Confirmed testers & guests:
1. ADAM WILT, AUTHOR, VIDEO SYSTEMS ENGINEER & DP @ www.AdamWilt.com (TESTER) [unbiased]
2. BARRY GREEN, DIRECTOR, PRODUCER, AUTHOR & CAMERA OPERATOR @ www.icexpo.com (TESTER), [advocte of MAC & Panasonic]
3. NATE WEAVER, DIRECTOR & DP @ www.nateweaver.net (TESTER), [advocte of MAC & JVC]
4. JAY (not sure who this is but Barry said he'll be there)
and
A. SHANNON RAWLS, MOVIE PRODUCER @ www.CinemaHill.com (DOCUMENTARIAN) [advocate of PC & whatever looks best]
B. EMPLOYEES OF THE STUDIO (GUESTS) [advocates of unknown]
C. whoever else Barry has invited that I don't know about

Barry has invited the following unconfirmed testers & guests:
I. MIKE CURTIS @ www.HDForIndies.com (GUEST) [advocate of MAC & Panasonic]
II. GRAEME NATTRESS @ www.nattress.com (TESTER) [advocate of MAC & Panasonic]
III. GORGEOUS BLONDE ACTRESS, head-shots or perhaps a cinema-style scene setup (GUEST)

Current "real world" & "practical" performance tests to be conducted:

AUDIO
1. Using a simple Audio-Technica AT897($250) shotgun mic fed into a simple Behringer MXB1002($99) mixer, line level out split to all 4 cameras at the same time, cf25/24p/24f modes, Line Level In, Limiters/Noise Reduction/Auto Levels all ON
2. Using onboard built in Microphones that come with each camera set at an equal distance from the male human subject, cf25/24p/24f modes, Limiters/Noise Reduction/Auto Levels all ON
audio note: we will record roughly 10 seconds of audio & video of the male human subject as he reads the headline on the 1st page of the Los Angeles Times Newspaper.

VIDEO
1. RESOLUTION TESTS:
All 4 cameras will be white balanced to the same white card and placed in MANUAL MODE with AUTOFOCUS ON. Their picture profiles/custom presets/image settings will all be turned OFF. If there is a setting that cannot be turned off, then it will be set to "0" or whatever the 'middle' setting is. Each camera will be placed in the exact same spot as its competitor when testing. We will record 10 seconds of resolution chart footage for each....
1080 HD
a. cf25/24p/24f modes, 0db gain, 1/48th shutter speed¹, aperture TBD (z1u/xl-h1/hvx200)
b. cf30/30p/30f modes, 0db gain, 1/60th shutter speed, aperture TBD (z1u/xl-h1/hvx200)
c. 60i mode, 0db gain, 1/60th shutter speed, aperture TBD (z1u/xl-h1/hvx200)
720 HD
a. 24p mode, 0db gain, 1/60th shutter speed, aperture TBD (hd100/hvx200)
b. 30p mode, 0db gain, 1/60th shutter speed, aperture TBD (hd100/hvx200)
480 DV
a. cf25/24p/24f modes, 0db gain, 1/48th shutter speed¹, aperture TBD (all 4 cameras)
b. cf30/30p/30f modes, 0db gain, 1/60th shutter speed, aperture TBD (all 4 cameras)
c. 60i mode, 0db gain, 1/60th shutter speed, aperture TBD (all 4 cameras)

2. CAMERA MOTION TESTS:
All 4 cameras will be panned left, right, up & down across a correctly lit subject to evaluate a moving cameras motion cadence and consistency. Cameras will be hand-held by Adam Wilt with Image Stabilizers OFF, MANUAL MODE, matching gain, matching aperture....
1080 HD
a. slow pan: cf25/24p/24f - 1/48th shutter speed¹ (z1u/xl-h1/hvx200)
b. fast pan: cf25/24p/24f - 1/48th shutter speed¹ (z1u/xl-h1/hvx200)
c. slow pan: cf30/30p/30f - 1/60th shutter speed (z1u/xl-h1/hvx200)
d. fast pan: cf30/30p/30f - 1/60th shutter speed (z1u/xl-h1/hvx200)
720 HD
a. slow pan: 24p - 1/48th shutter speed (hd100/hvx200)
b. fast pan: 24p - 1/48th shutter speed (hd100/hvx200)
c. slow pan: 30p - 1/60th shutter speed (hd100/hvx200)
d. fast pan: 30p - 1/60th shutter speed (hd100/hvx200)

3. SUBJECT MOTION TESTS:
All 4 cameras will be locked on sticks simultaneously recording 20 seconds of a correctly lit subject to evaluate a moving objects motion cadence and consistency. The subject will be Shannon doing the old man version of the Karate Kid and possibly trying to break-dance *smile* Cameras will have Image Stabilizers OFF, MANUAL MODE, matching gain, matching aperture....
1080 HD
a. cf25/24p/24f - 1/48th shutter speed¹ (z1u/xl-h1/hvx200)
b. cf30/30p/30f - 1/60th shutter speed (z1u/xl-h1/hvx200)
720 HD
a. 24p - 1/48th shutter speed (hd100/hvx200)
b. 30p - 1/60th shutter speed (hd100/hvx200)

4. LOW LIGHT TESTS:
part 1 = We will create a dark shooting environment probably lit by a single candle being handheld up to their face by a human subject. All 4 cameras will be on sticks with Image Stabilizers OFF, MANUAL MODE, matching aperture & GAIN level locked at 0db....This test should allow us to determine which camera gives the brightest pictures of the "out of the box" internal circuitry performance. We will roll 20 seconds of footage simultaneously....
1080 HD in Low Light @ 0db Gain
a. cf25/24p/24f - 1/48th shutter speed¹ (z1u/xl-h1/hvx200)
b. cf30/30p/30f - 1/60th shutter speed (z1u/xl-h1/hvx200)
720 HD in Low Light @ 0db Gain
a. 24p - 1/48th shutter speed (hd100/hvx200)
b. 30p - 1/60th shutter speed (hd100/hvx200)

part 2 = We will now do the same above test, but this time we'll keep the camera who got the best performance @ 0db. As for the other 3 cameras we will turn the GAIN LEVEL up until we see the first signs of Grain in the image. If a smidget of Grain is detected, we will back down to the next lower decibel until it's gone. At which point we will determine which camera can "SEE" the best in the dark regardless of its Gain setting. This test should allow us to determine which camera is slower or faster then its competitor in terms of iris STOPS and lens performance.
1080 HD in Low Light
a. cf25/24p/24f - 1/48th shutter speed¹ (z1u/xl-h1/hvx200)
b. cf30/30p/30f - 1/60th shutter speed (z1u/xl-h1/hvx200)
720 HD in Low Light
a. 24p - 1/48th shutter speed (hd100/hvx200)
b. 30p - 1/60th shutter speed (hd100/hvx200)

5. SENSITIVITY/LATITUDE/DYNAMIC RANGE TESTS:
Different titles but they all have the same definition: This test will determine each cameras ability to "see" BOTH details in shadows and in highlights.
This test should ONLY be done with picture profiles/custom presets/image settings all turned OFF. If there is a setting that cannot be turned off, then it should be set to "0" or whatever the 'middle' setting is.
I'm not sure how Barry, Adam & Nate will test this

6. COMPRESSION ARTIFACT TESTS:
This can be tested by fast camera motions (whip-pans) in bright daylight.
This test should ONLY be done with picture profiles/custom presets/image settings all turned OFF. If there is a setting that cannot be turned off, then it should be set to "0" or whatever the 'middle' setting is.
I'm not sure how Barry, Adam & Nate will test this

7. NOISE LEVEL TESTS:
This test will look for shadowy areas with blockiness and pixels that get translated to downconverted DV and film blow-up's.
This test should ONLY be done with picture profiles/custom presets/image settings all turned OFF. If there is a setting that cannot be turned off, then it should be set to "0" or whatever the 'middle' setting is.
I'm not sure how Barry, Adam & Nate will test this

8. COLOR PERFORMANCE TESTS:
This is how we determine how the CCD's handle the various colors of the rainbow and displays them on screen. Personally, I think this is a waste of time, since each camera has tweakable color settings that are infinitly changeable. If we do do this test, it should be done with picture profiles/custom presets/image settings all turned OFF. If there is a setting that cannot be turned off, then it should be set to "0" or whatever the 'middle' setting is.
AND AFTERWARDS, this same test should be done with each camera set at it's OPTIMAL SETTING for the shot, including stretching blacks, cine-gammas, matrix settings, etc...
I'm not sure how Barry, Adam & Nate will test this.

9. FILM-LIKE / MOVIE-LIKE FEEL & EMOTION TESTS:
SO MANY parameters are involved in effectively doing this. I personaly think we should setup some type of gorrilla style scene where we utilize the actress outside in broad daylight and shoot all hand-held.
Either way, this test should ONLY be done with each camera set at it's OPTIMAL SETTING for the shot, including stretching blacks, cine-gammas, matrix settings, etc...
I'm not sure how Barry, Adam & Nate will test this.

____________________________________

In addition to the Internet site Chris will be setting up and DV Magazine that this test will be published in by Adam Wilt, I also think the behind the scene photos and video I will be taking should be authored to a cool DVD available right here at DVINFO.NET for $5.00 each. The proceeds should be split between Barry Green for setting up the shootout and Chris Hurd for hosting it.

¹The Sony HVR-Z1U cannot shoot 1/48th shutter speed. In this case, we will have it set at 1/50th.

- ShannonRawls.com

[Shannon Rawls]

Also,

what I propose, is that after the testing is done. We create high quality .WMV video & .PNG photo clips of each test and then upload the results to Chris Hurd (who I believe is forced to be fair & unbiased due to the nature of this website) and simply have him setup a small section on this site and allow people to vote on the footage without revealing which camera is which. Next Monday, reveal the camera that belongs to its clip and publish the vote results. I will keep my lips zipped if everyone else does. *smile*
Alternatively, we can just let the cat out the bag and announce the winners right away.

- ShannonRawls.com

[Graeme Nattress]

I can't make it, but really would have loved to. Although I'm not a Panasonic advocate, really. I'm highly interested in the concept of recording video to memory, hence my interest. I'm also on record saying both HDV and DVCproHD are "overcompressed".

I think this is going to be a fascinating experiment.

I think what we'll see if that each camera is a compromise - the answer is the one to pick the compromise that meets YOUR needs, based on facts, not predjudice.

Graeme

[Jim Michael]

Originally Posted by Shannon Rawls

part 2 = We will now do the same above test, but this time we'll keep the camera who got the best performance @ 0db. As for the other 3 cameras we will turn the GAIN LEVEL up until we see the first signs of Grain in the image. If a smidget of Grain is detected, we will back down to the next lower decibel until it's gone. At which point we will determine which camera can "SEE" the best in the dark regardless of its Gain setting. This test should allow us to determine which camera is slower or faster then its competitor in terms of iris STOPS and lens performance.

Can someone explain why the best performer in the 0 db test is held at 0 db in part b, while the other cameras are effectively adjusted to their best low light setting? Looking forward to the results. Thanks.

[Mike Marriage]

Shannon,

That looks outstanding! Thanks to everyone involved!

Just a few points to raise, (by which I mean no disrespect of your hard work):

1)
The DSPs are probably doing weird auto-gain processes. Well it sounds like the HVX is anyway. For the low light testing, it may be best just to set the cameras all up so that they each show the same brightness level whilst shooting the same, dark scene. The gain level doesn't really matter because it is something only the operator sees and has no comparative value across the cameras. We can then judge which has the best low light performance taking into account the nosie level.

2)
The XLH1 is clearly doing some intelligent deinterlacing when in 24F/25F/30F mode. Therefore a static chart will give an artificially high result. A moving shot is required, although I can't see how this could be done and still achieve a numeric value. It would have to be a simple "eyeball judgement" of resolution.

3)
Can you sling an XL2 (DVX, DSR 450) into the mix as a baseline of what we know already?

4)
Please don't make us wait until Monday! The riots will destroy civilisation as we know it!

Thanks again, in advance to all involved!

[William Hohauser]

6. COMPRESSION ARTIFACT TESTS:
This can be tested by fast camera motions (whip-pans) in bright daylight.
This test should ONLY be done with picture profiles/custom presets/image settings all turned OFF. If there is a setting that cannot be turned off, then it should be set to "0" or whatever the 'middle' setting is.
I'm not sure how Adam & Barry will test this

In addition to a whip pan, I suggest a whip tilt. Try several speeds of pans; slow as if filming a dramatic scene, medium as if following a runner, fast as if following a car race. I'm not suggesting finding these subjects just simulate the camera movement. It will be interesting to see if the compression breaks down past a certain speed of movement.

A more complicated suggestion is to find a natural scene such as trees in the wind or boats on water where the camera is stationary but the scenery isn't. This was where the JVC HD-100 started to lose it with JVC's own demo footage. Their shot was a medium angle down on a penguin floating in wavy water.

Otherwise, kudos to everyone participating in the test! You are helping thousands of people!

[Mathieu Ghekiere]

I agree about maybe putting the XL2, one of the best SD cams out now, next to the test, if someone has the cam and willing to help with the test.
It could give us maybe a better point of refference, or an EVEN better point of reference.

And I forgot, thanks indeed to everyone involved!

[Petr Marusek]

You should test Z1 @ CF25, not CF24. CF24 is bad; we all know it and for all practical purposes there is no difference between 24 and 24 fps, because the conversion is simple. Europe is 25, America is 24, you always do the conversion for the other market. Why is then the Sony camera being tested at 24p?

XL2 should naturally be tested too as a benchmark/point of reference.

[Petr Marusek]

Originally Posted by Shannon Rawls

¹The Sony HVR-Z1U cannot shoot 1/48th shutter speed. In this case, we will have it set at 1/60th.

It can be set to CF25 and 1/50 sec. shutter, which for all practical purposes is the same as 24 fps and 1/48 sec.

[Glenn Chan]

Resolution test #1: I think you should also try panning the camera very very slowly. This will be a bit fairer since it won't penalize camcorders when the line lands "between" pixels.
Also, the panning will show any aliasing artifacts that sometimes happens on edges.

If you want to be really technical, you could measure sharpness across various f-stops.

8. COLOR PERFORMANCE TESTS:

If you break light up into individual wavelengths (i.e. with a prism or diffraction grating), you can see how the camera performs at the cross-over frequencies.
Each element in a camera's sensor has a color filter placed over it. The color filter rejects light at certain frequencies, and lets light from other frequencies enter. If you plot this on a spectral power distribution graph, you will see a plateau.

Where the color filters do or don't overlap, you'll see interesting things happen.

I'm not sure how useful such a test would be. It could be used to predict metamerism, or how well two different cameras will match.

2- Pet request: Maybe you can test bokeh and depth of field?
Bokeh is how out of focus images look. It would be particularly good to see how an out-of-focus point light looks light as that reveals a len's bokeh.

It's also good to see a rack focus (as close to real world as possible)... it would show bokeh, depth of field, and breathing in the lens.

3- I think it's good to have more real world tests. Sometimes, results from synthetic tests don't translate to the real world. Synthetic tests are good however for making differences visible, and making these differences measurable.

Unfortunately some real world tests would be hard to conduct. A good test would be to take the camera out on a real project and shoot with it. Of course your results would not be very objective, and is subject to tester bias.

Another real world test you could conduct is to shoot various scenes (i.e. landscapes, indie film, etc.), and screen them in front of an audience of professionals and typical viewers. Do a double blind test to judge:
A- picture quality
B- whether the audience can figure out what images came from what camera.

Other possible real-world tests:
Can I shoot a wedding / corporate / whatever video with this camera?
What's good and bad about the camera for shooting such a project.

Try to shoot something which maximizes exposure latitude using all available camera settings.

Shoot something and bring the footage through an entire workflow. Color grade it, and make a deliverable from it. If extensive color grading is a part of some workflows, throw in some secondary color correction. In my opinion: the secondary key/mask from HDV (from the JVC 24p footage) looks nasty, as does DVCPRO HD. Very noisy in both cases. The dark areas especially have problems.

Do some green screen footage.

Try to match the cameras to the Sony Cinealta or Panasonic Varicam.

Throw a SD cam in the mix!

[Charles Papert]

I agree with the idea of shooting a moving image such as water or fire which will be a more challenging test to the compression than a pan.

[Barry Green]

A couple of updates: Nate Weaver, master of the HD100, will be joining us to represent and ensure best performance from it. Rush Hamden of DVInfo.net sponsor EVS will be coming by to participate in the "blind judging." And the studio is providing the CineAlta, the VariCam, the monitors, waveform, vectorscope, and an excellent camera engineer.

We have been graciously supported by the providing of equipment from some very kind companies. Thanks to BandPro for supplying us the $4,000 Ambi Combi 2 resolution chart (no cheeseball home-printer printouts here!); thanks to DSC Labs for supplying their top of the line ChromaDuMonde resolution chart; thanks to ProMax Systems for supplying a tricked-out Mac/FCP/BlackMagic uncompressed editing system for side-by-sides, and of course big thanks to Big Vision Studios for making available such top-end gear and their space.

As far as the testing regimen, there is no way that we can satisfy all requests, and with six cameras in the mix we will probably be fundamentally overwhelmed in getting half the tests that Shannon listed.

Our main goals are these:
1) Get definitive answers on each camera's resolving capability, in the crucial modes (1080i & 1080/24f for Canon, 720/24p for JVC, 1080/25F for Z1U, 1080/24p & 720/24p for HVX, 720p for VariCam, 1080/24p for CineAlta). This will include simple static res chart testing, as well as some expanded testing being supervised by Adam Wilt.

2) Get definitive answers on sensitivity for each.

3) Get definitive answers on dynamic range for each

4) Get definitive answers on noise performance for each

5) Conduct at least one real-world outside setup, and one properly-lit indoors setup

6) Evaluate footage against the benchmarks (VariCam/CineAlta) as well as against each other, in blind comparisons and split-screens, where the viewers will not know which camera's footage they are seeing.

6) Optionally, if there's time, conduct a green screen shoot & composite on each

The subjective tests are designed to see how each looks, regardless of test chart performance. To determine which ones are most pleasing, and how the judges feel they compare against the benchmarks, and against each other. Test chart answers are nice to know, but they're not what people see when they watch the image; we want to try to find out both answers: how do they perform in the lab, and how do they perform in the real world.

We've done two- and three-camera comparisons before. Two is easy. Three is exponentially harder. Four would likely prove exponentially harder than three. Six will likely prove absolutely daunting. So there may not be a lot of time to do a wide variety of tests; it is our goal to make sure that the tests we are able to conduct will be conducted thoroughly, properly, and error-free. And we'll go until we run out of time! :)

[Shannon Rawls]

Originally Posted by Barry Green

Nate Weaver, master of the HD100, will be joining us to represent and ensure best performance from it.

Excellent! This is a good opportunity to meet Nate in person since I may be hiring/flying him to Atalnta with us for a Music Video at the end of the month that I want him to DP!
NATE...Bring those videos on DVD for me bro!

- ShannonRawls.com

[Graeme Nattress]

I'd also be very keen to hear what the 1080i60 resolution of the HDCAM Cinealta is if it has that mode as well as 24p.

Graeme

[Pete Bauer]

This is really great; thanks in advance to everyone involved! May I also ask that the XL H1 be tested using both HDV-firewire and HD-SDI via either uncompressed or Cineform Prospect HD...or if you can't do both, please make sure to specify the data path actually used. Hope you can do both HDV and HD-SDI, though.
As everyone, I eagerly await the results!