Why no serious professional comparisons of prosumer equipment?

[Joseph H. Moore]

Why no serious professional comparisons of prosumer equipment?

This site is an incredible resource, so this is not meant to be a slam, but why is no one doing serious (dare I say scientific) reviews and head-to-head comparisons of these cameras the way that sites like dpreview.com do for still cameras?

Is the low-end pro video/indie film market that much smaller than the prosumer photography market?

Surely I'm not the only one dying for identical res charts and studio shots to accurately, and objectively, evaluate these cameras?

[Pete Bauer]

Mainly because the cameras either just have, or are just about to, reach the hands of shooters. No doubt a shoot out will happen pretty soon.

[Mathieu Ghekiere]

They are preparing a big test with all HD and HDV cams next to each other. But they want to take it very seriously, so they are taking the time they need.
(Barry Green even proposed to test the Varicam and Cinealta next to them, that would be great!)

[Joseph H. Moore]

That will be great, I surely look forward to it!

But the still camera review sites are loaned cameras even before official release dates and usually have detailed reviews day and date. In fact, this is the case with almost all gear. Why is "pro" video equip treated so differently?

Again, I'm not disparaging the incredible work that those on this board do for us all ... I'm just curious as to why the situation is so different with video cams versus still cameras? Are the manufacturers trying to hide? Is there not a big enough audience to support that kind of review site?

[Douglas Spotted Eagle]

Originally Posted by Joseph H. Moore

Are the manufacturers trying to hide? Is there not a big enough audience to support that kind of review site?

No conspiracy to be found here. First of all, no manufacturer is happy to have a pre-production camera examined as a "test" and it's not fair to do so when being tested against other cams.
Second, they're not all that keen to get cameras out there for testing by independent parties prior to shipping. It doesn't necessarily help sales, and can certainly hurt sales. Look at what happened to JVC in the first month of sales alone.
Finally, none of the camera manufacturers have really allotted time for cameras going out early. The moment a cam comes off the line, it's been already long-ago paid for by an excited user.
Professional products are rarely pre-reviewed in most any industry, so the clear answer to your question is that yes, professional products are marketed and managed differently than consumer products.

[Alister Chapman]

So many of the tests that I have seen on other sites tend to concentrate on feature lists and test charts. What we really need is unbiased tests by experienced camera operators who have no ties to any brand or product and no bias towards any manufacturer.

This is hard to find, as an example I have been using Sony kit almost exclusivly for 20 years, and being honest it has served me well. It's not always been the best or the greatest, but I have made a living using it, as a result if someone gave me 2 products to review and one was a Sony, my opinion would be biased before I had even opened the box.

This site does however give us the opportunity to read others real world experiences and that is worth more than any scientific test. You just have to figure out who likes which brands!

[Joseph H. Moore]

I'd have to dissagree with the statement that professional equipment is rarely available for early review.

Pro level DSLR's routinely are. (Production models, not pre.) The same is true of a lot of computer equipment. For that matter, most cars are a lot more expensive than these rigs and they're made available for serious testing before their street dates.

At the very least review equipment is made available day and date. I rarely see pro video cameras being gifted to anybody but fanboys.

So I'm left to ponder why the same companies (Sony, Canon) treat their DSLR customers so differently?

I think this situation is peculiar to this industry because we have a history of puting up with it. It's "pro-level" and we shouldn't question them beyond that. ;-)

[Douglas Spotted Eagle]

Originally Posted by Joseph H. Moore

At the very least review equipment is made available day and date. I rarely see pro video cameras being gifted to anybody but fanboys.

I truly wish this were either correct, or that folks that are getting free cameras or long-term "loaners" would say exactly what they're getting if they're doing reviews.

As far as pro gear, I've been reviewing professional audio, video, and musical instruments for nearly 25 years now, published in print and webzines, and have yet to see an early release of pro audio gear, video gear, or musical instruments. They're usually not made available until the day the product starts shipping to resellers/retailers. There are a couple rare exceptions, but they're rare.

The video industry is microcosmically small to what the photo industry is, so it's not at all an accurate comparison. Video is growing, but so is high end photo markets.

Further, it's a much bigger deal to be sending around a few 5,000.00 cameras, or higher cost cameras, than it is to send around a few 500.00 cameras. They only allot one or two units for review even after they're shipping in most cases. As a reviewer, on many occasions lately, I've had to use a credit card as a means of assuring that I'll return the product once it's been reviewed. Many big companies like Grass Valley, Ikegami, Sony, and others are turning to this policy because they can't afford to have a 7500.00 switcher or similar not coming back within an allotted time.

[Ash Greyson]

They will show whatever the tester wants... I can PROMISE you that there will be tests that will show JVC, Sony, Panny and Canon are EACH the best. The people who gravitate toward one or the other will focus on the data that supports their natural loyalty. Human nature, I mean, if I go drop $10K on an HVX and some P2 cards and someone evaluates that it has the lowest rez of all the aforementioned... I naturally wont be too happy. I may decide to focus on color space, dynamic range or someone elses test that shows otherwise.

I saw a test done by someone last night between a Z1 and an XLH1 and the EXACT SAME RESULTS were used to show both were the better camera! XLH1 advocate says, "look how much more resolution and dynamic range the Canon has." Z1 advocate says, "The Z1 has better resoultion to the corners, not just in the middle and has dramatically less noise."

Bottom line, these cameras are different. Choose them based on a workflow you like, what the project is most suited for, or what is within your budget. There is no best camera...period endstop...



ash =o)

[Joseph H. Moore]

Hehe, and I wish a Canon EOS 5D or a BOXX workstation were $500!!! ;-)

I understand what you're driving at, though.

Still, lets limit ourselves to shipping equipment (all of the major contenders are in the marketplace now) there still is not a site (that I've found) that offers objective, unbiased, critical (repeatable) testing of this gear.

And that's a shame. We're left relying on the kindess of a few early adopters to try to divine the truth. I don't do that with $10k workstations, and I shouldn't have to for $10k cameras.
<End of rant.>

[Les Dit]

I think the Internet, with places like this, will solve the problem. Pre-Internet, we had magazine reviews. But they never had 'scientific' , to use your term, tests. Most of the time, they said the good things about how great the camera worked. Sell Sell Sell ! Guess who pays the bills? The same companies placing ads in the mag. The Icing on the cake was the review itself being confirmation of a fantastic product.
Perhaps I'm being overly critical of the bias.

Another reason they are lacking in quantitative tests is the public they are selling to are not educated to understand and have trust in the numbers.

Maybe I'm wrong.
-Les

Originally Posted by Joseph H. Moore

Why no serious professional comparisons of prosumer equipment?

This site is an incredible resource, so this is not meant to be a slam, but why is no one doing serious (dare I say scientific) reviews and head-to-head comparisons of these cameras the way that sites like dpreview.com do for still cameras?

Is the low-end pro video/indie film market that much smaller than the prosumer photography market?

Surely I'm not the only one dying for identical res charts and studio shots to accurately, and objectively, evaluate these cameras?

[Petr Marusek]

Tests are usefull but you have to learn how to read between lines. The same goes for any type of products, cars, light bulbs, or whatever. Often it is the company that pays for the test, directly or indirectly, is the one, whose products come on top.

[John Jay]

In the UK, at the moment, a PAL DVX100BE is on offer at <£2K and according to tests can record a 540x540 resolution signal on a 25 Mbps tape

How can justification be made for 100 Mbps recording on special media for a 600x600 resolution signal from a HVX200E?

I just dont get it, it seems like its more to do with selling P2 than selling a decent camera head.

[Pete Bauer]

Originally Posted by Joseph H. Moore

Surely I'm not the only one dying for identical res charts and studio shots to accurately, and objectively, evaluate these cameras?

Woohoo! Looks like we found a volunteer to take care of all the planning, coordination, travel, leg-work, clean-up, and most importantly, financial backing for an HD shoot-out! ;-)

But silliness aside, Joseph, a number of us have expressed interest in putting together something (this has been all off-line discussion), but it really does take a lot of time, planning, work, and money to make something like that happen and be meaningful to devotees of digital video...not to mention the HVX200 is barely starting to hit the streets. Even when we know we can get all four major contenders together in one place at one time, from a business standpoint, what financial incentive is there for anyone interested in video to put forth that kind of effort? I think that's why you don't see real head-to-head in-depth comparisons out there...it isn't in anyone's business interest to do so.

So it is left to us DVinfo'ers to put something good together, for no reason more than the love of digital moving art. No doubt some sort of shoot-out will happen in due course...

[Joseph H. Moore]

That's why I asked about audience size. i'm sure dpreview and similar sites don't do their reviews out of the goodness of their hearts! ;-)

If the audience is sufficient, then the cost can be dealt with. (Can you tell I'm an advertising CD by day?)