VideoAktiv Digital Comparisons of Camcorders

[Larry Horwitz]

The German website linked below has full resolution videos for the current and recent vintage camcorders which contain very nice, repeatable tests at both high and low light levels.

Downloading and comparing allows an excellent means to see how the three high end competing camcorders perform.

Select the camcorder test video option on the left hand side of the page to get a list of all the manufacturers / camcorder choices. It is called "Cam-Test Vidios"

I would interested in opinions of other users as to which really does look the "best" with low light conditions.

Larry

Google Translate

[Dave Blackhurst]

Well, aside from the "reviewers" being a bit clueless and apparently unable to find the low lux button on the Sonys... CCI just botched their low light evaluation (well there you go again), I guess the reviews are semi-accurate... sorta...

The German site was very slow, downloads were glacial (gave up on trying), and the "video" that was posted to the site for quick playback looked so pixellated I wouldn't evaluate anything based on it (at least not the camera... the operator, that might be another story).

I have a big problem with "nice repeatable tests", because they STILL always have noticeable variables... Reviews can help you evaluate the performance of a camera, but must be taken with a grain of salt (about the size of a sugar cube in many cases). The OTHER German site has posted a couple "optimized" low light tests that better reflect what actual user experience has been, and would give the Sony a clear win.

So many variables, and in some cases long running and somewhat obvious biases, you learn to sort through it.

Panasonic has definitely upped the ante significantly with their latest cameras, I sure wouldn't be embarrased to shoot with one, and Canon is keeping the heat on too, but having had enough experience with the Sonys, I know what they can do if you know how to use them.

When you see "low light" tests that simply let the other cameras shift into slow shutter modes or alternate frame rates (Sony is set to OFF by default, yep, is 60i... and holds onto 1/60 shutter unless you allow it to go slower), and not turn the function ON on the Sony (or worse yet alter the lens range/aperature like CCI did) it's just goofy.

If the image degrades significantly in low light (the Sony tends to hold it together, quite gracefully), I don't care if it's "brighter". Brighter and noisy, or brighter and grainy with muddy or blotchy spots are not what I am looking for. When I'm shooting in "bad" lighting conditions, I want the camera to give the least noisy, sharpest image, with usable brightness, as possible. Short of the possibilities with a DSLR-V, the little Sonys are about as good as you can expect to get.

PS - I love it when the reviews are "bad" on the Sonys - the prices drop and I can pick 'em up cheaper from people who read reviews rather than shoot!

[Larry Horwitz]

Dave,

I too have found the website to be extremely slow this weekend. My prior downloads for other camcorders with the same size files (approx 170 MB .mts originals) took 20 or 30 seconds previously, but took 7 minutes yesterday. Perhaps they are having server issues but not entirely sure. I have not tried to use the "quick" samples, and am not surprised to hear that they are pixellated and inadequate for critical examination.

The files I have downloaded for the Canon HFS21, Sony 550, and Panny TM700 seem extremely useful to me. They are all taken in the same controlled panning, lighting, and studio subjects which have a wide range of color and tonal range. They tests are all the same, with only the camera changing from one file to the next.

I can't tell you whether they forgot to hit some switch on the Sony, and thus gave it some penalty. My purpose in posting this and the prior post for another comparison site was merely to inform critical buyers that these sites exist.

I will not attempt to justify or rationalize how accurate, repeatable, or reliable this data may be, and I have no dog in this race. The Sony may indeed be the very best of the 3, but I am not posting here to start a thread of "my choice is better than your choice". I merely want to guide users to other possible sources of comparative information, no more, no less.

As a retired electrical / electronics engineer, I look askance at many if not most testing methods and organizations, and am often disgusted, occasionally appalled at how seldom good scientific method is applied in testing and evaluating products in general, and sophisticated electronic products in particular.

Nonetheless, the more sources of information become available, the better the prospect for gaining some insights. And to my own taste, camcorderinfo.com is a very weak source of useful information, as is Videomaker magazine, another widely distributed source. I frankly think the 2 German sites I have posted in this and my prior thread are vastly superior.

I hope others also find this to be of some use.

Larry

[Dave Blackhurst]

Slashcam (the other German site) seems fairly accurate in general, with some questionable "test" methodology, CCI is fine if you remember they really don't like Sony very much, and seem to introduce a LOT of errors.

Information is all fine and good, but INaccurate information isn't isn't really information at all...

That's one of the advantages here of being able to communicate with people who know a little about cameras, how they work, what the strengths and weaknesses are, and can discuss the finer details in a civil thoughtful forum dedicated to advancing the use of video... whatever it's shot with.

There's lots of room to critique ANY piece fo technology, it's just frustrating to see completely inaccurate portrayals or "opinions" put out as "fact", and being "sold" to unsuspecting readers - I noticed that VideoAktiv was eager to sell copies of their reviews, and promote their magazine.

I won't be a "buyer" of their reviews or waste the time to go back based on what I saw. But it never hurts to take a look when the lookin's free, and I was intrigued by their apparent use of "hot" pink/flourescent swatches to test the cameras, still wondering how accurate or "off" those colors really were! The Panasonic shots were so "hot" they practically vibrated off the screen! I've got to wonder whether the sample swatches were REALLY that over the top...

[Larry Horwitz]

Dave,

I do not have any of the 3 new camcorders I wish to compare, so I personally do rely on the sites I visit to offer some degree of accuracy and integrity.

I really would appreciate if you would clarify / illustrate your specific criticisms which I have lifted verbatim from your prior post:

"questionable test methodology"
"they introduce a LOT of errors"
"completely inaccurate portrayals"

These are harsh accusations, and I would welcome your comments to actually substantiate these concerns you have voiced.

Larry

[Dave Blackhurst]

Take a look at the comments already posted on the CCI review of the CX550V... In the review they rave about the new WA lens, but then zoom it in during testing (thus reducing the iris) and declare it lousy in low light. IIRC, even they admit they are not so sure about their testing methodology, or the results.

They give a "0" for the lack of a gain control on the Sony, but given some of the processing tricks that are in use in these cameras, I'm not even sure there IS gain in a traditional sense. In use, the noise levels in images from these cameras seem to indicate some other digital magic is going on, yet because it doesn't have a familiar label, it ranks a "0"...

I'm not being harsh, just saying there are errors in methodology that are glaring, and some odd comments (like those about the LANC interface, the iA button, which IS NOT the same as the old EASY button), and so on... the reviews are interesting and entertaining, but like a weather report, don't use 'em exclusively for your decision making process.

If you want to accept the poor low light performance reviews given the Sony, which are completely 100% diametrically opposed to the optomized low light still captures from Slashcam, as correct, that's up to you. I can only suggest that there are some obvious contradictions even between these various "reviews", and that's before you get to real life user experience.

I've got "last years models" - the XR and CX500, they are good in low light and once low lux mode is turned on, they perform under horrid light conditions and still give good image quality, with low additional noise. Maybe Sony "broke" the CX550, but I doubt it. I've got a couple of their P&S cameras, and they improved the low light significantly between model years.

ALL the manufacturers make improvements and feature changes, it's hard to keep up with the specs and performance, let alone the ocaissional incorrect info that the manufacturers themselves release (see the thread on the Sony MC50E...). You ALWAYS have to double check specs and such, those minor errors happen regularly in "internet time".

FWIW, all the flagship cams this year are pretty close, unless there are specific features that one has and the others don't. I don't care about 24p & 30p - if you do, that would be a important part of your decision making process, and so on.

I used to think the news was accurate, then I started to notice that in areas I knew something about, "accuracy" was pretty iffy, or "just good enough" to slide by. In the end, you still need to check your "information" if it's your money on the line.

As far as cameras go, the Panasonic seems to be the leader in a lot of respects, and you probably can't go wrong with it. From everything I've seen, it's a bit sharper than the already quite crisp Sony. Personally, I want the wired LANC, the bigger LCD, and I'm a little unsure about the color rendition I've been seeing from Panasonic samples (probably correctable if you play with the camera a bit, but somehow I doubt everything is "neon" in Germany...and those aren't the first samples that just seemed a bit odd to my eye).

I also am happy with the very effective OIS approach Sony uses (the CX550 did win on that count in the CCI review), and I get great "real world" low light results since I know how to use the camera... I don't know how the reviewers are getting such off the wall results, but it raises some questions when such a key "feature" as low light performance is rated so negatively.

[Larry Horwitz]

If your prior comments regarding the integrity of testing on review websites was explicitly referring to, then I share your belief that their site is extremely weak.

The 2 sites I referenced in this thread and my prior thread, both of which are written in Germany, impress me as being far, far, less guilty of the issues you raised. It was for that reason that I suggested and cited them.

Your reply to this thread and my prior thread implied (to me and perhaps others) that you felt this negatively about the web sites I had recommenced.

Just to clarify this point, are you also stating that the 2 sites I recommended are also guilty of the same problems.

Thanks for the clarification.

Larry

[Dave Blackhurst]

Larry -
Where you're making a mistake is to "rely on the sites I visit to offer some degree of accuracy and integrity".

If you consider "some degree" to be anything over a "C" (passing, barely) grade, that's a safe assumption... with wildly varying grades from an A to an F (or a 1 to a 5 or whatever) depending on the situation. Sometimes even the guys here on DVi (myself included) will stick our foot in our mouth or make a mistake, but typically if you follow the thread, we self correct. In fairness, I've seen CCI publish addendum/corrections, but if an initial review is too far out of gamut, IMO they should hold the review rather than publish something obviously "off" (at least they did issue a disclaimer of sorts on their low light rating). Sites like Tom's Hardware will go back to the manufacturer if they get oddball results, and heavily note when they think something is wrong with their test results.

This is the internet, and things operate in "internet time" where SPEED tends to have a higher priority than accuracy. It's particularly true for Sony products which I've noticed tend to be available to the public sometimes weeks before the review sites get a crack at them (people here had the CX550V weeks ago). This no doubt leaves the review sites scrambling to get "something" out to maintain their relevance, and so I have suspected they get limited time for "hands on" use to learn what the cameras can really do.

Put in a different perspective, I'm sure there were lots of glowing reviews of the Toyota Prius (and I think they are nifty myself, though I have no use for one)... and now there are some issues.

Reviews give you a "snapshot", some sites do a better job than others, and if you realize that going in, you'll find them useful to some degree.

FWIW, Slashcam is the more useful and thorough of the three from what I've seen. Their results tend to be pretty consistent to "real world" results, and their comparison tests are consistent enough to allow a viewer to develop an opinion. CCI is a good read, but I don't put a lot of weight on their reviews - for whatever reason they seem to be more familiar with Canon and Panasonic and not as fond of, or knowledgeable, on Sony product. I hadn't heard of VideoAktiv, and offhand I'm not that impressed, but I'm sure you can learn some useful information from the site - based on what I was able to access, I'd put it closer to CCI than Slashcam...

One of the crucial skills of the internet age is being able to sort out the wheat from the chaff, the signal from the noise, and the true from the erroneous. Before you "rely on" anything you find published, I'd apply the "trust, but verify" approach...

I have the last generation Sony cameras, the real life low light results are significantly better than what these "reviews" claim. With such a glaring error, confirmed by other owners of these cameras, including several here on DVi, I have to call it as I see it.

[Larry Horwitz]

Dave,

I have no doubt that Internet journalism is far less reliable than truly authoritative and well researched testing done by qualified people, preferably with technical degrees as well as hands-on practical experience.

Despite this, most if not all of us who contemplate a purchase will gather information from a variety of sources, and attempt to derive a consensus based on reading several opinions. Although forum members like you and me are able to share first hand, anecdotal information which often is far more valuable than the review web sites, I personally believe that there are huge benefits to be gained by knowing about, reading, and evaluating these sites, and I thus entirely reject the notion that these sites should be either dismissed or avoided since they are guilty of ""questionable test methodology", "introduce a LOT of errors", and are "completely inaccurate portrayals". To be sure there are examples of sloppy, inaccurate, and erroneous data to be found, but I personally believe that drawing a consensus from reading a lot of sources is a stunning benefit of the Internet which was difficult if not impossible to achieve in the era preceding online reviews.

You keep coming back to your specific Sony, and seem to be especially dismayed as to how it has been maligned. I have no particular knowledge of how great or poor the Sony actually is, but I would certainly seek out the opinion of users like you before making a selection. This in no way whatsoever diminishes my interest in using as wide a collection of web review sites as I can find. Perhaps they have, collectively, under-rated the Sony, or made some other glaring errors, but in general I think that some technical value can be found in reading and comparing these sources.

I therefore seek out and aggressively read world-wide opinions, and have found and use many foreign and U.S. sources for my purchases of everything from lenses to automobiles to my recent purchase of a new washing machine.

Having subscribed to Consumers Reports since the 1960's without any interruption, and having spent most of my life in electrical / electronic engineering, I am always looking for more information and different opinions to add to my knowledge.

Therefore, I have a real problem with any general dismissal or categorical rejection of Internet sources as failing in the manner you describe, but I do agree that any buyer would be best served by seeking owner opinions on forums such as this one to truly understand the pros and cons of specific products.

Larry

[Dave Blackhurst]

You sound like you research the same way I do... and that is GOOD. Where I hesitate is when one knows a site is prone to bias or inaccurate information, pointing the uninitiated towards that site. Not everyone has a well developed FUD filter.

Remember that you or I might be able to sort out the inaccuracies, but the average person might well go to let's say CCI, and conclude the Sonys are blind as can be in the dark - they've consistently "underrated" Sony cameras on low light, this is NOT the first time, it's a consistent "bias". This would lead to a completely incorrect conclusion for the reader.

So to simply point to a site and say 'download and compare" for an "excellent means to see how the three competing camcorders perform" when the accuracy of the data is suspect is not a reasonable way to "compare" in my book (I'm going from the posted stills, since the downloads were glacial). It's a way to add data points, but you have to remember to toss the "bad" data, and I am always more cautious of accepting "data" from a source which has clearly suspect information presented as accurate. Wheher caused by sloppy testing or an agenda, one has to be aware of such things and filter accordingly.

[Cristian Adrian Olariu]

I just downloaded a bunch of samples from VideoAktiv and I didn't find anything suspicious until I was shocked by the Sony NX5 sample which looks awful compared to Sony AX2000, but the samples should look pretty much identical. IMO the website is not reliable.

[Larry Horwitz]

The VideoAktiv web suite is now back to normal speed, and they apparently had a problem this weekend. My downloads of the two Sony camcorders referred to by Christian each took less than 60 seconds apiece for a total of about 300 megabytes. The weekend "glacial speed" issue was only a temporary glitch, as I had previously suggested. Try it for yourself now.

As regards the validity / accuracy of the video tests which have been made and posted here. I cannot attempt to explain why one camcorder looks better than another, or why one looks "awful" as Christian reports.

The site has 122 camcorders from 17 manufacturers, (some of which are not even marketed in North America) and perhaps their testing apparatus or methods are suspect. For the 3 recent AVCHD camcorders I personally was interested in, the Canon HFS21, Sony 550, and Panny 700, they provide to me a very useful way of seeing video taken under the same conditions in both light and dark illumination, and nobody else offers such a comparison on the Internet that I am aware of.

I will leave it up to each individual to decide whether the site is reliable, unreliable, useful, useless, biased, or something in between.

It continues to impress me as an excellent and no cost and easy to use way to gather additional information, in a range of products which are not well covered elsewhere in this manner.

I will end this by asking those who do research on camcorders if they either know of any other Internet source where video comparisons can be made using repeatable test sequences, or for that matter, whether any library of over 100 camcorders can be compared in any manner, not even including test videos and test images?

Larry

[Dave Blackhurst]

OK, was able to download a little faster (still took a while, but at least wasn't a long process).

#1 - they made the EXACT same mistake as CCI - the XR and CX550 (I downloaded both, they are IDENTICAL imaging blocks, but still look slightly different in the video samples... hmmm) have a VERY wide lens - in order to zoom in to the "same" framing they WILL lose a significant amount of light - simple lens physics (ramping) in action, their iris is going to be shut down significantly when compared to the Canon and at least somewhat when compared to the Panny (which also has a fairly wide lens this year). They either have to move the camera in closer to achieve the same framing (looked to me like they used a fixed tripod), or reconsider the framing.

To prove what I'm saying, look at the last few brief frames of each clip when they zoom out, and look at the differences between the 3 framings! Apples and oranges.

I've been wondering how the new wide lens will be for my use - last night I had to mount .7X lenses on my 500 series cams to do a shoot... I'm ready for the 550 now. Yeah, I'll lose the long end, but...

#2 - look CAREFULLY at the Panasonic as they switch the light off (the Canon does this TOO) - first you see the light switch off, the image goes dark, then you can see the cameras gain up! And bring on the noise - nasty blue faeries in the case of the Canon, and generally gritty image with the Panny. NOW, look at the Sonys - when the lights drop, the image goes dark, but remains VERY clean noise wise... there's absollutely NO SIGN of gaining up - the "low lux" functions on the camera are DISABLED!! SO, in this "test" the Sony is crippled and not allowed to do what the other two cameras did! Take a look at the few frames where the Canon and Panasonic haven't yet gained up and compare... OR allow the Sony to do its magic. Fair is fair...

#3 - color... hmmm wildly different representations of the colors, the Panasonic again has a "look" about it that I'm not quite sure about - vibrant, but looks almost toylike to my eye, not what Im looking for, but I presume you could adjust it out. Liked the Canon, the Sonys looked a touch flat, but without knowing what that still life actually looks like... I've always felt that "some" manufacturers push the reds to get "vibrant" color, and it just ends up "vibrating"...


To fairly judge the Sonys in "low light", you've got to pause on the last couple of frames of each clip as each cam reaches max zoom out - now tell me which is "better" (crisper, cleaner, more detail). Imagine what happens if you let the Sony actually USE the low lux mode (which is STILL very low noise).

OK, now you have my analysis which I believe backs up my statements:

"questionable test methodology"
"they introduce a LOT of errors"
"completely inaccurate portrayals"

Any questions?

[Dave Blackhurst]

I decided to download the clips for the 500 series Sonys just to see if they had "better" low light performance because of the zoom ranges used. The differences are pretty obvious.

There is an adjustment lag when the light goes off, and you can see the 500 series camera adjust, indicating it wasn't "maxxed out", whereas the zoomed in 550 apparently was, as it can't manage to open up any more to pull in more light while zoomed in - since the sensors are the same, just different lens, I think this confirms that the camera loses significant light gathering ability when zoomed in.

I found in the description of their testing methodology the statement that they run in full auto, and turn off anything that will alter the shutter speed automatically... OK, that's fine, since Sony defaults to "off", but how exactly is that fair when the other cameras apparently ramp up gain and probably slow shutter to achieve "good" low light performance? Automatic isn't all the same from camera to camera.

I'm guessing that while the Canon and Panasonic clips represent as good as it gets in low light for those cameras (maybe switching to 30 or 24p might add a little, as this slows the shutter speed), the Sony still has a lot more available just at the press of the low lux button...

FWIW

[Chris Hurd]

Originally Posted by Larry Horwitz

I will end this by asking those who do research on camcorders if they either know of any other Internet source where video comparisons can be made using repeatable test sequences, or for that matter, whether any library of over 100 camcorders can be compared in any manner, not even including test videos and test images?

This site -- DV Info net -- has the potential to offer all of that and much more. I am happy to provide the bandwidth and the site framework, plus I can lend my solid reputation with all of the camera manufacturers in terms of arranging loans and test-drives, etc. What I don't have is the time or skill set to develop and execute the testing methodology, and I would gladly welcome any interested parties who would be willing to assist with this. I'm confident I can find a way to insure that the effort a rewarding one. DV Info Net is what you make it. Just let me know.