Beta Test Protocols at DVinfo.net

Carroll Lam · November 29th, 2007, 11:08 AM

It is obvious that Sony chose to include us early adopters of the EX1 in the beta testing program for the camera. I have no particular problem with that (assuming that currently identified problems are fixed in a timely manner) but it would have been nice to have had some "release notes" included with the first deliveries indicating the known problems. The vignetting problem was obviously identified by Adam Wilts so we know at least that problem was known. I suspect that some of the other reviewers of "engineering prototypes" found other problems that are starting to pop up here like the "cannot proceed" warning for some functions that can only be removed by a reset of all parameters.

So...given that we are (in)voluntary participants in such a program we should attempt to make the most use of our time with the camera under the assumption, of course, that someone with "clout" from Sony follows the the postings here and makes sure key information gets back to the developers. Sony is famous for their "inscrutability" regarding what they may or may not be doing with respect to fixes on publicly-identified problems. One can only hope that useful information generated by us will be used by Sony to improve the product.

In some prior incarnations of my life I was involved in the development, testing, and deployment of complex software systems. I would offer the following tips/guidelines for your testing of the camera to provide maximum-useful feedback to Sony:

1. Repeatability. Unless a software problem can be repeated with known conditions, it will not be possible to correct. If you encounter a problem make every attempt to determine the conditions that caused it and will cause it again when those conditions are met.

2. Documentation. Keep good notes during your testing. This helps with #1 preceding and assists in deciphering trends or unusual conditions.

4. Planning. Plan your testing so you know what particular aspect of functionality you are testing. Although random tests may show up unknown problems the isolation of the problem to given conditions requires some thought and planning.

5. Focus. Don't try to test every condition possible on the camera. Focus on the functionality you expect to use most. There are enough variety of users that most of not all features of the camera will be covered.

6. Report. When posting a found problem include the conditions you have found to cause the problem. Early reporting of a problem without being able to report the specific conditions causing it is fine. Just keep focusing on that problem until you can.

I think the EX1 is already a great camera. It just needs some of the current problems fixed.

Sony, are you listening?

Carroll Lam

Walter Hicks · November 29th, 2007, 11:14 AM

I don't think they will listen to #3...

Craig Seeman · November 29th, 2007, 11:23 AM

See my comments here
http://dvinfo.net/conf/showthread.php?t=108433&page=13

I also have experience with QA relating to video.

Carroll, I do hope Sony pays attention to us level heads. We are the customers who had enough confidence in Sony to be early adopters. I think technology companies (wise ones at least) know the first wave of adopters can be their best (or worst) word of mouth marketers depending on how things go.

I think we should work with Chris Hurd and any contact he has with Sony to work out a procedure that is most helpful to all involved.

Ray Bell · November 29th, 2007, 11:34 AM

This thread should become a sticky...

During the documentation and reporting process all of the issues should be brought up to the first post in the thread...

This way you can identify the issue, communicate the setup creating the issue and be able to replicate the issue in a organized fashion that
will serve two purposes...

The first purpose will be that anyone getting a new camera will have a list of tests to perform.

The second purpose is to help Sony have a list of issues, how to duplicate the issue for potential ramification...

Dave Elston · November 29th, 2007, 11:40 AM

Quote:

Originally Posted by Walter Hicks

I don't think they will listen to #3...

I think Carroll must be suffering from what is commonly known as "King Arthur syndrome".... ;0)

http://www.youtube.com/watch?v=xOrgLj9lOwk

;0)

Seriously though, Sony should really put their hands up on this and offer some formal statement either through the EX product pages or via the official resellers.
Which is going to hurt sales more; internet chatter getting out of hand or a clear admittence and plan of action from the 'horses mouth' ..?

Even if this is just a first-run batch showing teething trouble, Sony need to communicate this. For this reason I have decided to wait a little longer before placing any pre-order, not that I can wait forever (!)

November 29th, 2007, 11:08 AM	#1
Carroll Lam Contributor Join Date: Apr 2005 Location: Tucson, AZ Posts: 128	Beta Test Protocols It is obvious that Sony chose to include us early adopters of the EX1 in the beta testing program for the camera. I have no particular problem with that (assuming that currently identified problems are fixed in a timely manner) but it would have been nice to have had some "release notes" included with the first deliveries indicating the known problems. The vignetting problem was obviously identified by Adam Wilts so we know at least that problem was known. I suspect that some of the other reviewers of "engineering prototypes" found other problems that are starting to pop up here like the "cannot proceed" warning for some functions that can only be removed by a reset of all parameters. So...given that we are (in)voluntary participants in such a program we should attempt to make the most use of our time with the camera under the assumption, of course, that someone with "clout" from Sony follows the the postings here and makes sure key information gets back to the developers. Sony is famous for their "inscrutability" regarding what they may or may not be doing with respect to fixes on publicly-identified problems. One can only hope that useful information generated by us will be used by Sony to improve the product. In some prior incarnations of my life I was involved in the development, testing, and deployment of complex software systems. I would offer the following tips/guidelines for your testing of the camera to provide maximum-useful feedback to Sony: 1. Repeatability. Unless a software problem can be repeated with known conditions, it will not be possible to correct. If you encounter a problem make every attempt to determine the conditions that caused it and will cause it again when those conditions are met. 2. Documentation. Keep good notes during your testing. This helps with #1 preceding and assists in deciphering trends or unusual conditions. 4. Planning. Plan your testing so you know what particular aspect of functionality you are testing. Although random tests may show up unknown problems the isolation of the problem to given conditions requires some thought and planning. 5. Focus. Don't try to test every condition possible on the camera. Focus on the functionality you expect to use most. There are enough variety of users that most of not all features of the camera will be covered. 6. Report. When posting a found problem include the conditions you have found to cause the problem. Early reporting of a problem without being able to report the specific conditions causing it is fine. Just keep focusing on that problem until you can. I think the EX1 is already a great camera. It just needs some of the current problems fixed. Sony, are you listening? Carroll Lam

November 29th, 2007, 11:14 AM	#2
Walter Hicks New Boot Join Date: Jun 2006 Location: Rydal PA Posts: 8	I don't think they will listen to #3...

November 29th, 2007, 11:23 AM	#3
Craig Seeman Inner Circle Join Date: Feb 2004 Location: Brooklyn, NY, USA Posts: 3,841	See my comments here http://dvinfo.net/conf/showthread.php?t=108433&page=13 I also have experience with QA relating to video. Carroll, I do hope Sony pays attention to us level heads. We are the customers who had enough confidence in Sony to be early adopters. I think technology companies (wise ones at least) know the first wave of adopters can be their best (or worst) word of mouth marketers depending on how things go. I think we should work with Chris Hurd and any contact he has with Sony to work out a procedure that is most helpful to all involved.

November 29th, 2007, 11:34 AM	#4
Ray Bell Regular Crew Join Date: Nov 2004 Posts: 1,414	This thread should become a sticky... During the documentation and reporting process all of the issues should be brought up to the first post in the thread... This way you can identify the issue, communicate the setup creating the issue and be able to replicate the issue in a organized fashion that will serve two purposes... The first purpose will be that anyone getting a new camera will have a list of tests to perform. The second purpose is to help Sony have a list of issues, how to duplicate the issue for potential ramification...