By Ann Ray
On January 20, 2009
Updated October 9, 2012

Rule #1: Surveys are about information, not justification

There are three possibilities when you have a theory (or better yet, your boss or client has a theory) about survey results and start reviewing it in the data:

You were right! All is well in the universe, the sun continues to shine.
They like your feature best, frequent buyers have higher satisfaction levels, and last year’s hybrid matrix re-org was the best thing since sliced bread.

Chaos reigns! The opposite of what you expected is showing up in the data.
Your feature is unpopular, frequent buyers have more complaints than those who rarely contact the company, and employees are reporting lower productivity since the re-org?

Crickets. Nothing seems to be happening.
Everything’s just, well, Good, and you can’t find much in the way of differences between customer segments or before and after the re-org. Huh.

We all love to be right, but the important thing is to remember that a survey is all about getting information. Because if we’re not open to getting accurate information—whatever it may be—we’re just putting blinders on and setting ourselves and others up to stumble around the business landscape.

This is setting aside the minority who make ethical choices to slant "research" through loaded questionnaires or outright falsification of data. Instead it's a common sense reminder to everyone who's trying to juggle their organization's performance metrics, group goals, survey data, personal theories, and any other elements, of what's important to the organization at the end of the day. Also, remember that survey data is always only part of your decision process, integrated with secondary research and the knowledge of key stakeholders—never taken by itself as Sole Truth, no matter how good your sample and methodology.

Happy Surveying!

Filed under:
Management
Reporting

Tagged with:
Ann's Rules
Biased research
Ethics

1 Comment

Note: New comments disabled for a few days while debugging.

Vineedh Mathew | September 28, 2012 11:00 AM | Reply

Hi
I want to do a survey to understand the correlation between performance of large projects (>50 person months effort) and factors influencing them. I am not sure how to proceed to do the analysis statistically. Which of your articles would provide me with this information...

Thanks
Vineedh Mathew