Six questions to ask any statistic—before you make that business decision

1. Who does this represent?

If the statistic is about the opinions of Fortune executives, it's probably not going to be useful for predicting the actions of rural farmers. While this is obvious, the problem with many statistics is that they don't tell you anything about who answered the study.

2. How many people does it represent?

Some studies, particularly ones which are very expensive to conduct, will draw conclusions from less than 100 participants. Others will have thousands of respondents. The main factor for gauging reliability is the portion of the target population (who the statistic represents) that answered. If you do come across a statistic based on very few people, it's not necessarily wrong, it's just more hypothesis than fact until further research is done to confirm it.

3. How were people reached?

So the statistic claims to represent our Fortune executives, but when you dig a little deeper you discover the data was gathered via phone and mall intercept interviews. Would you still think it represented executives? Selecting the right method for collecting answers is as much art as science, but it's an important aspect to check.

4. How were questions phrased?

We've all seen "polls" which were designed to produce a certain result. Make sure the research was done in a properly neutral manner.

5. Who commissioned the research?

Even if they have the best intentions for pure research, a special interest group has a "truth" they believe, and sometimes a bias is introduced into the questions and analysis. At the least, try to find a study from the other side for comparison.

6. When was the study conducted?

Some really old statistics are still running around. What happens at times is the statistic becomes “laundered,” and the date or source cited becomes a later quotation (often a well known organization), rather than the original study.

In practice, of course...

We're often not able to ask all these questions. However, the more holes there are in a statistic's back-story, the more risk you run if you rely on that sound byte alone as a "fact." Even your own custom survey research should always be balanced by information such as industry studies, competitive positions, experience, and financial data. And when you’re not doing research on a topic—when you’re simply taking in the news—keep in mind that for every study you see quoted, there are dozens or even thousands more conducted that you’ll never know about, some of which are likely to have more moderate or radically different conclusions.

Joel Best's Damned Lies and Statistics is excellent if you'd like to learn more about twists on research.