By Ann Ray
On February 24, 2006
Updated March 12, 2015

Six questions to ask any statistic—before you make that business decision

Apart from being such lovely sound bytes, numbers have an apparent precision, which is why we often give them more weight than they deserve. And these days, we’re all getting hit with one alarming statistic after another, so it’s a good time to dissect exactly what those two or three digit bytes actually represent.

Here are six questions to ask any statistic, whether it's one you're generating via a survey, contemplating over your morning latte, or incorporating in a marketing plan:

1. Who does this represent?

If the statistic is about the opinions of Fortune executives, it's probably not going to be useful for predicting the actions of rural farmers. While this is obvious, the problem with many statistics is that they don't tell you anything about who answered the study.

2. How many people does it represent?

Some studies, particularly ones which are very expensive to conduct, will draw conclusions from less than 100 participants. Others will have thousands of respondents. The main factor for gauging reliability is the portion of the target population (who the statistic represents) that answered. If you do come across a statistic based on very few people, it's not necessarily wrong, it's just more hypothesis than fact until further research is done to confirm it.

3. How were people reached?

So the statistic claims to represent our Fortune executives, but when you dig a little deeper you discover the data was gathered via phone and mall intercept interviews. Would you still think it represented executives? Selecting the right method for collecting answers is as much art as science, but it's an important aspect to check.

4. How were questions phrased?

We've all seen "polls" which were designed to produce a certain result. Make sure the research was done in a properly neutral manner.

5. Who commissioned the research?

Even if they have the best intentions for pure research, a special interest group has a "truth" they believe, and sometimes a bias is introduced into the questions and analysis. At the least, try to find a study from the other side for comparison.

6. When was the study conducted?

Some really old statistics are still running around. What happens at times is the statistic becomes “laundered,” and the date or source cited becomes a later quotation (often a well known organization), rather than the original study.

In practice, of course...

We're often not able to ask all these questions. However, the more holes there are in a statistic's back-story, the more risk you run if you rely on that sound byte alone as a "fact." Even your own custom survey research should always be balanced by information such as industry studies, competitive positions, experience, and financial data. And when you’re not doing research on a topic—when you’re simply taking in the news—keep in mind that for every study you see quoted, there are dozens or even thousands more conducted that you’ll never know about, some of which are likely to have more moderate or radically different conclusions.

Joel Best's Damned Lies and Statistics is excellent if you'd like to learn more about twists on research.