Recently Filed in Reporting

If you ever took a SurveyPro training class from me, you know my approach to reporting:

• Insert a simple table or chart
• Add a filter
• Try a cross-tab instead
• Copy, switch the new figure to a different chart type
• Regroup the data into different ranges
• Add a benchmark column
• Mix in another question or metric
• Add an inferential statistic like Chi Square
• Explore some more

I attended a Tableau event last week, and was delighted to see their software demonstrated in the same way.

I've been helping a client develop an assessment which they will deploy across many companies. It's relatively easy to run a survey today for one particular firm, but when you want to slice the data you'll have two years from now, it gets a bit more interesting.

You study, you brainstorm, you have endless meetings to find the best metrics—and then spend years accumulating, trending, and applying data. But while a metric can be long-lived, they're unlikely to be immortal.

If you're lucky, something dramatic happens to highlight the need for a new measure—such as when Lufthansa developed thinner seats which provided more usable space in less “seat pitch,” throwing a wrench in one of the most common cabin metrics.

But more often, it’s simply an accumulation of technical, marketplace, and fashion shifts which might have you coming up with a slightly different set of metrics—if you thought about it today.

So how long has it been since yours had a check-up?

Negative feedback is hard to deal with. Some remarks strike a tender spot, and we hear the criticism so loudly it drowns out more moderate—or even positive—responses. Other remarks contradict our reality, so we build bulwarks, dismissing comments as vindictive, erroneous, or an outlier.

So no matter how unpleasant the message, it's important to remember that all feedback is just information. What matters is how we evaluate it and what we do next.

This article isn’t about how to crunch the statistic—any stat book or Excel help can tell you how to do that. Instead, it’s my usual theme: understanding what you’re working with and making sure you’ve got something you can count on for your business decisions.

There are three possibilities when you have a theory (or better yet, your boss or client has a theory) about survey results and start reviewing it in the data:

• You were right! All is well in the universe, the sun continues to shine.
  They like your feature best, frequent buyers have higher satisfaction levels, and last year’s hybrid matrix re-org was the best thing since sliced bread.

by Joel Best

While a fascinating read for all of us, this is most applicable if you're combining secondary research with your surveys. You'll never look at "facts" the same way again.

Amazon

by Darrell Huff

There’s a reason this is still in print after 50 years, and that’s because we still fall for the same creative charting tricks.

Amazon

When we write surveys, we often have to make a choice for dollar or frequency questions. We can ask for a precise number such as:

Chi Square lets you know whether two groups have significantly different opinions, which makes it a very useful statistic for survey research. It's applied to cross-tabulations (AKA pivot tables) which are simply breakdowns like this:

This article starts with the theory, and then has guidelines for using the statistic:

• Understanding the calculations
• Calculating Chi Square in real life
• Applying Chi Square to surveys

"What do you mean I have to clean the data? I used the Web so I wouldn't have to do any data entry."

While Web and scannable surveys will minimize data entry costs, some degree of data cleaning is required for all projects. Data cleaning has two elements: checking that the forms submitted are of good quality, and coding typed/written responses so they can be analyzed most effectively.

As with all aspects of a survey, you may need to make some trade-offs on your projects. I recently spoke to someone who was analyzing 20,000 responses per week on one survey, so optimizing all the comments she received wasn't feasible. However, if you have a few hundred responses to a one-time or quarterly study, it will probably make sense to invest a bit more on a per-form basis. Likewise with high stakes surveys, such as employee feedback, you'll want to make sure you get the most from the data.

If you pick up a classic market research text you'll come away with the impression that sampling is an integral part of every survey. It's not.

The reasons sampling has historically been such a big deal are:

• High per-respondent cost
• Mass market issues
• The sampling error calculation

I was on GameSpot recently and under the user reviews found these three pie charts:

In our sound byte culture we love condensed statistics, magic values which will let us know how many customers will buy again or whether employees are engaged in their work—all at an easily compared glance. While summary information such as means (averages) is very useful, it can also obscure some critical details and trends within your data. When many factors are condensed into a single index value for executive dashboards, even more can be hidden.

Apart from being such lovely sound bytes, numbers have an apparent precision, which is why we often give them more weight than they deserve. And these days, we’re all getting hit with one alarming statistic after another, so it’s a good time to dissect exactly what those two or three digit bytes actually represent.

Here are six questions to ask any statistic, whether it's one you're generating via a survey, contemplating over your morning latte, or incorporating in a marketing plan: