Spiral notebooks

Are your product development surveys all about new bells and whistles? That’s critical information—and fun—but it’s never the whole picture.

On the less fun side is asking what irritates your customers. This is also more expensive to research, both because collecting the information is more involved (verbatims, interviews, forum mining), and because digging through both measured and SHOUTED complaints about our products is exhausting.

I recently completed a detailed customer survey for a software program I subscribe to. It was a conjoint analysis, which presents different feature profiles and price points to the respondent, asking them to pick the one they prefer.

What they will know from my responses? I’m cheap.

What they won’t know? Why.

Anyone else feel like their head is spinning this election? Here's some info and resources to evaluate all the headlines.

Rocket Surgery Made Easy

Steve Krug’s “Rocket Surgery Made Easy” is an engaging, pragmatic, and quick read from an experienced expert in Web usability. If you’re considering qualitative studies, or want to do them but are stopped by budget constraints, this is a great introduction to the basics of user testing.

I only have one thing I’d add to his text. It’s easy to focus on finding all the problems—and only on finding problems. When doing any kind of testing or editing, I also try to pay attention to what is working, because:

  • It balances out “Everything is horrible!” impressions
  • You can use what’s working as a model for similar areas
  • If we don’t note what ain’t broke, it’s easy for it to be casually changed by someone’s preference

Krug gets at this a bit via his mandate “When fixing problems, try to do the least you can do”—I just like to give it a smidge more focus.

As always, I’m a fan of my local library, but you can also buy it on Amazon.

Happy testing!

Analysis by Layers - Practical Surveys

| No Comments

If you ever took a SurveyPro training class from me, you know my approach to reporting:

  • Insert a simple table or chart
  • Add a filter
  • Try a cross-tab instead
  • Copy, switch the new figure to a different chart type
  • Regroup the data into different ranges
  • Add a benchmark column
  • Mix in another question or metric
  • Add an inferential statistic like Chi Square
  • Explore some more

I attended a Tableau event last week, and was delighted to see their software demonstrated in the same way.

Using Skips in Surveys - Practical Surveys

| No Comments

One of the best ways to shorten a complex survey is to use skips, also sometimes called branching. This makes it a more relevant experience for respondents, as well as less effort—both factors which can impact your completion rate.

Agile Market Research - Practical Surveys

| No Comments

A couple months ago, I saw a presentation about agile research by Zach Simmons of Discuss.IO. He comes at it from the qualitative side, with a great platform for on-demand remote interviews, but it reminded me of some survey ideas I’d been mulling.

Most market research mirrors classic product development (waterfall model), with a steady progression through needs analysis, research & development, and delivery. It assumes big releases which stay stable for an extended period, which is a good fit to physical projects like manufacturing and construction.

Agile development works in smaller chunks, with products continuously evolving. The idea is to release a good start and keep improving it, rather than a giant push to create the end all be all specification. You may have noticed many of your smart phone apps update frequently, sometimes as often as every two weeks—this is a reflection of agile methodology.

I've been helping a client develop an assessment which they will deploy across many companies. It's relatively easy to run a survey today for one particular firm, but when you want to slice the data you'll have two years from now, it gets a bit more interesting.

Note: This article assumes you're familiar with common web technologies, so if client-side vs. server-side is a bit of gibberish, start here.

We can never know exactly what environment a respondent will be using to complete our web surveys, so how do we add bells and whistles without the forms breaking? (Seriously, never. I've had IT people give me rigid specs on their employees' systems, only to have the server logs show other configurations.)

There's a range of tools for processing the surveys on your web site, and while you may not be writing code, understanding the landscape can be helpful. Forgive me going back to some basics, but people tend to have different holes in their knowledge, so I'm trying to cover all the bases.

When you access a web survey or other page, you use a browser such as Internet Explorer, Safari, or Firefox. This downloads a collection of HTML, CSS, and scripting files, which are interpreted by your desktop or phone and browser for display. The combination of device + browser + personal settings (installed plugins, large fonts, security) results in a huge range of possible environments in which a survey needs to function. Which is why widow/orphan line breaks are the last thing we worry about.

Usually when I’m asked about random or shuffled questions and values, it’s because someone came across the function as The Right Way to research, or spotted it in a software feature list. Rarely does it pop up because of a serious problem with order biases. My general advice is to first look at all the other tweaks you can make to your survey, from sampling to usability to scale phrasing to analysis—then decide if random makes your survey’s must have list.

For when it is an issue—or anyone thinking “Why not use it if I have it?”—here we go...

You study, you brainstorm, you have endless meetings to find the best metrics—and then spend years accumulating, trending, and applying data. But while a metric can be long-lived, they're unlikely to be immortal.

If you're lucky, something dramatic happens to highlight the need for a new measure—such as when Lufthansa developed thinner seats which provided more usable space in less “seat pitch,” throwing a wrench in one of the most common cabin metrics.

But more often, it’s simply an accumulation of technical, marketplace, and fashion shifts which might have you coming up with a slightly different set of metrics—if you thought about it today.

So how long has it been since yours had a check-up?

Attacks vs. Feedback - Practical Surveys

| No Comments

Negative feedback is hard to deal with. Some remarks strike a tender spot, and we hear the criticism so loudly it drowns out more moderate—or even positive—responses. Other remarks contradict our reality, so we build bulwarks, dismissing comments as vindictive, erroneous, or an outlier.

So no matter how unpleasant the message, it's important to remember that all feedback is just information. What matters is how we evaluate it and what we do next.

Respondents have to say “Yes” again and again before they finish, while a single “No” can send them out the door. And this doesn’t even take into account outside factors, like technical problems or interruptions.

Survey Respondent Flowchart

Someone on the Apian discussion list had a question about the SurveyPro 5 Answer Summaries feature, which is essentially:

  • You marked these responses, are you sure?
  • Print your replies for your records.

This got me to playing around with it a bit, and I ended up with a largely self-explanatory—because it's annotated—play survey here. (At least for a while, and if I eventually bump it for something else I'll update this post with a PDF or other details :-)

And for your experimenting fun, here's a zip of the SP5 file. For those who don't have SurveyPro 5, you can achieve a similar effect with Answer Piping, which is covered in the Dynamics chapter of your NetCollect User Guide.

GreenGeek Survey
http://www.maark.com/greensurvey/

Normally I don't promote my clients' surveys, especially not bugging my own network, but this one is kind of like asking "What do you want for your birthday?" It's all about an upcoming eco idea contest, and asking what prizes to offer--rather than the sponsor just assuming what participants will want most.

So if you know any college-ish geek types (your own spawn, students, friends of friends, whatever :-) who might be interested in an environmental idea competition, then please pass the word, either directly or just down the six degrees. The survey's only online through the beginning of next week, but it's legit and even offers a fair chance of getting an Amazon certificate for their time.

As much as I'm a general advocate for respondents, I know it can be a challenge when they're an amorphous group and your manager or client is an immediate voice. But, there are two very pragmatic reasons for putting yourself in the respondent's shoes long enough to make sure your survey will be a good fit.

Because we're not binary beings, we have text domain names such as:
   practicalsurveys.com
to reach Websites, but what really directs traffic around the Internet are the corresponding numeric identities, IP (Internet Protocol) addresses:
   209.197.67.60

In addition entering IP addresses as destinations, whenever you (or your e-mails) travel the Web you're also leaving little IP address footprints wherever you go. Because of this, survey managers sometimes want to use the respondent's IP address in one of two ways:

  1. To prevent ballot box stuffing by only allowing one response per IP address
  2. As a supplemental source of information

Before you join them, let's look a bit more about exactly what the respondent's IP may or may not tell you.

Recently I decided it was.

People generally agree investing in usability for Websites and Web applications is a “good idea” when it comes to retaining visitors and users—or in my specialty, survey respondents. The challenge is assigning a value to that investment because product managers, graphic designers, user interface specialists, technical support managers, trainers, and executives will often have wildly differing opinions.

This article isn’t about how to crunch the statistic—any stat book or Excel help can tell you how to do that. Instead, it’s my usual theme: understanding what you’re working with and making sure you’ve got something you can count on for your business decisions.

There are three possibilities when you have a theory (or better yet, your boss or client has a theory) about survey results and start reviewing it in the data:

  • You were right! All is well in the universe, the sun continues to shine.
    They like your feature best, frequent buyers have higher satisfaction levels, and last year’s hybrid matrix re-org was the best thing since sliced bread.

"A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines. With consistency a great soul has simply nothing to do."

Apart from the common misquote which drops "foolish," most people are unaware of how Ralph Waldo Emerson closed that paragraph:

"To be great is to be misunderstood."

Cover image by Joel Best

While a fascinating read for all of us, this is most applicable if you're combining secondary research with your surveys. You'll never look at "facts" the same way again.

Amazon

Cover imageby Darrell Huff

There’s a reason this is still in print after 50 years, and that’s because we still fall for the same creative charting tricks.

Amazon

When you've exhausted intangible payoffs, it's time to reach for the payola. Sometimes you'll get lucky and you can find an incentive that's cheap for you to provide—such as a product upgrade or free month's service—and sometimes it will be a straight cash deal.

Cover image by Howard Schuman & Stanley Presser

Despite being in a quantitative industry, surveyors rarely conduct tests to measure what happens when we rearrange questions, add a neutral point in a scale, or make other adjustments. If you're ready to absorb some more advanced issues, complete with footnotes, this is a great book to pick up. Note that the 1996 copyright is simply a reprint of the 1981 text.

Amazon

Cover image by Harry Beckwith

When you want to broaden your perspective, this will help you understand how customer and employee satisfaction mixes with and reinforces other marketing efforts. While the author focuses on services, it's useful in any industry—we're all competing on intangibles these days.

Amazon

Recently I completed a telephone survey, and in the course of the 22 minute conversation (estimated at 12-15) I was asked:

Would you recommend a friend or family member attend University of California Davis?

Recommend for what? I'm aware of the school's solid reputation in engineering and veterinary medicine, but have no notion where their other programs rank.

Recommend for whom? I have to think of an individual as to whether the programs, lifestyle, location and tuition (in-state resident vs. full) would be a fit.

Most everyone wants to measure (and improve) customer satisfaction, but how?

First and foremost, if your organization is new to surveys and doing this in-house, start simple! The goal of any survey is better information for decision-making, and a modest quantity of information that you actually use is far more valuable than a complex picture that may be flawed or too troublesome to maintain.

A common concern is how to deal with duplicated survey responses. In practice, this is an issue when the benefit to multiple submits outweighs the effort of making them. For most surveys, the challenge is getting people to complete once.

A client working on a project for a non-profit recently sent me two questionnaires:

Version 1
Written by my client, and primarily driven by their contact, the CEO. The survey focused on evaluating the organization as a whole, though a significant emphasis was on communication and fundraising.
Version 2
Written by one of the non-profit's board members. The questionnaire was far more granular, focusing on specific programs offered.

One of my most popular articles via Web searches is about typical response rates. What many researchers forget to look at is the abandonment or completion rate. If you extend an invitation to someone for the survey (mail, e-mail, banner ad, phone, dancing monkey, etc.) and they begin the questionnaire, do they finish it?

One of the great things about the Web is that almost any functionality is possible—it's just a small matter of programming (and budget and time and compromises). Sometimes you can imagine a widget which will make your respondent's or visitor's experience smoother or richer. Sometimes it's a function which will make your site easier to manage.

For Web surveys, there's a huge range of tools and services, so someone may already offer your dream feature. However, there are times when you just need something custom.

I recently had lunch with a fellow consultant who focuses on employee surveys. Contrary to what you might expect in the grand scheme of employer/staff relations, he spends much of his time getting executives to pay attention to "small" problems. In one case, it was a departmental laser printer long overdue for replacement—very like the machine which met a violent end in Office Space.

As great as the Web is for surveys, there are times paper is a better fit (and phones and in-person too of course). When paper is what you need, people often look to scanning as a "no data entry" solution. While you can get close with some surveys, as with all technology there's fine print.

These days, most of us do more in Excel than the VisiCalc creators would have dreamed. Following are a few of my favorite tricks for working with data in Excel (the raw side, not the chart side). This article includes:

Comma Separated Values (AKA comma-delimited ASCII) are one of the most common formats used to exchange information. This is because they're compatible with applications on just about any operating system—going back to DOS days. On many computers, a .CSV file will be automatically associated with Microsoft Excel, and when you double-click on the file it opens using the automatic settings.

This is great—unless you have certain types of data such as East coast Zip codes.

Some surveys have very infrequent responses, going days or even weeks without a submission. Rather than heading out to your Web host to keep checking for responses, it can be handy to just get an e-mail alert when someone completes the survey.

A few premium services such as SurveyHost and KeySurvey provide this feature, including sending e-mails with an individual's answers and only under certain conditions (such as marking "Contact me"). However, low volume surveys often don't justify that type of budget, so here's another way.

Most survey products and services let you specify a Thanks URL for when respondents finish. By sending respondents to this script, you can get an e-mail alert upon each completion. It's written in PHP, so you can use it with both Unix/Linux and Windows Web servers.

Download the Zip file with the script and instructions.

"Piping" at its most generic refers to moving information in, out, and around a survey. This makes it a very useful technology for any dynamic survey (Web, telephone, kiosk), potentially enriching the respondent experience or increasing data quality.

It's also one of the many survey terms that can mean different things to different people, so before you assume your survey tool does what you want, make sure you're using the same definitions as your vendor. This article covers four main approaches to piping:

Just because you have a Web server doesn't mean it's the best choice for running your Web surveys. While your own box has the advantage of full control, it also has some drawbacks. This doesn't mean you have to go with an Application Service Provider (ASP) survey vendor for your projects. Many Web survey software vendors also provide "self-service" hosting for their software users.

When you're conducting market research, one of the challenges is locating potential respondents. You have a few options for reaching beyond your customers:

  • Banner ads
  • Viral spreading of the link
  • Postal mail invitations
  • Survey panels

Note that some of these approaches don't generate a random sample, and you may need to provide an incentive for respondents.

Answer every one you can.

When I spot a "Take our survey" link on a Web site or feedback URL on a store receipt, I generally answer the questionnaire. Sometimes it's a smooth, well orchestrated experience. Occasionally I find myself wondering "What were they thinking?"

Just as an editor reading a manuscript watches for grammar, clarity, flow, and other issues, there are a number of aspects you'll want to watch for when taking a survey. While some of the issues I list here are an abstract analysis of the survey, the most important thing to pay attention to is your personal reactions to the questionnaire. After all, you are a respondent, and what you feel while completing the survey will be happening for other people as well. When you get started you'll only notice a few issues with each questionnaire, but over time the evaluation will become second nature—just like an advertising executive dissecting Super Bowl commercials.

When we write surveys, we often have to make a choice for dollar or frequency questions. We can ask for a precise number such as:

How many times have you visited any of our stores in the past twelve months prior to your most recent visit? (Please enter a number from 0-365)

Chi Square lets you know whether two groups have significantly different opinions, which makes it a very useful statistic for survey research. It's applied to cross-tabulations (AKA pivot tables) which are simply breakdowns like this:

  Yes No Total
Female 45 5 50
Male 15 35 50
Total 60 40 100

This article starts with the theory, and then has guidelines for using the statistic:

In a recent workshop, a participant asked how to deal with his Zip code data. Every time he did a survey, he was manually grouping the codes into regions for analysis. If you're in a similar situation, here's a better approach.

First, you need to be able to merge two pieces of data: your survey responses and the Zip code information. This means you'll need to add some extra fields to your questionnaire to handle the city/state/CBSA information. In a Web form, you can use hidden fields to stash the information. On a paper form you can add them to your survey software or database after you print the questionnaire.

It's become accepted knowledge that people talk more about bad service experiences than good ones. That extra energy people put into spreading the word was bad enough when they talked to friends and family. Now that energy is going elsewhere, and seriously bad experiences are no longer just someone complaining, they're now news and entertainment thanks to technologies like blogging and YouTube. In other words, they're mass media bad PR.

A common design for surveys is to make them like a PowerPoint slide show, with one question on each page. In practice, what this does is make the survey longer for respondents, which is a good way to irritate them and increase your abandonment rate. For illustration, let's consider a modest size survey:

  • Five individual questions
  • Two Five question grids
  • Five demographics

I love my new little laptop, so when the manufacturer popped up a survey invitation today (time delayed 3 months from purchase) I was happy to provide feedback. The survey was primarily about how I'd been using the system, which should have been easy questions to answer. However, as I went through the survey many of the questions weren't easy for me, because 9 of the 19 questions were missing my preferred answer.

Two representative questions are:

Which best describes your game-playing habits?
     Playing online PC games
     I do not play games

Which of the following music-related activities have you performed on this PC? (Select all that apply)
     Transferred music to a portable player device
     Purchased and downloaded music from an online Service
     Streaming music through the Internet (Internet Radio)
     Created a music CD
     None of the above

In addition to my work on surveys, I'm also Chapter Leader of DigitalEve Seattle"http://digitaleveseattle.org/, an association which supports women in technology.

We have an e-mail discussion list, and it's been quite lively the past few days talking about gender assumptions in the workplace. This was started when someone asked how to deal with a job applicant who became defensive when she suggested "Gentlemen" may not be the best salutation in this day and age.

On a similar note, I have a recommendation for anyone conducting telephone or in-person interviews. Train your interviewers to address women as "Ms." instead of "Mrs." unless your list explicitly identifies the marital status of the person being called. You may also spot other places in your scripts or through monitoring where assumptions are being made about your respondents.

After all, why open on an awkward note if you don't need to?

"What do you mean I have to clean the data? I used the Web so I wouldn't have to do any data entry."

While Web and scannable surveys will minimize data entry costs, some degree of data cleaning is required for all projects. Data cleaning has two elements: checking that the forms submitted are of good quality, and coding typed/written responses so they can be analyzed most effectively.

As with all aspects of a survey, you may need to make some trade-offs on your projects. I recently spoke to someone who was analyzing 20,000 responses per week on one survey, so optimizing all the comments she received wasn't feasible. However, if you have a few hundred responses to a one-time or quarterly study, it will probably make sense to invest a bit more on a per-form basis. Likewise with high stakes surveys, such as employee feedback, you'll want to make sure you get the most from the data.

Recent Comments

  • Ann Ray: Thank you so much for the link! I knew there read more
  • WK: Great information! Your explanations are fabulous. Interactive Chi Square table: read more
  • Reginald Hedinger: Thanks, I've recently been searching for information about this subject read more
  • vps server host: so much wonderful information on here, : D. read more
  • best vps hosting: Thankyou for this post, I am a big big read more
  • Ann Ray: Thanks! Need to get back to the keyboard for my read more
  • blackhat seo forum: Wondering if you maybe wanted to start writing my blog read more
  • Ann Ray: Thanks! I'll get the ebook out one of these days, read more
  • black hat: I intended to draft you the little bit of remark read more
  • Vineedh Mathew: Hi I want to do a survey to understand the read more