Consistency is never foolish in a survey
"A foolish consistency is the hobgoblin of little minds, adored by little statesmen and philosophers and divines. With consistency a great soul has simply nothing to do."
Apart from the common misquote which drops "foolish," most people are unaware of how Ralph Waldo Emerson closed that paragraph:
"To be great is to be misunderstood."
And the one thing we don't want in a questionnaire is to be misunderstood!
A few ways surveys can be more or less consistent:
- Minimizing the number of answer scales you use (but not so much it becomes awkward) and staying with "common" answer labels.
- Making sure your scales don't flip like the example below did between questions 11 and 19. In extreme cases like this, where the scale not only flipped but that information was buried in the instruction paragraph, many respondents are likely to have answered it oppositely and the entire grid's results would need to be discarded.
- Smoothing out the number of questions per page (see article).
- If randomizing/shuffling elements, making sure your technology won't re-order items when the respondent uses forward/back buttons or pause/resume—the survey should appear static to them.
- When applicable, organizing repeated themes in a way that make it easy for respondents to identify these elements and answer more quickly the second and third time. For example, if you're evaluating internal service in an organization, you may have a core set of ratings for many departments. If you place these common questions at the beginning of the list, it's easier for respondents even if these questions "belong" in the topical sub-sections.
On the positive side, consistency reduces respondent effort, which means fewer errors and greater likelihood of completion, all things we like as they skim past instructions in their busy days.
I always hear your voice in my head ‘if it is taking you too long, there is a simpler way to do it!’
Market Research Analyst
Alberta Motor Association