Real world sampling
If you pick up a classic market research text you'll come away with the impression that sampling is an integral part of every survey. It's not.
The reasons sampling has historically been such a big deal are:
- High per-respondent cost
- Mass market issues
- The sampling error calculation
Let's take them in order.
If you're offering an incentive for every completion or using interviewers, this is a significant cost issue and sampling makes sense.
If you're using Web or paper surveys, the per-respondent cost is much smaller. While some Web survey hosting services have a per-response model, others are based on a monthly subscription. You can also get software to run surveys on your own servers, some of which allows unlimited responses.
Remember that even if you have no data entry costs because you're scanning or using the Web, you'll always have a data cleaning cost for any written answers.
Mass market surveys
If you're predicting cola market shares, you don't need a census to find the answer. With a well constructed survey and representative sample, you'll be 95% certain you're within +/- 2% at just a couple thousand responses.
However, many surveys deal with smaller populations where it is feasible to reach out to the entire group. Remember you'll still end up with a "net sample" for any survey where you don't get every single person in the population to answer. You just don't have to bother with sampling who you invite for smaller groups.
This can become a red herring in your survey projects. It's easy to fixate on the sampling error because it will give you a tidy +/- % of accuracy. The problem is this error only covers the distortion from having a portion of population answer. It does not include errors introduced by poorly phrased questions, missing scale options, non-random sampling techniques, etc.
How to sample
If you do have a large population or significant per-respondent costs, here are some basics on how to sample. I also recommend doing a bit more reading.
The end goal is always a representative sample of respondents. If you're measuring call center satisfaction, intercepting callers for a couple weeks will do the trick. However if you want to measure overall customer satisfaction, you also need to be reaching out to people who didn't call recently.
You always want to select people randomly, so you want an automatic selection of every 5th, 20th, etc. caller rather than having the service representative select who they offer the survey.
Here's a table for picking the size of your net sample (responses, not invitations) based on whether you want to be 95% or 99% certain of your results and whether you're comfortable with a +/- 5% or +/- 2% margin of error. If you have a very diverse population or plan to break the data down into smaller sub-groups, you'll want to get more responses.
|Population||95% Confidence||99% Confidence|
|+/- 5%||+/- 2%||+/- 5%||+/- 2%|
When never to sample
Do not sample employees! You can decide to run a survey for only one division, but do not sample within that division. Even if an employee will not bother to answer the survey, they still want to be asked.
Likewise, with employees be sure you've got all the stakeholders on an issue. If you're talking about internal management, you can just poll the R&D division. But if you're talking about new product development, it's a good idea to include Manufacturing, Marketing and Sales.
I always hear your voice in my head ‘if it is taking you too long, there is a simpler way to do it!’
Market Research Analyst
Alberta Motor Association