To pop-up or not?

If you want a pop-up, go ahead and use one—just back it up with a “Take a survey” or “Feedback” button within the site itself. Apart from being a fall-back from blockers, the button also allows visitors to change their mind and provide feedback later. A few things which will help usability and therefore your response rate.

Tell them the survey purpose

On the pop-up, let visitors know whether the survey is feedback on the site, input on new products, service ratings, etc., how long the survey will take, and any incentives. This can be done in very few words such as:

Take 5 minutes to give us feedback on our site and enter to win an iPod!

Too often these invitations are generic “Take a survey” requests which ask the respondent to blindly contribute an unknown amount of time to a site. For the in-site button, use an interim page to convey this information rather than putting it on the first page of the survey where pop-up respondents would see it a second time.

Let them pick the timing

Unless the survey is relevant only to the page they're currently on (in which case it should be embedded feedback as you see in knowledgebases), give them options on the pop-up such as:

• Take survey now
• Ask me when I leave the site
• No thanks

Too often "Evaluate our site" invitations will pop-up when someone first hits a home page with no option to defer feedback. Not only does it strike an odd note with the respondent, it also degrades your data quality.

Don't interrupt their flow in the site

When they click on the pop-up invite or in-site button, it may be more natural for the survey opens in a separate window with clear “close window” instructions so when they finish, they can return to what they were doing.

Remember they answered

Save a cookie on their system with the survey ID and whether they declined or completed. If you’re rotating surveys and have repeat site visitors, add a time stamp so visitors don’t get a pop-up for a different survey for at least 90 days.

Watching the sample quality

Make sure you’re collecting and periodically reviewing stats on:

• % of visitors seeing automatic pop-ups (depending on your site design you may already have this)
• What option they click in the pop-up
• % of surveys completed from the in-site button vs. pop-up
• % of in-site button clickers who stop after the time/purpose information page

Collecting these stats can be done with simple HTML tricks and standard server logs—you don’t need custom dev, just advance planning.

While you may be tempted to look for the holy grail of reliable browser stats, there’s simply not a solid source out there. If you search, you’ll find as many links saying the stats are bad as giving breakdowns, so your best source remains your own site’s logs. When looking at your particular site visitors, remember that there may be significant differences in how casual visitors interact with the site versus customers with accounts.