The following article appeared in
Strategic Communication Management, April/May 2003
LLP Limited,

.

Get The Credit You Deserve
From Surveys

By Angela D. Sinickas, ABC

The wonders of technology have opened up easy-to-use on-line survey creation and analysis. Yet if you take the numbers the surveys provide at face value, you may be under-representing your audience's true responses. The following examples demonstrate how to phrase questions for more accurate results.

In the last few years, quite a few on-line resources have become available on the Internet to help communicators conduct surveys, such as SurveyMonkey, Zoomerang, PollCat and at least a dozen others. Some of them are free of charge, some charge a small monthly fee, and others charge users for advanced survey analysis or for large numbers of respondents.

However, with the convenience and low prices come some hidden hazards. The way the surveys calculate your results depends on how you ask the questions. Communicators should be wary of how they interpret what the numbers are reporting.

To illustrate some of the design and analysis issues to consider, take a look at the three sample survey questions below and the data reporting that might be provided by an on-line survey site. For each example, there’s a management question and some tips on how to interpret the raw data more realistically.

To understand the example tables below, note that the numbers in parentheses show the actual number of survey respondents who chose each response. The percentages show what proportion of all respondents to the question chose each response.

Example 1:

Example 1

Management’s question: What proportion of employees believes supervisory communication is better now than it was a year ago?

(A)  15%

(B)  38%

(C)  50%

The first reaction might be to take the survey’s results at face value. We could add up the two percentages of respondents who said communication is either much better or a little better, which would give us answer (B), 38%.

However, the survey’s programming calculates the percentages based on all the respondents to the question. In this case, 300 respondents have not been at the company for 12 months, so they can’t possibly recall what communication was like a year ago. Their opinions are not meaningful. For example, someone who joined the company a month ago is far more likely to say nothing has changed

To arrive at the most accurate answer, we would need to do our own math by adding up the number of people who said communication was better (500) and dividing by the total number who have been at the company at least 12 months (1300 – 300). So the right answer is (C), 50%, or 500/1000. Instead of only one-third of respondents seeing an improvement, we find that half of all respondents who have been at the company a year feel communication has improved. That’s a much more successful, and more accurate, conclusion.

Example 2:

Example 2

Management’s question: How effective was this article in influencing its readers to consider buying a thumb drive?

(A)  30% of readers likely to buy

(B)  60% of readers likely to buy

(C)  70% of readers likely to buy

The easy answer would be (A), 30%, because that’s the number the survey software provides. Unfortunately, this answer is much too low because it’s based on all the people who answered this question, not just those who actually read the article. No one can be expected to learn something from a communication they haven’t seen.

To find the true impact of how well this article was written, we first determine how many people read it. Again, we need to look at the number of people (not percentages) who answered this question (1000) and subtract from that total the number who haven’t seen it yet (500). That leaves 500 respondents who saw the article. Of those 500, 300 say they’re more likely to want to buy a thumb drive, or 60%. Now we find this article was twice as effective in influencing consumer behavior as we might have originally told management.

The survey results also tell us that we have a problem in that 50% of respondents don’t recall reading this article. If we can find a way to expand the visibility of this article among our target audience, we’ll have even more possible buyers for our thumb drive. However, this final statistic is a measure of how effectively the article was distributed, not how well it was written.

Example 3

Management’s question: How many of the site’s visitors found the content valuable enough to return at least once?

(A)  45%

(B)  50%

(C)  90%

This one is even trickier. A first approach might be to add up the percentages of all those who visited the site twice or more, which would be 45%, answer (A). 

However, if management wants to know how valuable the site is in encouraging repeat visitors, we need to calculate the multiple visitors as a percentage of visitors overall, not as a percentage of those who were invited to take the survey. Some of the invitees might not have access to the Internet. Others might not have heard about the site yet due to patchy marketing efforts. Others might have heard about it and have access, but feel they have no need for the site because they already consider themselves experts on this topic.

To answer this question, we need to look first at the total number of respondents (2000) and subtract those who haven’t actually seen the site (1000). Now we can look at the number of those who have seen the site (1000), compared to those who returned for at least a second visit (400+300+200), showing that it provided valuable information for them. That’s 900/1000, or 90% (C). That’s twice as successful as 45%.

However, if we really wanted all 2000 in our survey population to be multiple visitors, we would need to ask additional questions of those who said “0 times” to identify why they have not visited the site and solve that problem as well.

Avoid problematic questions

Many of the on-line survey sites offer flexibility in how you phrase questions. If you don’t want to do as much math as was necessary in these examples, you might ask those who are ineligible for a question to skip it. That way the percentages will automatically be calculated only on the basis of those eligible to reply. 

Most of the survey reports will also tell you how many respondents skipped a particular question. You could then do a little math to calculate what percentage of the total number of people responding to the survey did not respond to a particular question. This could show what percentage don’t have access to a communication or are unaware of it.

Another way to design these questions is to ask a qualifying question first. For example, you could ask, “Have you visited our Web site? Yes or No.” Answers to this first question will tell you what percentage of your total audience has visited the site. Many of the on-line surveys allow for “conditional” or “logic” questions. You can program the survey to provide one set of follow-up questions to those who said “Yes” to identify how they use it. You can ask a different set of questions of those who said “No” to find out why they don’t visit your site. The inappropriate questions are suppressed so one group of respondents never knows that other respondents saw a different set of questions.

Setting up questions in this way makes it easier to interpret the numbers accurately without doing as much arithmetic on your own. You’ll also be sure to get full credit for how successful your communications really were.

Angela Sinickas, ABC, is president of Sinickas Communications, Inc., a communication consultancy specializing in helping corporations achieve business results through targeted diagnostics and practical solutions. You can visit her new website, CommToolbox.com, to see the automated planning, measurement, and benchmarking tools she has developed based on her manual, How to Measure Your Communication Programs.