Evaluating and Managing Surveys

By Angela D. Sinickas, ABC

While surveys aren’t the only research tool available to HR managers, they are the most useful one when “hard” numbers are needed on how many people see things a certain way and when it’s important to track differences among subgroups or improvement over time.

Some of the difficulties HR managers experience with surveys fall into several categories:

·        Designing a good survey that is easy for respondents to complete and provides HR with actionable data.

·        Administering it to the right number of people and obtaining a high enough response rate for statistical validity.

·        Choosing and managing vendors when outside help is needed.

Designing survey questions and formats

Whole books have been written about wording survey questions, but here are some of the most important tips to remember:

Make sure the wording will result in specific actions you can take.

Think about the aspects of your benefits or HR communication program that could be changed, and ask questions that will let you know if your employees think they should be. Then work backwards to phrase questions in a way that will suggest actionable answers.

After you draft your initial questions, pretend first that you received a highly favorable response, and then a highly negative one. 
Then ask yourself: Did I learn enough to know what actions to take to turn a negative response into a favorable one? 
For example, let's say you ask people to agree/disagree with the following statement: "The benefits newsletter should 
continue to be published once a month." If they disagree, you don't know whether to increase or decrease the frequency. 
It would be better to ask people to select their ideal frequency from a list of options (weekly, monthly, quarterly, etc.).

Another example of a question with an unclear response would be asking employees to agree or disagree with the statement:  “Benefits communication has improved in the last year.”  For those who disagree with the statement, do they perceive it has become worse or stayed the same? A better scale would be one that asks if communication has become worse, become better or stayed the same.

Strive for maximum clarity.

Use clear and simple phrasing. Do a reading grade level check on the survey to keep it between grades 8-10. 
(This article, for example, is written at grade level 10.8 as measured by the readability statistics under the Grammar and 
Spelling Tool in Microsoft Word®.)

Avoid words that can be interpreted differently. Obvious ones include words like bimonthly, which can mean twice a month 
or every other month. Other typical words or phrases that need to be either avoided or defined when you use them include a 
great deal of HR terminology that doesn’t mean the same thing to employees as it does to us. For example, if we talk about total 
compensation being competitive with the marketplace, we mean it’s about average. Employees more often think of “competitive” 
meaning better than average. 

Be sure each question asks about only a single item. For example, don't ask people how well they understand their pay and benefits programs. They may understand one better than the other, but this question doesn’t allow them to tell you which one they need more information about. Then you won't know which problem to fix.

Minimize the likelihood of inaccurate responses.

Before you offer a neutral, “no opinion” response for a question, be sure that it is a meaningful option. Often, “not applicable” may be a more appropriate choice. At some organizations, a very high percentage of respondents choose “no opinion” on a sensitive issue when in fact they hold a negative opinion but are fearful of being identified individually and punished by their managers.

Phrase questions in a way that prevents people without a legitimate, informed opinion from answering. For example,
 if you ask if claims administration “has improved,” “has worsened” or “has stayed the same” during the last 12 months, 
you need to include an option that says: "I haven't been here 12 months." Otherwise, people who have been hired recently 
would probably choose "stayed the same" and dilute the true results from those who have been here during the entire time period.

Finally, pretest your survey with a small cross-section of the types of respondents you're likely to have. Ask them which questions are difficult to understand or to answer, which questions are missing a response category they'd like to have available, etc. Ask them how they interpret different words or phrases.  Pretesting will prevent at least two problems: 

·        Employees interpreting questions differently from how you will interpret their responses and

·        Employees making errors in how they complete the survey, compromising the validity of your data.

Administering surveys

Some of the most common questions about surveys are: “How many surveys do I need to distribute?” “What’s the right sample size?” “What’s a good response rate?”

The right sample size

The number of surveys to distribute depends on how many employees you have and what rate of response you are likely to achieve. If you have a relatively small number of employees, you might need to send out surveys to everyone. If you have over 2,000 employees, you would need only 500-600 completed surveys to have fairly reliable results for your population as a whole, assuming the respondents accurately reflect the demographics of the entire group. So, if you expect to have a 100% response rate, you would mail out surveys to a random sample of 600. More realistically, if you typically have a response rate of 50%, you’d need to survey 1,200 people (600 divided by 0.5).

Most organizations also want statistical validity in comparing organizational subgroups against each other (locations, business units, etc.). This typically requires a much larger number of responses than 600 so that you have a sufficient proportion of each subgroup participating. For example, if you have two locations of several thousand employees each, you’ll need 600 responses for each location. Also, smaller subgroups may need a larger proportion of the group responding for statistical reliability than larger subgroups.

Deciding a sample size is best done with the help of a statistician who knows not only how to sample employees, but also how to pick a random sample. If you don't do this step carefully, some executive with even a little background in statistics may invalidate all the results of your survey when you're finished.

How to maximize your response rate

You need to achieve a good response rate or you’ll wish you had sent out more surveys. The response rate on voluntary surveys generally varies from 20% to over 80%. Where your own response rate will fall depends on a number of factors:

·        The length of the survey. The longer it is, the lower the response rate.

·        Demographics questions. If there are too many of them, or if they are all at the beginning of the survey, the response rate plummets.

·        Previous experience. If a company has administered many surveys and never reported back results or made changes based on the 
surveys, the response rate will continue to go down with each new survey.

·        Management support. If senior management lets middle management know that they really want to see the results and want to see 
good participation in all units, managers make sure employees are given some time to complete it. Otherwise, they chide employees 
about "not working" while they're completing surveys.

·        Incentives. If there is a reward for the location or the department with the best response rate, or if every location with at least a 
threshold return rate receives a reward, that makes a big difference. Then peer pressure gets results; for example, if you offer an 
extra day off around a three-day weekend for each "winning" location.

·        How and where the survey is administered. Paper surveys sent to the home will have a lower response rate than those distributed individually at work. (However, you need to be aware that some employee groups don't have the physical environment at work that provides so much as a writing surface.) Of course, group administration in small meetings will get the best rate. Electronic surveys (Web, e-mail or phone) tend to have responses come in more quickly. With paper, many delay until closer to the deadline. However, with some electronic administration methods, people are more likely to feel that they could be identified individually, which reduces return rates. If you're asking questions of a sensitive nature, such as intention to stay with the company, you might get a lower response rate electronically than on paper.

How to avoid “over surveying”

Management often believes employees are over surveyed before employees do. Employees feel over surveyed in some of the following 
situations:

·        Surveys are developed by individuals with no survey design background and include questions that are difficult to understand or 
respond to, or seem to have no relevance for improving employees' own work experience.

·        When employees never hear the results of past surveys.

·        When employees see changes in the company, but there is no reference made to the relationship of the changes to past employee 
surveys.

·        If the employee population is relatively small (under 1,000) so that every survey is sent to every employee every time, rather than 
sending different surveys to different randomly selected groups of employees.

Techniques that help overcome "survey fatigue":

·        Coordinate all employee surveys through a clearinghouse so that the timing of surveys doesn't overlap and you don't ask questions 
for which answers are already available.

·        In the introduction to a new survey, begin with key findings and changes made based on a previous survey or other form of 
employee research.

·        Send the survey to only a sample of employees. If you know several surveys will be administered about the same time, pick 
mutually exclusive samples at the same time so that no one person receives more than one survey during that time period.

·        Literally connect changes the company is making with employee survey results when you announce the changes.

·        Consider doing very short "stealth surveys" for which you don't obtain advance permission. People might not even know they've 
been surveyed. For example, obtain a list of 400 to 600 randomly selected names and divide the list among 10-15 of your 
colleagues at work (or assign this project to an intern). Then have your deputized research team call these employees on the 
phone, identify themselves and then say they're wondering about what people think about a couple of topics you're planning to 
communicate about. As the research team talks with employees, you would actually be recording their answers on survey forms. 
Or, somewhat less scientifically, you could stand near lines in the cafeteria or the credit union and ask people a few questions 
while they're waiting in line. It's not statistically defensible, but it will certainly give you a good directional reading of where a 
wide range of different types of people stand on a topic.

Tips on vendors and survey software

Factors to consider in bidding a survey project

Here are some ideas to help you obtain comparable quotes from all survey vendors to help you make meaningful comparisons before choosing your final survey partner.

·        Have all vendors assume the same number of questions in their pricing (fixed-choice and open-ended), same number of respondents and same number of demographic identifiers. Even though these are likely to change during the course of the actual project, pricing differences for the changes will likely be proportional to the original estimates.

·        Identify if you will want to see only responses to individual questions, or indexed summaries of categories, or advanced statistical analyses. The more analysis you want, the more it will cost. Some consultants include more or less analysis in their base-price assumptions.

·        Describe the type of report will you want: narrative or slides? Mostly numbers, mostly graphics or mostly words? Each of these decisions will affect the time it will take to create the reports, and therefore the cost.

·        Clarify to what extent you will want demographic variations discussed in the final report. Will the computer data report show which variations are statistically significant and which are not?

·        List which steps will be done in-house versus by the consultant (pre-testing, printing, pulling a random sample, reminders, data processing, writing a summary to share with employees)?

·        Identify how many meetings of what length you will expect the consultants to attend to present the results and discuss recommendations.

Questions to ask during the vendor selection process

Once you receive a number of bids and identify your finalists, it is time to ask questions to clarify how you will work together and what possible other costs there might be. Be sure to discuss:

·        How much you can personalize the questions. Some vendors will offer a very low price, but then will expect you to use a specific list of questions that will help grow their database, but may not have much relevance for you.

·        Whether you will be required to offer a no-opinion option or if you can use some forced-choice questions.

·        Whether they will provide database comparisons. If so, ask how extensive the database is and how many companies like yours are in there.

·        Whether there will be a mark-up on outside vendor costs.

·        If the firm charges an “overhead” charge in addition to the quoted fees and expenses. These can range from 3% to nearly 15%, which has a major impact on your total budget.

·        Who will be your regular contact and who will be doing different parts of the work. Some firms bring out their senior consultants for sales meetings and then hand off projects to junior staff once the projects begin. If that will happen, usually to keep the budget manageable, be sure to have the junior staff attend the proposal meeting so you can judge how easy they will be to work with and how knowledgeable they seem to be.

Final thoughts

While surveys shouldn’t be your first choice for research in all situations, when a survey is the right choice, be sure to:

·        Pretest questions to ensure clarity and actionability.

·        Use an expert to determine your sample size.

·        Get reports that will meet your needs.

·        Communicate findings and ACT on them!

© 2004 Angela D. Sinickas, All rights reserved

Angela Sinickas, ABC, is president of Sinickas Communications, Inc., a communication consultancy specializing in helping corporations achieve business results through targeted diagnostics and practical solutions. You can visit her new website, CommToolbox.com, to see the automated planning, measurement, and benchmarking tools she has developed based on her manual, How to Measure Your Communication Programs.