Measuring Real Knowledge

Q: Can you please help me figure out how to measure what employees actually know versus what they “think” they know. For example, “85% of employees understand the organization’s corporate strategy, etc.” How do we know whether employees “get it” or whether they think they get it? Is it acceptable to actually test their knowledge by supplying say five choices for answers, three of which are correct? From your experience, do employees have any problems with this “testing” approach?

Many thanks for your assistance.

Cathy

A: Dear Cathy,

People’s perceptions of what they know can be much higher than how much they really know. However, the perceived knowledge is also a very useful measure.

• You want to see which subjects people feel they know more about than others. If they think they understand a topic very well, that might indicate that they are less likely to read an article, visit a website or attend a meeting on that topic, unless you find a way to intrigue them into learning more.
• Also, tracking perceived understanding of a single topic over time can indicate whether your audience feels it is getting more information over time. This would be good to track against the volume of information you are sending out on that topic, and the channels you’re using.

I absolutely recommend asking actual knowledge questions to track real improvements in understanding among your audience. This can be in the form of True-False or multiple-choice answers.

• You can ask a couple of knowledge questions anonymously at the beginning of a meeting, training session, sales conference, etc. and then repeat them at the end of the session to measure the impact of that communication event.
• Knowledge questions can be administered as part of a larger opinion survey or they can be asked one at a time just before you launch an informational campaign on a topic. For example, you could call 400 randomly selected employees and ask them up to three knowledge questions on a topic you’re about to communicate heavily. This would give you a baseline to compare against during and after the campaign.
• Depending on your audience’s access to online communication channels, you could also administer these questions online.

I remember hearing about one company that turned a knowledge test into something very like a radio contest. They would randomly contact people either by phone or on the plant floor and ask them to name the three objectives for the year. Anyone who got it right won a mug, t-shirt, cap, etc. with the company logo and had their name put into a drawing for a really big prize. Obviously, news of the “contest” spread on the grapevine and everyone started studying so that they would know the right answers if they were called upon. This particular approach to measurement also has the advantage of actually improving the level of knowledge during the measurement process, which is the overall objective anyway. Even the company president got into the act and would ask employees the question when he was wandering around different work areas.

A few words of advice, though, on administering knowledge questions.

• Be sure you know what the “right” answers are throughout the organization. I had one client where the corporate message was that a particular company initiative was NOT a cost-cutting program; however, during an in-house training session I conducted for them, we discovered that one of the business units was communicating to its employees that the initiative WAS a cost-cutting program.
• It helps to introduce knowledge questions with a comment like: “The following X questions do have right and wrong answers. We’re trying to find out how well the Communication Department has done its job of informing you about topics the company wants you to know about. Please circle the answers that you believe are correct. However, if you don’t know, don’t guess. Just choose the option that says ‘I’m not sure.'”
• The “I’m not sure” option helps provide you with richer data. First, you’ll know how many people know that they don’t know and might be more open to learning something about the topic. You’ll also discover how many people think they know the right answer, but are wrong. They will be a tough audience to reach if they already think they know all there is to know on a topic. You may have to “market” the information to them in a way that reaches them before they know what hit them. The people who get the answer right is the number you want to track over time as a measure of the success of your communication program.

Angela D. Sinickas