The following article appeared in
Strategic Communication Management, February/March 1998
Melcrum Publishing Ltd, London

© 1998 Angela D. Sinickas. All rights reserved.
 

.

Alternative Ways to Measure the Effectiveness of Your Publications

By Angela D. Sinickas, ABC

Q: I've conducted readership surveys before, but is there some other way I can measure the effectiveness of my publication?

Readership surveys are probably the most commonly used measurement tool for communication. Typically, they ask readers questions about:

• How frequently they receive the publication;
• How much of the publication they usually read;
• Which types of articles or recurring columns they prefer to read;
• How easy the writing is to understand;
• Whether the articles are too long, too short or just right;
• How effective the photos and illustrations are;
• How easy the layout is to follow;
• What the preferred frequency of the publication is;
• What the preferred distribution method is.

Generally, most editors find out that most of their readers read most of the issues most of the time and like most of the items in the publication mostly pretty well. This is especially true if the readership survey is included with an issue of the publication itself, which should be avoided. This is because the survey would reach only the people who are receiving the publication and opening it: the publication's biggest fans.

If you want to go beyond the usual limits of a traditional readership survey that tells you how well received a publication is, first clarify your objectives. Then you might include additional "impact" questions on your next survey, conduct in-depth focus groups with readers, and conduct some objective, "audience-free" measurements of the publication to see how well those objectives were met.

Additional Questions for Readership Surveys

A key question management wants answered is what impact communications are having with audiences. It's not enough that employees or customers receive and enjoy the publications.  Are they also doing something useful for the organization because of what they read?

To get at this issue for an employee publication, you can ask questions on your next readership survey like:

• To what extent has reading the employee publication helped employees discuss something about the company more positively with people outside the company?
• With which outside groups have employees discussed publication articles?
• To what extent has reading the publication changed the way employees do their jobs?

Focus Groups

You can adapt a technique used in advertising research called the "Starch Test" (Exhibit One) to obtain in-depth information on many of the topics glossed over in a readership survey.

Convene a focus group of people who say they have read the latest issue of your publication. First show them the cover of the publication and ask them to write down what they remember having read or seen in that issue. This is known as "unaided recall." You can then write the results on a chart pad to see which items have the greatest recall.

Then hand out to each participant a copy of the publication and a worksheet that asks them to identify how much of each article or column they read. This is called "aided recall." After you collect the worksheets for further analysis, you can discuss each article with the group to identify the specific reasons that encouraged and discouraged your readers from getting the most out of each article.  You can also use this technique as a pretest of a publication before it goes to press.

Sample Starch Test Worksheet

Articles/Columns	Skipped It	Skimmed It	Read It All
Cover		X
Letters to the Editor			X
President's Column	X
"How our customers see us"		X

Exhibit One: Starch Test Worksheet

Objective Measurements

Even without going to our audiences to ask their perceptions of what they received, we can conduct a number of measurements to ensure that we are sending out what we should.

Clarity of Writing

One of the simplest, though infrequently used, measures is to determine the difficulty of the writing in our publications. Most word processing software has a built-in tool to do just that. It is usually found under the "tools" or "edit" menus, although it is sometimes disguised as part of a grammar check.  While there are a number of different formulas used, most of them are based on the number of words per sentence and the number of syllables per word.

The scoring of these formulas is often stated as the number of years of formal education required to understand the. Many popular consumer publications are written to be understood by those with seven to nine years of education (approximately ages 13-15).  Unfortunately, many organizational publications are written at a much higher level, partly because of the editorial review process our publications go through at the hands of executives and attorneys. We need to monitor each issue of our publications to make sure the writing can be understood by our average reader.

Content Analysis

A content analysis ( see Exhibit Two, below) is both a measurement and a planning tool to ensure that our publications are focused on topics that meet the publication's goals, such as supporting our organizations' objectives, reinforcing company values, etc.  You first need to identify the ideal content through discussions with management and with your audience, and then track each issue of the publication against the ideals. You can count either the number of inches, centimeters, pages or articles covering each topic. You can then graphically summarize your results to indicate heavy, light or no coverage.  Possible topics would be:

• Covering each key company objective and value;
• Covering locations or business units in proportion to their size;
• Including quotes in articles from average employees rather than only executives;
• Reflecting the diversity of your organization in the photos you use.

  Exhibit Two: Sample Content Analysis

  Ideal Content	Jan.	Feb.	Mar.	Apr.	May	Jun.
  Financial results
  Customer satisfaction
  Employee productivity

  • Heavy	Light	None