The following article appeared in Strategic Employee Publications, Nov. 2000 Ragan Communications, Chicago
	Assess Your Publication's Value and Impact Use these measurement tools to build effective communication By Angela D. Sinickas, ABC
	The next time your boss asks you, "So what has the publications done for us lately?", have some of the following measurements to hand over. Readership Survey Readership surveys are probably the most commonly used measurement tool for communication. (See Chart #1) Typically, they ask readers questions about how satisfied readers are with various aspects of the publication. Some examples: How frequently they receive the publication How much of the publication they usually read Which types of articles or recurring columns they prefer to read How easy the writing is to understand Whether the articles are too long, too short or just right How effective the photos and illustrations are How easy the layout is to follow What the preferred frequency of the publication is What the preferred distribution method is Generally, most editors find out that most of their readers read most of the issues most of the time and like most of the items in the publication mostly pretty well. This is especially true if the readership survey is included with an issue of the publication itself because the survey would reach only the people who are receiving the publication and opening it&emdash;the publication's biggest fans. Self-reported outcome questions If you want to go beyond the usual limits of a traditional readership survey, you might include additional impact or outcome questions on your next survey. A key question management wants answered is what impact communications are having with audiences. It is not enough that employees or customers receive and enjoy the publications.  Are they also doing something useful for your organization because of what they read? To get at this issue for an employee publication, you can ask questions on your next readership survey like: To what extent has reading the employee publication helped employees discuss something about the company more positively with people outside the company? With which outside groups have employees discussed publication articles? To what extent has reading the publication changed the way employees do their jobs? For publications with external audiences, you might ask: To what extent have you purchased additional products or services from the company because of something you read in the publication? To what extent has your impression of the company improved because of something you read? To what extent has the publication affected your donation to a non-profit organization? Starch Test You can adapt a technique used in advertising research called the Starch Test (named after its inventor, Daniel Starch) to obtain in-depth information on many of the topics glossed over in a readership survey.  A readership survey quantifies the readers' perceptions fairly broadly, using the Starch Test in a focus group can pinpoint exactly what they do and don't read and why. How to do this:  Either convene a focus group of people who say they have read the latest issue of your publication or talk in person to individual readers. Show participants the cover of the publication and ask them to write down what they remember having read or seen in that issue. This is known as "unaided recall." Write the results on a flipchart pad to see which items have the greatest recall. Hand each participant a copy of the publication and a worksheet that asks them to identify how much of each article or column they read. This is called "aided recall." Before you collect the worksheets, have the facilitator discover what elements keep people reading and what stops them from continuing.  These results can show which types of articles are actually best read, which may be different from what readers say on a readership survey; the ideal length for different kinds of articles or columns and which design and editing techniques inhibit or increase readership and recall.  After you collect the worksheets for further analysis, you can discuss each article with the group to identify what specifically encouraged and discouraged your readers from getting the most out of each article.  You can also use this technique as a pretest of a publication before it goes to press. Reading grade- level test One of the simplest, though infrequently used, measurements is to determine the difficulty of the writing in our publications. Most word processing software has a built-in way to do just that. It is usually found under the "tools" or "edit" menus, although it is sometimes disguised as part of a grammar check.  (With the more recent versions of Microsoft Word, you also have to be sure to check the box for "Show readability statistics" under Tools/Options/Spelling & Grammar in the pull-down menu. Otherwise you won't see the readability scores after performing the grammar check.) There are a number of different formulas used to determine readability, such as the Gunning-Meuller Fog IndexÔ and the Flesch-Kincaid test. Most of these formulas are based on the number of words per sentence and the number of syllables per word. The scoring of these formulas is often stated as the number of years of formal education required to understand the writing. Many popular consumer publications are written to be understood by those with seven to nine years of formal education (approximately ages 13-15).  A few caveats when performing a reading grade level test: Heavy use of bulleted lists that do not end in periods may be read by the computer as a single, very long, sentence. When reading complex material, many readers prefer to read two grade levels below their maximum potential. When reading on a computer screen, people are more likely to skim, reading fewer of the words in a piece of text than they would if it were on paper according to Sun Microsystems. Your readers may have a good understanding of technical words in your own industry, but those words may drive up the score because of the number of syllables. Fog Index worksheet If you'd like to perform a readability test by hand, you can use the following worksheet. Count out 100 words (don't use bullet lists in sample) 1. Including the entire last sentence that begins within the 100-word sample, how many total words? _________ 2. How many sentences in the sample (including the final incomplete one)? ÷________ 3. Divide #1 by #2 =________ 4. Looking only at the first 100 words, how many words of three or more syllables (not including proper nouns, compound words ending in "es" or "ed" or industry words each employee should know)? +________ 5. Add #3 and #4 =________ 6.Multiply #5 by 0.4 x 0.4 7. This is the grade level =________ Content analysis A content analysis is both a measurement and a planning tool to ensure that a publication is focused on the topics that meet the publication's goals. (See Chart #2)  How to do this: Identify the ideal publication content through discussions with management and with your audience. Track each issue of the publication against the ideals. You can count either the number of centimeters, inches, pages or articles covering each topic (including photos, headlines and magazine covers). Of course, many articles may count toward several content areas. For example, one article might cover two different business goals, three geographies and four business units. Graphically summarize your results to indicate heavy, light or no coverage.  Or you could illustrate your findings to show what percentage of all the content is devoted to various ideal topics. Possible areas to measure may include: Key company objectives and values. Various products or services. Brand attributes. Locations. Business units. Percentage of proactive articles preparing employees for change, versus reactive stories reporting on events. The number of quotes in articles from average employees rather than executives. The diversity of your organization as reflected in the photos, quotes and examples you use. State Farm Insurance Companies has created a content analysis tracking tool using a data base program. They not only track the content of each issue of their flagship publication, but they also enter information by article about the author, sources, and other details. They can track productivity of different staff writers, they can quickly pull a list of all articles on a particular topic when a manager requests it, and they can easily follow-up on stories starting with their original sources list. International publication review Try this technique if you have a global publication and want input on how audiences in various parts of the world are reacting to the writing and design, but you can't afford to do a survey or focus groups in all those countries. (See Charts #3, #4) How to do this: Line up communication experts in those countries who specialize in either internal or external publications like yours and have them critique the publication from their own cultural perspective. Determine which aspects of communication are important to measure for these media. For example, writing accuracy, brevity, clarity, organization; effective use of graphics; consistency with corporate graphic standards, etc. Evaluate each item as if it were an entry in a communication contest. The evaluators can score items numerically using a seven-point scale, as IABC uses for its Gold Quill Awards program:  On each criterion, they start by assuming a score of "4" for satisfactory until they find reasons to either mark it as better or worse than satisfactory, up to a "7" for excellent or down to a "1" for poor.  Enter all scores on a spreadsheet and calculate the averages at the bottom of the columns for each channel and across rows for each criterion you're measuring.  Once all this information and the scores are entered on a spreadsheet, you can use the "sort" function on various rows and columns to see if you notice any trends. Distribution assessments Just producing a perfect publication isn't enough. We need to make sure our readers actually are receiving the publication. (See Chart #5) How to do this: For a publication being distributed as an attachment to an e-mail, send it "receipt requested." Then track which individuals opened the e-mail on which dates. Some e-mail systems, however, might send back a receipt even if the item was deleted without being opened. Check with your technology department to find out how this technique can be best implemented in your organization. For internal publications, call people at the bottom of the organization chart in each of your organization's locations.  Ask them when they received a particular publication that should have reached them about a week earlier. Ask them how it was distributed:  in a personal mailbox, in a box on the floor near the time clock, posted on a bulletin board, in a staff meeting. Keep track of the responses by location. Repeat at least twice a year for each different medium you distribute:  publications, video news magazines, job postings, a memo from the president, a poster. You will probably be unpleasantly surprised by your findings and need to take some follow-up actions. Finally, a more comprehensive assessment of distribution can be done on a survey, as long as the survey is not included within the channel of communication you are measuring. Benchmarking You can also benchmark the cost of your publication, or the number of staff it takes to produce it, against other publications of similar length, frequency and use of color/design. (See Chart #6) However, be careful to compare the right kinds of numbers so the comparisons are meaningful. For example: For staffing comparisons, make sure all the organizations count full-time, part-time, freelance and volunteer staff in the same way. You might convert all the variations into full-time equivalent staff (FTEs). For budget comparisons, make sure the budgets all include or don't include the same items, such as staff salaries, freelance budgets, printing and distribution costs. Finally, if you're measuring the efficiency of the publication through comparisons of budget and staffing, also be sure to compare measures of effectiveness. You might be near the high end of the staffing or cost comparisons, but if your readership surveys have higher scores on similar questions, than your organization is receiving a better return on that higher investment.         Chart #1 (return to text)       Chart #2 (return to text)       Chart #3 (return to text)   Chart #4 (return to text)         Chart #5 (return to text)           Chart #6 (return to text) © 2000 Angela D. Sinickas. All rights reserved. Angela Sinickas, ABC, is president of Sinickas Communications, Inc., a communication consultancy specializing in helping corporations achieve business results through targeted diagnostics and practical solutions. You can visit her new website, CommToolbox.com, to see the automated planning, measurement, and benchmarking tools she has developed based on her manual, How to Measure Your Communication Programs.