Plan Element #6: Evaluating Your Communications Initiative

Why and When to Evaluate
Other spheres of public health—disease control, emergency preparedness, and health care services for the uninsured, for example—conduct internal evaluations as a regular part of professional practice. LHD communications, too, should evaluate its performance continually. Evaluations will help assure effectiveness and excellence in communications.

Ideally, a communications plan should include an evaluation component. The evaluation should be designed before the communications intervention takes place. That way, the evaluation can be more complete, a sufficient budget can be secured, and data—especially baseline data—can be collected efficiently. But, an evaluation also can be designed after the intervention takes place.

There are various ways to classify evaluations. In the health sphere, evaluations are usually considered either process evaluations (examining how healthcare or public health is provided) or outcome evaluations (examining the results of a health care or public health intervention).

Another way to classify evaluations separates those that present quantitative data from those that present qualitative information, such as anecdotal or narrative reports.

Tip: The CDC Framework for Program Evaluation in Public Health is an Excellent Resource

A user who masters this document will be prepared to talk intelligently with a professional evaluator, or to design and implement a simple evaluation. More »

CDC Framework for Program Evaluation in Public Health

Adapted from: Centers for Disease Control and Prevention, Morbidity and Mortality (MMWR) Weekly Report, MMWR Recommendations and Reports, Framework for Program Evaluation in Public Health, September 17, 1999/48(RR11);1-40

Effective program evaluation is a systematic way to improve and account for public health actions by involving procedures that are useful, feasible, ethical, and accurate. The recommended framework was developed to guide public health professionals in using program evaluation. It is a practical, non-prescriptive tool, designed to summarize and organize the essential elements of program evaluation. The framework comprises steps in evaluation practice and standards for effective evaluation.

The framework is composed of six steps that must be taken in any evaluation. They are starting points for tailoring an evaluation to a particular public health effort at a particular time. Because the steps are all interdependent, they might be encountered in a nonlinear sequence; however, an order exists for fulfilling each—earlier steps provide the foundation for subsequent progress. Thus, decisions regarding how to execute a step are iterative and should not be finalized until previous steps have been thoroughly addressed. The steps are as follows:

Step 1: Engage stakeholders.
Step 2: Describe the program.
Step 3: Focus the evaluation design.
Step 4: Gather credible evidence.
Step 5: Justify conclusions.
Step 6: Ensure use and share lessons learned.

In the Spotlight

Six Steps Cycle

Adhering to these six steps will facilitate an understanding of a program's context (e.g., the program's history, setting, and organization) and will improve how most evaluations are conceived and conducted.

Source: CDC Framework for Program Evaluation in Public Health

To view a larger version of the six steps cycle click here.

Tip: Engage Stakeholders in the Evaluation Plan to Ensure their Perspectives are Understood

Stakeholder input contributes important elements of a program's objectives, operations, and outcomes. If the evaluation does not address their concerns or values, the findings could be ignored, criticized, or resisted.

Assigning Value to Program Activities
Questions regarding values, in contrast with those regarding facts, generally involve three interrelated issues: merit (i.e., quality), worth (i.e., cost-effectiveness), and significance (i.e., importance). If a program is judged to be of merit, other questions might arise regarding whether the program is worth its cost. Also, questions can arise regarding whether even valuable programs make important differences in a community’s health. Assigning value and making judgments regarding a program on the basis of evidence requires answering the following questions:

What will be evaluated? (That is, what is the program and in what context does it exist?)
What aspects of the program will be considered when judging program performance?
What standards (i.e., type or level of performance) must be reached for the program to be considered successful?
What evidence will be used to indicate how the program has performed?
What conclusions regarding program performance are justified by comparing the available evidence to the selected standards?
How will the lessons learned from the inquiry be used to improve public health effectiveness?

These questions should be addressed at the beginning of a program and revisited throughout its implementation. The framework described in the CDC report provides a detailed, systematic approach for answering these questions.

Tip: Agree on What you Want to Measure and How you are Going to Measure It

If you don't get this information right, then results of the evaluation are unlikely to be used.

Classifications

In public health, evaluations are also usually considered either process evaluations, which examine how healthcare or public health is provided or outcome evaluations, which examine the results of a healthcare or public health intervention. In either type of evaluation, the data that are collected may be quantitative (numerical information and data), qualitative (descriptions, narratives, or anecdotes), or both.