Measuring the Effectiveness of Mass-Mediated Health Campaigns Through Meta-Analysis

Abstract

A meta-analytic review was undertaken to examine the effects of mass communication campaigns on changes in behavior, knowledge, and self-efficacy in the general public. A review of the academic literature was undertaken and identified 1,638 articles from 1966 through 2012. Using strict inclusion criteria, we included 63 studies for coding and analyses. Results from these efforts indicated that campaigns produced positive effects in behavior change (r = .05, k = 61) and knowledge (r = .10, k = 26) but failed to produce significant increases in self-efficacy (r = .02, k = 14). Several moderators (e.g., health topic, the theory underlying the campaign) were examined in relation to campaign principles that are prescribed to increase campaign effects. The major findings are reviewed, and the implications for future campaign design are discussed.

Notes

¹ Noar’s (2006b) principles of effective campaign design also include recommendations to segment the audience, conduct process evaluation, and implement appropriate methods of outcome evaluation. These factors are not addressed in the current study.

² Campaigns relying on mass-mediated channels to communicate a message to the community (e.g., television public service announcements) that were merely supplemented by efforts in controlled environments (e.g., schools, worksites) were included in the current analysis.

³ Carlson and Schmidt (1999) presented suggestions for including multiple study designs in a single meta-analysis to avoid the loss of power associated with the current approach. However, Carlson and Schmidt’s analysis of effect sizes in the literature on training evaluation also demonstrated that (a) control groups often experience change over time; and (b) that posttest standard deviations are often inflated, resulting in the calculation of differing effect sizes based on study design. Although corrections exist for these problems (see Carlson & Schmidt, 1999, for a discussion of this issue), a decision was made to include studies of a single design to avoid bias in the calculation of effect sizes.

⁴ To support the explicit definition of a mass-mediated health campaign employed in the current analysis, the primary author and a second coder examined approximately 10% of the original 424 articles and coded each according to whether it met the established definition of a health campaign (Cohen’s κ = 0.77). Differences were resolved by discussion of inclusion criteria between the two coders, and the remainder of the sample was coded by the primary study author. The majority of the 338 (n = 273) were eliminated because they did not meet our definition of a mass-mediated health campaign. Smaller portions of studies were eliminated because of insufficient reporting of results (n = 49) or lack of a pretest (n = 13).

⁵ A wide array of studies were eliminated for incomplete information, such as studies employing a pre-/posttest with control group design that (a) presented/evaluated posttest data only, (b) presented analyses unrelated to health behavior change (e.g., cost-effectiveness), (c) utilized a national control group rather than a neighboring community as a control, or (d) presented incomplete or inadequate data (e.g., missing information on the standard deviation of pretest scores, presentation of data pertaining only to the standard error in change scores).

⁶ In the current article, studies are identified with reference to the article(s) from which the pertinent effect size was derived. References to supplementary studies that provided additional information on moderator coding are not included in the interest of space, but this information is available from the first author.

⁷ Those studies that referenced a theory only in the discussion of campaign outcomes were not considered theoretically based, as theory presumably did not drive the implementation of the campaign but served only as a framework for discussion.

⁸ Per the definition of health campaigns provided earlier in the current article, each campaign had to include at least one brief message for planned delivery (e.g., use of a public service announcement on television). Occasionally campaigns employed a brief message for planned delivery (e.g., newspaper advertisement) in one channel but also employed more extensive educational efforts (e.g., an interview with the campaign creator on a radio talk show) that would not, on their own, meet the definition of a health campaign provided in the current article. In such instances, channels containing the more extensive educational content were still coded as present, as the channel was still utilized to transmit information to the target audience.

⁹ Initially we also intended to examine attitudes and intentions as relevant outcome variables. However, too few studies measured these outcomes to conduct meaningful analyses.

¹⁰ As noted in the Method section, we did aim to locate supplemental studies on each campaign, but it is possible that formative research efforts remain unidentified or unpublished.

¹¹ It is worth noting that CIs pertaining to each subset of studies (i.e., those employing/failing to employ formative research) were both positive and did not include zero. Thus, although campaigns without formative research appeared to be more successful than those using the technique, campaigns still outperformed control communities even when formative research was employed.