Perceived Message Effectiveness and Campaign-Targeted Beliefs: Evidence of Reciprocal Effects in Youth Tobacco Prevention

ABSTRACT

Perceived message effectiveness (PE) has been widely used in campaign formative research and evaluation. The relationship between PE and actual message effectiveness (AE) is often assumed to be causal and unidirectional, but careful conceptualization and empirical testing of this and other causal possibilities are generally lacking. In this study, we investigated the potential reciprocity in the relationship between PE and AE in the context of a national youth tobacco education campaign. In so doing, we also sought to generate much needed evidence on PE’s utility to predict campaign-targeted outcomes in youth tobacco prevention. Using five waves of campaign evaluation data (N = 1,128), we found significant lagged associations between PE and campaign-targeted beliefs, and vice versa. These results suggest a dynamic, mutually influencing relationship between PE and AE and call for greater attention to such dynamics in campaign research.

Introduction

Perceived message effectiveness (PE) has been widely used in campaign research, both as a tool to assess message potential in formative research (Dillard, Weber et al., 2007; Fishbein et al., 2002), and as a surveillance device to monitor audience receptivity during campaign implementation (Duke et al., 2015; Rath et al., 2019). The last few years have seen a surge of interest in this concept and its application, leading to increased awareness of some important gaps in the literature (Noar, Bell et al., 2018; O’Keefe, 2018a, 2020; Yzer et al., 2015). At the core of the ongoing discussion is the relationship between PE and the attitudinal and behavioral outcomes that campaigns are ultimately interested in (termed actual effectiveness [AE]). While previous research suggested a robust relationship between PE and AE (Dillard, Shen et al., 2007; Dillard, Weber et al., 2007), recent reexamination of the literature has called this conclusion in question (O’Keefe, 2018a, 2020). Concurrently, there is also concern that many PE measures have been developed and put to use without rigorous validation within the proper campaign context (Noar, Bell et al., 2018). This also adds murkiness to the current evidence base.

The purpose of this study is two-fold. First, it revisits the directionality issue in the PE-AE relationship and considers the possibility of mutual influence between PE and AE over time in a real-world campaign context. This possibility is assessed through a cross-lagged panel model using longitudinal evaluation data from The Real Cost (TRC), a national youth tobacco prevention campaign. Second, through the same analysis, this paper also seeks to provide validation evidence for the PE scale used in TRC evaluation and formative research (Davis et al., 2013). This scale has been previously validated in the context of adult-targeting smoking cessation interventions. Its ability to predict prevention outcomes in tobacco education campaigns targeting youth has not yet been formally established. This study hopes to bring relevant evidence to the assessment of its utility in this important campaign context.

Literature review

A general definition of PE is “an estimate of the degree to which a persuasive message will be favorably evaluated – in terms of its persuasive potential – by recipients of that message” (Dillard, Weber et al., 2007, p. 617). The concept of PE and its various operationalizations have appeared with regularity in the literature. A recent systematic review focusing on tobacco education campaigns alone found 75 studies employing 126 PE measures in 21 countries (Noar, Bell et al., 2018). The widespread use of PE is often attributed to its expected utility in message development and selection in the formative phases of persuasive campaigns (Dillard, Weber et al., 2007; Noar, Bell et al., 2018; O’Keefe, 2018a; Yzer et al., 2015). The general idea is that PE would offer a useful indicator of candidate messages’ eventual ability to bring about the intended campaign outcomes once they are placed into the market. Based on the same logic, PE measures have also been used by campaigns to monitor audience receptivity to already-deployed messages in order to detect unforeseen problems and enable mid-campaign adjustments (Duke et al., 2015; Rath et al., 2019).

Issues in PE research

Despite its widespread usage, important gaps exist in the PE literature and recent reviews have raised several related issues. The first is that PE research is theoretically underdeveloped. Indeed, recent reviews noted considerable ambiguity around the PE concept (Noar, Bell et al., 2018; Yzer et al., 2015). Part of the ambiguity is reflected in the lack of attention to the causal chain of message effects in which PE is embedded (Yzer et al., 2015). As noted earlier, the relevance of PE in campaign work is often premised on the assumption that PE is predictive of actual campaign outcomes. How this relationship takes shape and evolves in a dynamic campaign context, however, has not been adequately addressed.

A second issue is the marked heterogeneity in the operationalization of PE in the literature (Noar, Bell et al., 2018; O’Keefe, 2018a; Yzer et al., 2015). A line of work that employs more than one hundred different measures for the same concept is unlikely to produce highly coherent and replicable findings. Moreover, most measures that exist have not gone through careful validation (Noar, Bell et al., 2018). This adds further murkiness to the evidence bearing on PE’s relevance in campaign research and practice. Some more widely adopted measures, including the scale that will be further tested in this study (Davis et al., 2013; also see Zhao et al., 2011), have been validated in some campaign contexts but not in others. Health campaigns are highly situationally defined endeavors – whether and how a campaign research tool works tend to vary by population, behavior, time, and context. Continued testing of these instruments in additional campaign settings will further our understanding of the generality of their utility.

The third issue that has been raised directly questions the quality of the evidence in support of PE’s ability to predict actual campaign outcomes (AE). These outcomes are typically issue-specific beliefs, attitudes, and/or behaviors campaigns seek to influence. In an early meta-analysis, Dillard and colleagues found a fairly robust association between PE and AE (r = .41) across 40 studies (Dillard, Weber et al., 2007). A recent meta-analysis by O’Keefe (2018a), however, found that the relative standings of messages on PE did not consistently match their relative standings on AE (58% of the time, not significantly different from 50%). This finding calls in question the ability of PE to accurately predict AE and has sparked a lively debate among researchers (Cappella, 2018; Davis & Duke, 2018; Noar, Barker, & Yzer, 2018; O’Keefe, 2018b). It should be noted that, in O’Keefe’s analysis, when message discrepancies on PE were relatively large (d ≥ .326 [median]) or statistically clear (p < .05), the correspondence between PE and AE improved to a level (64% to 67%) that was significantly better than chance. Overall, it appears that it would be premature just yet to draw definitive conclusions about PE’s ability or inability to predict AE. More and better research would help enrich the evidence base.¹

The issues identified above are clearly interrelated and efforts to address them have started to emerge in the literature (Baig et al., 2019; Bigsby et al., 2013; Yzer et al., 2015). This study hopes to join these efforts by bringing additional clarity to the role of PE in the causal process of campaign message effects. More specifically, it considers the possibility that the causal flow between PE and AE may be bidirectional in light of both past PE research and recent developments in communication theory.

Causality in PE-AE relationship

In an early treatment of the causality issue, Dillard and colleagues discussed several theoretical perspectives and research traditions that might support a causal flow from PE to AE (Dillard, Shen et al., 2007; Dillard, Weber et al., 2007). Research on attitude toward the ad, for example, shows that feelings about an advertisement (akin to PE) are causally antecedent to attitude toward the brand and other downstream variables, such as purchase intention (Brown & Stayman, 1992; Shimp, 1981). Other research traditions, such as the cognitive response theory (Greenwald, 1968; Petty et al., 1981), similarly suggest that messages that elicit more favorable quality judgments will in turn produce greater impact on issue attitudes. These perspectives converge to suggest that ad evaluations such as PE should causally predict actual ad effects.

Dillard and colleagues also discussed the possibility of AE causally influencing PE (Dillard, Shen et al., 2007; Dillard, Weber et al., 2007). There is plenty of evidence that people tend to perceive messages that they agree with and accept as more logical, compelling, and persuasive (Kunda, 1990; Liberman & Chaiken, 1992; Lord et al., 1979). In other words, message acceptance can drive message quality evaluations. Motivated reasoning, for example, posits that when people are motivated to arrive at preferred conclusions, they will deploy biased strategies to process incoming information, resulting in over- or under-estimation of message quality (Kunda, 1990). Social judgment theory similarly suggests that message perceptions can be influenced by where the message-advocated position lands in relation to one’s current position (Sherif et al., 1982). Close proximity will lead to positive distortions while large discrepancy will lead to negative distortions in message perception. Together, these theoretical perspectives lend support to the possibility that PE might itself be affected by AE.

Dillard, Shen, et al. (2007) also briefly considered the possibility of reciprocal relationship between PE and AE as a logical extension of the two unidirectional possibilities. While this reasoning makes good sense, it should be noted that recent communication scholarship has increasingly gravitated toward a view that acknowledges the inherent bidirectionality between communication activity/process variables and individual attitudes, beliefs, and behaviors (Cho et al., 2009; Eveland, 2001; Shah et al., 2017; Slater, 2007, 2015). The cognitive mediation model, for example, considers information processing variables, such as attention and elaboration, to be important mediators between predispositional audience factors and political learning in the context of news viewing (Eveland, 2001, 2002). Empirical testing of the model using panel data further reveals that information processing variables and learning outcomes such as political knowledge mutually influenced each other over time (Eveland et al., 2003). To the extent that PE represents a form of cognitive response in information processing, it would be consistent with the cognitive mediation model to expect a reciprocal relationship between PE and AE in an evolving persuasive campaign context. Similar patterns of mutual influence between audience activity and media effects are also proposed in other dynamic models of communication, such as the reinforcing spirals framework (Slater, 2007, 2015).

While early research noted theoretical possibilities for different causal directions between PE and AE, evidence from several studies consistently favored the PE-AE causal order over the reverse (Dillard, Shen et al., 2007). Note, however, that in previous research, the possibility of mutual influence between PE and AE was not empirically assessed, partly due to study design limitations. To fully address the possibility of a bidirectional relationship, longitudinal data are needed with multiple assessments of both PE and AE.

PE and tobacco education campaigns

The PE-AE relationship has been examined across diverse health contexts, from dental hygiene and binge drinking to HIV testing and drug use (e.g., Davis et al., 2011; Dillard, Shen et al., 2007; Zhao et al., 2011). One applied context where PE has seen consistent use is tobacco education campaigns, where more than seventy studies were found to have used PE in a recent systematic review (Noar, Bell et al., 2018). Despite the number of studies in this area, relatively few have offered validation evidence for PE’s ability to predict AE beyond cross-sectional associations. Because PE itself is rarely manipulated (for an exception, see Dillard, Shen, et al. 2007, study 5), even experimental studies would typically only produce cross-sectional data when it comes to the relationship between PE and AE. Longitudinal studies are better able to support causal inference, but they are few and far between in the current literature. A recent meta-analysis located only six longitudinal studies, conducted by three research teams, all focusing on smoking cessation advertising targeting adult smokers (Noar et al., 2020). The results of the meta-analysis were overall encouraging – across available studies, PE was found to be prospectively associated with increased quitting intentions (r = .256, p < .001) and actual cessation behaviors (r = .201, p < .001). But the limited number and scope of such studies suggests a need for more research to further examine PE’s utility in a broader array of intervention contexts. Indeed, what is conspicuously lacking from the current literature is evidence for PE’s ability to predict youth tobacco prevention outcomes (Noar et al., 2020). Given that youth prevention is an integral part of tobacco control in the U.S. and worldwide, effort to address this research gap has clear value. Moreover, the existing longitudinal studies have exclusively focused on the causal path from PE to AE. The dynamic possibility of reciprocal effects has never been examined. This study will seek to fill these voids by examining the longitudinal, bidirectional associations between PE and campaign-targeted beliefs in the context of an ongoing national youth tobacco prevention effort.

Campaign context

The Real Cost (TRC) is a youth tobacco education campaign conducted by the U.S. Food and Drug Administration (Crosby, 2019). Launched in 2014, TRC is aimed at reducing tobacco use among U.S. youth ages 12–17. This includes the prevention of initiation among susceptible youth and the reduction of progression to regular use among youth already experimenting. For the first four years, the campaign focused entirely on cigarette use. As vaping becomes increasingly prevalent among youth, the campaign recently expanded its scope to also address electronic cigarette use among youth (Crosby, 2019). The current study will use data from the first four years of the campaign and focus only on cigarette use prevention messages.

TRC is guided by well-established behavioral and communication theories (Zhao et al., 2016) and follow the CDC guidelines on best practices in tobacco control interventions (CDC, 2018). Campaign effects are monitored through longitudinal evaluation surveys of a probability-based youth sample. Evidence so far suggests that campaign advertising has reached the vast majority of its target population and prevented a substantial number of youth from smoking initiation (Duke et al., 2015, 2018, 2019; Farrelly et al., 2017).

PE has factored prominently in the development and evaluation of TRC. The scale used in the campaign was developed by Davis and colleagues (Davis et al., 2013) and focused on the extent to which campaign messages would resonate with the target audience. Through a serious of studies, Davis and colleagues showed that the scale had desirable internal consistency and was able to prospectively predict both cognitive and behavioral outcomes in the context of smoking cessation advertising targeting adults (Davis et al., 2017, 2013). Encouraged by this existing evidence, TRC has adopted the scale for use in both formative research and outcome evaluations. But the scale’s ability to prospectively predict campaign outcomes has not yet been formally investigated.

This study takes advantage of the multiple waves of data collection in the TRC evaluation and report a cross-lagged analysis to look at the bidirectional relationship between PE and campaign-targeted beliefs over time. The logic model for TRC considers belief change a necessary and important intermediate outcome for the campaign (Crosby, 2019; Duke et al., 2018). While the ultimate goal of the campaign is to prevent smoking initiation, the nature of the behavioral outcome – once youth initiate smoking, they cannot return to a nonsmoking status – calls for a different analytical approach and will not be pursued in the current analysis. Focusing on belief change, the results of this study will offer novel insights into the possibility of a dynamic and mutually reinforcing relationship between PE and AE over time. They will also provide new validation evidence for PE in the realm of youth smoking prevention. Both types of evidence should serve to enhance our understanding of PE and its relevance in campaign research and practice.

Method

Survey and sample

Data for this study came from a five-wave national longitudinal evaluation survey for TRC conducted by RTI International. Using address-based sampling supplemented with market research databases, the survey generated a nationally representative sample of youth aged 11 to 16 at baseline. Baseline data collection was conducted through in-person interviews from November 11, 2013 to March 31, 2014. Follow-up data collection used both in-person and online interviews and occurred at an interval of roughly eight months between waves. The dates for the follow-up surveys were as follows: July 14, 2014 to October 27, 2014; April 6, 2015 to July 4, 2015; December 17, 2015 to April 5, 2016; and September 15, 2016 to November 22, 2016.

All data collection occurred after obtaining parental permission and youth assent. Youth participants received 20 USD for completing the baseline survey and 20 USD or 25 USD for completing the follow-up surveys, depending on how soon they completed them. Weighted household-level response rate (using AAPOR Response Rate Formula 3) was 43.7% at baseline. The person-level retention rate for the follow-ups ranged from 84.9% to 91.4%. The baseline sample included 6,742 youth and 4,210 of them completed all four follow-ups. For the current study, only youth directly targeted by the campaign (experimenters and susceptible nonsmokers) who reported exposure to at least one TRC ad (thus contributing data on PE) at each wave were included. To produce reliable estimates of target population parameters, baseline analysis weights that accounted for unequal probabilities of selection at each stage were adjusted for nonresponse at follow-up. Then, weights were calibrated to the Census 2010 population totals of the baseline target population with post-stratification for gender and race/ethnicity. The study was approved by the institutional review boards at FDA and RTI International.

Campaign advertisements

The central theme of TRC was “Every cigarette costs you something.” Campaign advertising was developed on three general message platforms: short-term health effects of smoking, loss of control due to addiction, and toxic chemicals in cigarettes. Ads targeting different beliefs were released on a flighted schedule to ensure full coverage of all three platforms over time while avoiding specific message/platform fatigue at any given point in time. TRC advertising appeared on national television, radio, the internet, and out-of-home displays, as well as in magazines, social media, mobile gaming, and at movie theaters. This study focuses on video advertisements, the only form of advertising where PE ratings were consistently obtained in the outcome evaluation. Description of all TRC video ads and their dates of airing are available in a previous publication (Duke et al., 2019).

Measures

Smoking status

Respondents were asked about their smoking experience and expectations at baseline. Those reporting never smoking (not even one or two puffs) were asked three questions about intentions to smoke in the future (Pierce et al., 1996), p. 1) Do you think that you will try a cigarette soon? 2) Do you think you will smoke a cigarette anytime during the next year? 3) If one of your best friends offered you a cigarette, would you smoke it? Answering options included definitely yes, probably yes, probably not, and definitely not. Following conventions in the literature, youth who answered definitely not to all three questions were categorized as nonsusceptible nonsmokers; all others were deemed as susceptible nonsmokers. Youth who reported smoking fewer than 100 cigarettes in their lifetime were considered experimenters. Youth who reported smoking 100 or more cigarettes in their lifetime were defined as current or former smokers. The target audience for TRC were susceptible nonsmokers and experimenters (Crosby, 2019).

Campaign-targeted beliefs (TB)

Eight campaign-targeted beliefs were measured across all waves of outcome evaluation to capture youth response to TRC advertisements. Seven beliefs followed the same stem: If I smoke, I will … 1) lose my teeth; 2) have problems with my teeth; 3) get wrinkles; 4) develop skin problems; 5) be controlled by smoking; 6) become addicted; 7) inhale poisons. One additional belief simply stated: 8) Cigarette ingredients are dangerous. These targeted beliefs were identified from a larger set of measured beliefs through systematic coding of their correspondence with the content of campaign advertisements. Intercoder agreement was adequate (overall kappa = .88; individual ad kappa = .72 to 1.0; Duke et al., 2018). Youth respondents indicated their agreement with each statement on a 5-point Likert scale from 1 strongly disagree to 5 strongly agree. For each wave, a summary scale was created by averaging the scores across the eight targeted beliefs. Exploratory factor analysis using iterated principal-factor method consistently revealed a single-factor solution in all waves of data (variance explained = .74-.76). Internal consistency of the scale was adequate for each wave (alpha = .80-.89).

Ad exposure

During each wave of data collection, TRC ads on air during the prior three months (a total of three to five ads per survey) were shown to respondents who then indicated how often they had seen each ad in the past three months. The response options were 0 never, 1 rarely, 2 sometimes, 3 often, and 4 very often. For each wave, the frequency of exposure across all tested ads were added together to construct an overall index of TRC ad exposure.

Perceived effectiveness (PE)

PE was measured using six items (Davis et al., 2013): This ad … 1) grabbed my attention; 2) is worth remembering; 3) is informative; 4) is powerful; 5) is meaningful to me; and 6) is convincing. Ratings were provided on a 5-point Likert scale: 1 strongly disagree to 5 strongly agree. In follow-ups 1 and 2, PE was measured for each ad with self-reported exposure (≥1 on the exposure measure). In follow-ups 3 and 4, due to larger sets of ads tested, PE was measured for two randomly selected ads from the available set for each respondent. This design ensured PE ratings for all ads without overburdening the respondents. For each follow-up, an ad-specific PE score was constructed by averaging the 6 items for each ad. A summary PE score for each respondent at each wave was then derived by averaging the ad-specific PE scores for all ads with PE ratings. This procedure resulted in an overall PE score that was unconfounded by the frequency of ad-specific exposure. Exploratory factor analysis using iterated principal-factor method consistently revealed a one-factor solution on the ad level within each wave (variance explained = .91-.95). Reliability coefficients for the ads were also consistently high within waves (alpha = .91-.96). On the individual level, reliability across ads could only be computed for the first and second follow-ups and it was excellent in both cases (alpha = .89 for both waves). For the third and final follow-ups, because of random selection of ads for PE measurement, respondents did not have PE ratings across all ads. Pairwise correlations between ads, however, indicated strong consistency across all pairs (r = .50-.92).

Awareness of other campaigns

Youth awareness of two additional national tobacco education campaigns – truth® and Tips from Former Smokers – was also measured. Respondents were shown the logo of each campaign and asked: In the past 3 months, have you seen or heard the following slogan or theme: truth®/Tips from Former Smokers? Responses were recorded as 1 yes or 0 no.

Demographics and other covariates

A number of demographic and background characteristics were measured at baseline, including sex, age, race/ethnicity, youth weekly income, presence of tobacco user in the household, and daily amount of television viewing across media devices. The baseline survey also measured a number of correlates of youth risky behaviors based on previous literature (U.S. DHHS, 2014), including sensation seeking, school aspirations, school environment, school performance, and relationship with parents. Additional environmental data from external sources were also added to the survey, including state adult smoking prevalence from the 2013 Behavioral Risk Factor Surveillance System, market median population size, market median income, and media market education level. More details about these control variables are available elsewhere (Duke et al., 2019).

Analysis strategy

Although TRC advertising varied in content and dosage over time, the evaluation survey consistently used the same set of targeted beliefs and PE measure to monitor campaign performance. This design lends itself well to a cross-lagged analysis. We analyzed the data using the general cross-lagged panel model (GCLM) approach (Zyphur, Allison et al., 2019; Zyphur, Voelkle et al., 2019). A key feature of this newly developed approach is that it contains a fixed effect component that controls for all time-invariant factors (Allison et al., 2017). In so doing, it separates stable trends on the between-individual level from within-individual dynamics over time. Research shows that failing to account for such stable factors on the between level can lead to biased estimates of both autoregressive and cross-lagged coefficients in traditional cross-lagged panel analysis (Allison et al., 2017; Hamaker et al., 2015; Leszczensky & Wolbring, 2019).

GCLM is a flexible framework that allows for different specifications in response to the research context (Zyphur, Allison et al., 2019; Zyphur, Voelkle et al., 2019). We compared different specifications, including different lag orders and dynamic effect possibilities, in preliminary analyses. Model selection was guided by both substantive considerations and statistical support. In particular, we considered a model without any lagged effects, assuming all reciprocal effects to be contemporaneous. This model produced significantly worse fit than one with lagged effects (p < .001), justifying examining these relationships over time. We also explored moving average effects which could potentially add to the model’s ability to capture different patterns of short- and long-run effects. Likelihood ratio tests, however, showed no improvement in model fit with these added parameters (p > .05). In light of this finding, also considering the relatively straightforward objectives of the study, we decided to specify a parsimonious model with only first-order autoregressive effects and direct cross-lagged paths between adjacent waves.

The model was tested using structural equation modeling. The core of the model was a series of cross-lagged paths allowing PE/TB at an earlier wave to affect TB/PE at the immediate subsequent wave (see Figure 1). Because PE data were not available at baseline, only a path from TB at baseline to PE at the first follow-up was included. Concurrent correlation, capturing potential contemporaneous mutual influences, between the two variables within each wave was allowed beginning at the first follow-up. First-order autoregressive relationships were included allowing each variable to influence itself at the next wave. All cross-lagged paths from TB to PE were constrained to be equal across waves, as were those from PE to TB. Autoregressive correlations for TB and PE, respectively, were also constrained to be equal over time. The fixed effects component was modeled by first specifying a latent variable for PE and another latent variable for TB. The observed PE and TB variables across all waves were specified to load on their respective latent variables. The latent variables were then allowed to correlate to enable control over all time-invariant factors (Allison et al., 2017; Zyphur, Allison et al., 2019; Zyphur, Voelkle et al., 2019).

Figure 1. Cross-lagged associations between perceived effectiveness and targeted beliefs.

In addition to the autoregressive and cross-lagged paths, the model also included three time-varying factors that could potentially influence PE and TB at any given time point: TRC ad exposure, awareness of Tips, and awareness of truth. At each wave, PE and TB were both specified to be influenced by these factors as measured within the same wave. In the testing of the core model, no additional control variable was included because of the fixed effects approach used. As a form of sensitivity analysis, we later estimated the same model with extensive controls added. According to GCLM, findings from modeling with or without time-invariant control variables should be similar.

Model estimation was performed in Mplus 8.1 (Muthen & Muthen, 2012–2018). Analysis was weighted to adjust for complex sampling design and nonresponse over time. The estimator used was MLR, a maximum likelihood estimator robust to data non-normality and non-independence. Model fit was assessed using chi-squired test, Comparative Fit Index (CFI), Root Mean Square Error of Approximation (RMSEA), and Standardized Root Mean Square Residual (SRMR). Following guidelines in the literature, nonsignificant chi-square, CFI close to .95 or higher, RMSEA close to .06 or lower, and SRMR close to .08 or lower were considered indicators of adequate fit (Hu & Bentler, 1999; Kline, 2015).

As a final methodological note, GCLM is a modeling technique that focuses on within-individual changes over time (Zyphur, Allison et al., 2019; Zyphur, Voelkle et al., 2019). The lagged paths represent the extent to which changes that occur in variable A within each individual at time 1 persist over time (autoregressive effects) or are related to changes in variable B at time 2 (cross-lagged effects). This focus on change, together with a time lag and the separation of between- and within-individual processes, enables cleaner causal inference. It also helps alleviate, to some extent, a major concern about PE measurement, that is, the confound between message differences (external shocks that produce within-individual changes in PE) and individual differences (between-individual variation in PE ratings) (for discussion of this issue, see Noar, Bell et al., 2018; O’Keefe, 2020; Yzer et al., 2015; Zhao et al., 2011).

Results

The current analysis included respondents who were the target audience of TRC (i.e., susceptible nonsmokers and experimenters at baseline) and reported exposure to at least 1 TRC ad at each wave (N = 1,128). Baseline sample characteristics are summarized in Table 1. Descriptive statistics and correlations of PE and TB are presented in Table 2. Descriptive statistics of time-varying TRC exposure variables and their correlations with PE and TB are presented in Table 3.

Table 1. Baseline sample characteristics (N = 1,128)

Characteristics		n (%) or M (SD)
Age
	11	100 (8.9)
	12	145 (12.9)
	13	199 (17.6)
	14	254 (22.5)
	15	237 (21.0)
	16	193 (17.1)
Sex
	Male	580 (51.6)
	Female	546 (48.4)
Race/Ethnicity
	White, Non-Hispanic	501 (44.5)
	Black, Non-Hispanic	89 (7.8)
	Hispanic	411 (36.4)
	Other, Non-Hispanic	127 (11.3)
Smoking Status
	Susceptible nonsmoker	898 (79.6)
	Experimenter	230 (20.4)
Living with a Smoker
	Yes	428 (37.9)
	No	700 (62.1)
Household Ban on Smoking
	Yes	920 (81.6)
	No	160 (14.2)
School Performance (Self-Assessment)
	Above average	672 (59.6)
	Average or below	429 (38.0)
Daily Hours Watching TV Across Media Devices
	0–4	401 (35.7)
	5–9	475 (42.2)
	10+	248 (22.1)
Youth Weekly Income
	None	136 (12.5)
	$1 – $10	285 (26.1)
	$11 – $50	497 (45.6)
	$51+	193 (15.9)
Sensation Seeking (5-point; higher score = higher sensation seeking)		3.35 (.78)

Note. Numbers and percentages do not always add up to 1,128 or 100% due to missing data and rounding.

Table 2. Descriptive statistics and correlations of perceived effectiveness (PE) and campaign-targeted beliefs (TB) across all waves

	TB_w1	TB_w2	TB_w3	TB_w4	TB_w5	PE_w2	PE_w3	PE_w4	PE_W5
TB_w1	1.00
TB_w2	0.47	1.00
TB_w3	0.36	0.44	1.00
TB_w4	0.28	0.39	0.44	1.00
TB_w5	0.30	0.35	0.41	0.44	1.00
PE_w2	0.26	0.30	0.22	0.19	0.22	1.00
PE_w3	0.24	0.27	0.35	0.24	0.29	0.47	1.00
PE_w4	0.12	0.15	0.19	0.32	0.24	0.28	0.40	1.00
PE_W5	0.20	0.18	0.23	0.25	0.38	0.35	0.39	0.33	1.00
M	4.00	4.18	4.21	4.26	4.30	3.95	3.99	3.88	3.93
SD	.68	.74	.75	.74	.73	.73	.75	.88	.85

Note. W1 = baseline; w2-w5 = first to fourth follow-up.

Table 3. Descriptive statistics of exposure to The Real Cost (TRC) and their correlations with perceived effectiveness (PE) and campaign-targeted beliefs (TB)

	TRC_w2	TRC_w3	TRC_w4	TRC_w5
TB_w1	0.07	0.07	0.05	0.05
TB_w2	0.08	0.02	0.04	0.02
TB_w3	0.03	0.07	0.04	0.06
TB_w4	0.04	−0.01	0.00	0.06
TB_w5	0.05	0.07	0.08	0.10
PE_w2	0.22	0.16	0.20	0.17
PE_w3	0.09	0.16	0.16	0.15
PE_w4	0.04	0.07	0.08	0.09
PE_w5	0.09	0.12	0.16	0.19
M	9.62	9.26	17.25	8.82
SD	4.28	3.82	7.83	3.88

Note. W1 = baseline; w2-w5 = first to fourth follow-up.

Estimation of the core model returned satisfactory fit: χ² (df = 90) = 100.80, p = .205; CFI = .99; RMSEA = .010 (90%CI = .000-.020); SRMR = .023. Key relationships in the cross-lagged model are presented in Figure 1. Note that, although the cross-lagged paths and autoregressive correlations were respectively constrained to be equal over time, their standardized coefficients varied slightly across waves as a result of standardization. As shown, PE was a significant predictor of TB at the subsequent wave (p = .003). Conversely, TB was also predictive of PE at the subsequent wave (p = .006). The latent variables for both TB and PE showed factor loadings of moderate strength, suggesting consistent influence of stable factors in both variables over time. Additionally, the correlation between the two latent variables was moderate (r = .31, p = .018), indicating that time-invariant factors were able to explain a modest amount of the covariance between PE and TB. Net of the fixed effects, the autoregressive correlation for TB was still positive and significant (p = .003), suggesting that the changes in TB were able to persist over time. On the other hand, the autoregressive correlation for PE was not significant (p = .671), indicating that wave-specific fluctuations in PE were relatively transient and did not carry over to later points in time. The PE-TB correlation within each wave was consistently positive and significant (all ps < .01).

For the sake of readability,Figure 1 did not include paths from the three time-varying variables in each wave: TRC ad exposure, awareness of truth, and awareness of Tips. Only a few paths involving these variables emerged significant: awareness of Tips to TB at baseline (b = .15, p = .016, β = .12); TRC ad exposure to PE at first follow-up (b = .03, p = .001, β = .21); awareness of truth to PE at first follow-up (b = .16, p = .004, β = .14); and awareness of truth to PE at the third follow-up (b = .28, p = .018, β = .14).

In auxiliary analysis, an additional model was estimated that controlled for the full set of individual and market/state-level background variables as described in the measures section (demographics and other covariates). These control variables were allowed to influence the latent variables of PE and TB (Zyphur, Allison et al., 2019). This expanded model resulted in substantial loss of sample size (remaining N = 806). However, the general pattern of results was similar to that reported above. In particular, both the lagged effect of PE on TB (b = .11, p = .046, β = .08-.14) and the lagged effect of TB on PE (b = .09, p = .007, β = .09-.11) remained significant.

Discussion

This study had two aims. On the more conceptual side, it proposed and empirically tested the possibility that PE and AE may mutually influence each other in an evolving campaign context. On the more practical side, it sought to obtain validation evidence on PE’s ability to predict AE for youth tobacco prevention efforts. On both fronts, the findings of the study were confirmative. Using five-wave evaluation data from a national youth tobacco education campaign, this study showed that PE prospectively predicted campaign-targeted beliefs over time, and vice versa. In the GCLM framework, cross-lagged paths represent within-individual dynamics (Zyphur, Allison et al., 2019; Zyphur, Voelkle et al., 2019). The significant path from PE to TB, thus, indicated that whatever changes occurred in PE at an earlier wave for any given individual was able to lead to corresponding changes in TB for the same individual at a subsequent wave. The same was also true with the path from TB to PE. These results provided useful new evidence to inform and advance understanding of PE and its relevance in campaign research and practice.

Despite its wide use in persuasion and campaign-related research, PE has been criticized for its conceptual ambiguity, heterogeneity in measurement, and murkiness in its relationship with actual persuasive outcomes (Dillard & Ye, 2008; Noar, Barker, & Yzer, 2018; O’Keefe, 2018a; Yzer et al., 2015). While fully addressing these issues is beyond the scope of the current study (or any single study), its findings do shed light on a key underlying concern, that is, the causal relationship between PE and AE. Previous research has often either simply assumed that PE is causally antecedent to AE or pitched the two causal directions (i.e., PE → AE, or AE → PE) as competing hypotheses (Dillard, Shen et al., 2007; Dillard, Weber et al., 2007). The possibility of mutual influence was sometimes noted but rarely considered as a theoretically viable and informative perspective in its own right. Recent developments in communication theory, however, have increasingly acknowledged the inherent reciprocity in the relationships embedded in the processes and effects of mediated communication (Eveland, 2001; Shah et al. 2017; Slater, 2007, 2015). Under these more contemporary frameworks, PE, as a form of message-oriented cognitive response, can both influence and be influenced by the targeted beliefs, attitude and behavioral outcomes of the communication messages. Such dynamics afford a more nuanced understanding of the role of PE in the persuasion process and are clearly supported by the current evidence.

The explicit acknowledgment of mutual influence between PE an AE does not mean that the examination and consideration of unidirectional influence are misguided. In fact, the second goal of the current study is squarely focused on the unidirectional impact of PE on AE, as this effect has the most direct implications for campaign research. The current evidence on a reciprocal relationship, however, does urge greater caution in the examination of unidirectional effects. In particular, cross-sectional associations should be interpreted with care as such relationships are likely to include influences in both directions. Depending on the nature and magnitude of the two influences, the overall relationship between PE and AE may or may not necessarily reflect the causal effect of PE on AE. Although this point seems simple enough, its importance becomes apparent when we consider the fact that many message testing studies measure PE as part of the post-exposure outcome questionnaire. As such, the evidence they generate on the relationship between PE and AE is still cross-sectional in nature and offers no firm ground to infer causality. It is important to note, however, that this criticism of single-shot message testing experiments pertains only to their ability to produce validation evidence for PE. It does not mean to suggest that such experiments and PE are not useful for assessing message potential. In fact, the effect of reverse causation is likely to bias PE ratings consistently across messages within individuals. The relative standing of messages on PE, thus, can still convey important information on their potential ability to persuade in comparative terms.

With this in mind, we view the lagged association between PE and AE unveiled in this study as more robust and clearer evidence for the causal effect between the two variables. This finding is consistent with earlier research that has used longitudinal designs to show that PE could predict tobacco use intention and behavior among adult smokers (Noar et al., 2020). The current study is the first to show that PE is also able to predict campaign-targeted belief change among a youth population where the messages are focused on the prevention of cigarette smoking rather than cessation. The TRC campaign has been using PE in both its formative and outcome evaluation research since its inception. Our finding offers assurance that using PE as a campaign research tool, both for message pretesting and continued monitoring of ad performance in the market, is justifiable in this campaign. It also suggests that the specific measure of PE tested in this study (Davis et al., 2013) has broader utility beyond its initial context of application. PE as a diagnostic tool for message development is well noted and a focus of much ongoing research (O’Keefe, 2018a). The current findings on the longitudinal dynamics between PE and AE suggests that PE is also an important factor in the complex processes that drive campaign effects over time. This latter perspective is well aligned with several research traditions in persuasion (e.g., cognitive response theory; Greenwald, 1968) and marketing (e.g., attitude toward the ad; Shimp, 1981). Future research and campaign practice should be more mindful of PE’s utility on this front.

This study used GCLM, a newly developed modeling approach, to conduct its cross-lagged analysis. This approach offers several advantages over traditional analytical methods (Zyphur, Allison et al., 2019; Zyphur, Voelkle et al., 2019). In particular, GCLM includes a fixed effect component that controls for all time-invariant influences. This enabled more accurate estimation of model parameters (Allison et al., 2017; Hamaker et al., 2015; Leszczensky & Wolbring, 2019). Its focus on within-individual change also sharpens the interpretation of both cross-lagged and autoregressive paths because they are unconfounded with between-individual differences. Wider application of this modeling technique to address other dynamic processes in health campaigns and mediated communication may be worth consideration.

The landscape of tobacco control is exceedingly complex. Intervention efforts often coexist on multiple levels (national, regional, and local) and evolve independently over time. To evaluate any single campaign, it is often necessary and important to monitor and account for exposure to other concurrent campaigns. Our analysis included several time-varying covariates which captured wave-specific exposure to both TRC and two other major national campaigns. Few paths involving these covariates emerged significant, however, suggesting limited covariation between these exposure variables and PE and TB in the current data. It should be noted that the current analysis was restricted to only those who reported exposure to TRC advertising at each wave of data collection. This sample restriction eliminated a critical source of variation in the exposure variable – that between those with and without exposure at any given wave. It also resulted in a smaller sample, thus reduced power, for the current analysis. Both of these factors may have contributed to the lack of associations between campaign exposure and other variables in this analysis.

In addition to a restricted sample, a few other limitations of this study should be acknowledged. First, the current analysis examined only campaign-targeted beliefs, not actual smoking initiation behavior. As a one-off behavior, smoking initiation cannot be properly modeled in a cross-lagged analysis. Other modeling techniques, such as survival analysis, are needed to appropriately examine the relationship between PE and smoking initiation in TRC and other youth tobacco prevention contexts. Second, the current analysis included three time-varying predictors to account for exposure to TRC and other tobacco education campaigns that changed over time. Other time-varying factors might exist that had influenced the relationships observed in this study. Measurement and incorporation of additional time-varying confounders could further enhance confidence in the current results. Third, the current study focused on general receptivity to TRC advertising and did not examine PE with respect to specific campaign themes that have varied over time. This may be an interesting direction for future research. Fourth, the time lag used in the current analysis was about 8 months. Whether this time lag is most conducive to revealing the dynamics between PE and AE is an open question. Other time lags may reveal different patterns of mutual influence, or complete lack of it. These possibilities are interesting avenues for future research. It should be noted that campaign effects are almost always accumulative in nature, building on repeated exposure to the same core messages over time (Hornik, 2002). With this in mind, relatively lengthy time lags, such as the one used in this study, should have advantages in capturing long-term campaign impact that will only materialize after sufficient time has lapsed (CDC, 2018).

In conclusion, this study has advanced understanding of PE by demonstrating clear reciprocity in the longitudinal relationship between PE and AE in a real-life campaign context. It also reassures campaign researchers and practitioners that PE is a valid and useful instrument in tobacco prevention efforts targeted at youth. Continued use of and research on PE in TRC and other campaign settings appear warranted.

Disclaimer

This publication represents the views of the authors and does not represent FDA/CTP position or policy.

Additional information

Funding

This study was funded by the U.S. Food and Drug Administration (FDA) Center for Tobacco Products (CTP) [contract HHSF223201310001B awarded to RTI International].

Notes

1. The current evidence on PE has also been challenged from another angle (O’Keefe, 2020). In a recent commentary, O’Keefe (2020) argues that individual-level relationship between PE and AE is irrelevant to the question of whether PE can identify relatively effective or ineffective messages – only message-level correlation between PE and AE can serve that purpose. This observation is important to note because it reminds us that, as a subjective rating measure, PE is necessarily confounded with individual differences (Noar, Bell et al., 2018; Yzer et al., 2015; Zhao et al., 2011). In other words, PE ratings include variation from both the individuals and the messages under investigation. Our confidence in PE’s ability to detect message-level differences is thus partly dependent on our ability to model or partial out the influence of individual (between-subject) differences through proper design and analysis. In our analysis, we ensured proper control of individual differences through a fixed effect component in the model (see analysis strategy for more information). It should also be noted that message-level analysis is often unfeasible in campaign research. Even well-funded large campaigns may only be able to test a few messages at a time (e.g., Zhao et al., 2016, 2019). Restricting testing to the message level will almost always result in analysis with extremely small sample size, causing its own problems of uncertainty and murkiness in study findings. While combining data across multiple waves of testing or multiple campaigns may improve power, the need to make timely message decisions in any given campaign can rarely take advantage of such pooled analysis.