Examining the Longitudinal Relationship Between Perceived and Actual Message Effectiveness: A Randomized Trial

ABSTRACT

We sought to examine the relationship between perceived message effectiveness (PME) and actual message effectiveness (AME) in a 3-week randomized trial of vaping prevention advertisements. Participants were US adolescents (n = 1,514) recruited in 2021. We randomly assigned them to view The Real Cost vaping prevention ads or control videos online. Participants viewed three videos at Visit 1, again at Visits 2 and 3, and completed a survey at each visit that assessed AME (susceptibility to vaping) and two types of PME – effects perceptions (potential for behavioral impact) and message perceptions (potential for message processing). At Visit 4, AME was measured. Compared to control, The Real Cost ads led to improved AME (lower susceptibility to vaping at Visit 4, p < .001). This was anticipated by The Real Cost ads eliciting higher PME ratings (higher effects and message perceptions at Visit 1, both p < .001). Furthermore, PME (both effects and message perceptions) at Visit 1 predicted susceptibility to vaping at Visits 1, 2, 3, and 4 (all p < .001). Finally, effects perceptions fully mediated the impact of The Real Cost ads on susceptibility to vaping (β = −.30; p < .001), while message perceptions only partially mediated the effect (β = −.04; p = .001). Our findings indicate a relationship between PME and AME, especially effects perceptions, and suggest that PME may be useful in message pre-testing to select messages with greater behavior change potential.

Perceived message effectiveness (PME) refers to an “estimate of the degree to which a persuasive message will be favorably evaluated by recipients of that message” (Dillard & Ye, 2008, p. 150). More than a decade of research on PME suggests that it may be a valuable proxy for attitude or behavior change elicited by a message (Brennan et al., 2014; Cappella, 2018; Davis, Nonnemaker, Farrelly, et al., 2011; Dillard, Shen, et al., 2007; Lavine & Snyder, 1996; Noar, Bell, et al., 2018; Rohde et al., 2021). PME is often used in formative research to select campaign messages (Dillard, Shen, et al., 2007), with some scholars arguing that PME ratings are necessary for message selection (Fishbein et al., 2002). Although PME is widely used, there are some criticisms of both the theoretical and empirical underpinnings of PME, particularly its relationship to actual message effectiveness (AME). While some theory and research support the predictiveness of PME on a series of biobehavioral and behavioral outcomes (Baig et al., 2021a; Cappella, 2018; Davis & Duke, 2018; Kang et al., 2009; Weber et al., 2015), other theoretical arguments and empirical data appear to refute the predictiveness of PME (O’Keefe, 2018, 2020; Pechmann et al., 2003).

The debate around the PME-AME relationship may, in part, be attributed to the heterogeneity of PME measures (Brennan et al., 2014; Cappella, 2018; Dillard, Shen, et al., 2007; Noar, Barker, et al., 2018; Noar, Bell, et al., 2018; O’Keefe, 2018; Pechmann et al., 2003; Yzer et al., 2015). Noar, Bell, et al. (2018) systematic review found wide variation in PME measures, including message recipients’ perceptions of cognitive elaboration, message credibility, argument strength, likability, and personal relevance as PME measures, among many other constructs. The diverse PME measures in the literature can generally be categorized into message perceptions or effects perceptions. Message perceptions are evaluations of a message in terms of its ability to promote further processing that is related to persuasion (Baig et al., 2021b), similar to Dillard and Ye’s (2008) conceptualization of message attributes. An example message perceptions PME item is “This message grabs my attention” (Davis, Nonnemaker, Duke, et al., 2013). Effects perceptions, on the other hand, refer to the potential of a message to change key precursors of behavior or behavior itself (Baig et al., 2021b), similar to Dillard and Ye’s (2008) conceptualization of message impact. An example of effects perception PME item is “This message discourages me from vaping” (Noar, Gottfredson, Vereen, et al., 2021).

While these two conceptualizations of PME (i.e., message and effects perceptions) are somewhat distinct, they are often used interchangeably or combined in single PME measures (Dillard & Ye, 2008; Noar, Bell, et al., 2018; O’Keefe, 2018), limiting our understanding of the similarities or differences between them. Only a modest number of studies have compared message and effects perceptions (Brennan et al., 2014; Noar, Rohde, et al., 2020; Rohde et al., 2021), with no studies doing so in the context of a randomized controlled trial (RCT) with longitudinal, repeated ad exposures. Next, we discuss extant approaches to studying the relationship between PME and AME, and then we describe the current RCT.

PME and AME

Researchers have used several approaches to examine the relationship between PME and AME (Bigsby et al., 2013; Davis, Nonnemaker, Duke, et al., 2013; Dillard, Weber, et al., 2007). One approach is experimental correspondence, which examines the direction of the effects of message conditions on PME and AME in experimental studies (Noar, Rohde, et al., 2020; Rohde et al., 2021). For instance, Noar, Rohde, et al. (2020) examined the effect of ads from The Real Cost vaping prevention campaign, a Food and Drug Administration (FDA)-funded communication campaign designed for youth tobacco prevention, on a US adolescent sample. They found that, compared to control videos, The Real Cost ads were rated higher on PME (both effects and message perceptions) and had better AME effects (i.e., more negative attitudes toward vaping, higher beliefs about the harms of vaping, and lower intentions to vape). In another experiment, Rayala et al. (2022) found that environment- and health-focused messages advocating for lower meat consumption were rated higher on PME and also led to greater intentions to reduce meat consumption compared to control messages. These studies show experimental correspondence between the direction of the effect of message conditions on PME and AME. While experiments have tended to apply this design at a single time point, studies could also apply similar methods to longitudinal experiments and evaluate PME’s ability to prospectively anticipate AME (compared to control messages).

A second approach is prediction, which examines the association between PME and AME (Alvaro et al., 2013; Davis & Duke, 2018; Dillard, Shen, et al., 2007). In a longitudinal study, Brennan et al. (2014) found that perceived personalized effectiveness (similar to effects perceptions) of anti-smoking messages predicted smoking behavior at a 3-week follow-up, whereas ad-directed effectiveness (similar to message perceptions) was not associated with behavior change at follow-up. In another study, Davis et al. (2017) examined the validity of PME ratings of Tips From Former Smokers campaign messages at baseline in predicting smoking quit attempts at follow-up and found that higher PME (message perceptions) at baseline was positively associated with odds of making a quit attempt at follow-up. In addition, a meta-analysis of the correlation between PME and message attitudes revealed associations across diverse topics (Dillard, Weber, et al., 2007), while another on tobacco education messages found longitudinal associations between PME ratings and message recall, beliefs about smoking, and intentions to quit smoking (Noar, Barker, et al., 2020). While providing much evidence for a PME-AME relationship, correlational analyses alone lack the ability to support a causal link between PME and AME (O’Keefe, 2020).

A third approach is mediation, which examines the role of PME as a mediator between messages and outcome measures in experiments or RCTs (Baig et al., 2021b; Brewer et al., 2019; Popova & Li, 2022). For example, Baig et al. (2021b) found that effects perceptions mediated the impact of messages about toxic chemicals in cigarettes on butting out, forgoing, and attempts to quit smoking cigarettes; however, message perceptions did not mediate the effects of message conditions on any of these behavioral outcomes. In another study, Popova and Li (2022) showed that PME (measured as negative message perceptions) mediated the relationship between emotions elicited by e-cigarette prevention messages and intentions to use e-cigarettes. Dillard and Peck (2000) proposed that PME should mediate the effect of public service announcements on attitudes toward an object, which should then affect intentions and behavior. To date, only a modest number of studies have tested PME as a mediator of message effects.

Given the strengths and weaknesses of different approaches to examine the relationship between PME and AME, rigorously designed studies that use multiple approaches are needed. In addition, studies that use robust PME measures of both effects and message perceptions, a longitudinal design, multiple exposures to messages, and control conditions are necessary to advance our understanding of the nature of different types of PME and how they relate to AME.

Current trial

The current trial examined the relationship between PME and AME, using two types of PME – effects and message perceptions – in a 3-week longitudinal, repeated exposure, RCT with US adolescents who were at risk of becoming addicted to nicotine (i.e., were susceptible to vaping or currently vaped). Specifically, we tested the PME and AME relationship using the three approaches discussed above: experimental correspondence, prediction, and mediation. We sought to examine the extent to which both effects and message perceptions (two types of PME) relate to the AME of vaping prevention advertisements. Based on prior PME research, we posed the following hypotheses:

H1:

Viewing vaping prevention ads (vs. control) leads to higher perceived message effectiveness (PME) scores and lower susceptibility to vaping. (Experimental correspondence)

H2:

Higher PME is associated with lower susceptibility to vaping. (Prediction)

 H2a:

Effects perceptions are more strongly associated with susceptibility to vaping than message perceptions.

H3:

PME mediates the impact of vaping prevention ads on susceptibility to vaping. (Mediation)

 H3a:

Effects perceptions more strongly mediate the impact of vaping prevention ads on susceptibility to vaping than message perceptions.

Methods

Participants

Adolescents aged 13 through 17 years (n = 1,708) living in the US who were susceptible to e-cigarette use were recruited from online panels administered by Qualtrics. Adolescents were considered susceptible if they answered anything other than “definitely not” to any of 5 questions about their openness to vaping (Strong et al., 2015). Among the sample screened, 151 declined to participate and 43 were unable to enroll since the trial quota had been met, resulting in a total of 1,514 valid responses (see Table 1 for participant characteristics). The target sample size in this trial was 1500, accounting for up to 33% of participants dropping out during the trial. With an estimated intraclass correlation of 0.70, the trial had power to detect an effect size of d = 0.25 or larger between (combined) intervention groups and the control group.

Table 1. Participant characteristics (N = 1,514).

		Both Real Cost Arms	Control Arm
	Overall (%)	(n = 1010) (%)	(n = 504)
Age, years (M, SD)	15.2 (1.2)	15.2 (1.2)	15.3 (1.2)
Gender identity
Male	1140 (75)	747 (74)	393 (78)
Female	358 (24)	248 (25)	110 (22)
Other responses	15 (1)	15 (2)	1 (<1)
Race
White	1081 (71)	712 (71)	369 (73)
Black or African-American	371 (25)	253 (25)	118 (23)
Asian	14 (1)	8 (1)	6 (1)
Native Hawaiian or other Pacific Islander	4 (<1)	4 (<1)	0 (0)
Multiracial	33 (2)	25 (3)	8 (2)
Other*	10 (<1)	7 (<1)	3 (<1)
Missing	1 (<1)	1 (<1)	0 (0)
Hispanic	176 (12)	115 (11)	61 (12)
Sexual orientation
Heterosexual	1432 (95)	947 (94)	485 (96)
Lesbian, gay, bisexual, pansexual, or queer	61 (4)	48 (5)	13 (3)
Prefer not to say/Missing	20 (1)	15 (2)	6 (1)
Adolescent’s education
<High school	233 (15)	157 (16)	76 (15)
Some high school	1067 (71)	715 (71)	352 (70)
High school or GED	123 (8)	79 (8)	44 (9)
Some college	85 (6)	55 (5)	30 (6)
Dropped out of school	5 (<1)	3 (<1)	2 (<1)
Missing	1 (<1)	1 (<1)	0 (0)
Mother’s education
<Bachelor’s degree	262 (17)	187 (19)	75 (15)
Bachelor’s degree	533 (35)	352 (35)	181 (36)
Master’s degree	578 (38)	378 (37)	200 (40)
Doctorate degree	123 (8)	82 (8)	41 (8)
Missing	18 (1)	11 (1)	7 (1)
Father’s education
<Bachelor’s degree	169 (11)	122 (12)	47 (9)
Bachelor’s degree	353 (23)	235 (23)	118 (23)
Master’s degree	699 (46)	463 (46)	236 (47)
Doctorate degree	273 (18)	176 (17)	97 (19)
Missing	20 (1)	14 (1)	6 (1)
Lived with somebody who …
Smoked cigarettes	539 (36)	340 (34)	199 (40)
Used e-cigarettes or vapes	443 (29)	279 (28)	164 (33)
Used chewing tobacco, snuff, or dip	218 (14)	145 (14)	73 (15)
Smoked cigars, cigarillos, or little cigars	203 (13)	126 (13)	77 (15)
Used another form of tobacco	119 (8)	82 (8)	37 (7)
Tobacco use (past 30 days)
Used e-cigarette	922 (61)	612 (61)	310 (62)
Used cigarette	825 (55)	539 (53)	286 (57)
Used OTP	871 (58)	572 (57)	299 (59)

SD = standard deviation; GED = general educational diploma; OTP = other tobacco product.

*Selected “other” option for race.

Procedures

We conducted an RCT with three arms and parallel assignment (see Figure 1). Adolescents participated in 4 weekly online visits over 3 weeks and completed an online survey at each visit. At Visits 1, 2, and 3, adolescents viewed campaign ads or control videos. In the first week (Visit 1), participants reported vaping behavior then viewed three ads in random order that corresponded to their trial arm: FDA’s The Real Cost vaping prevention health harms ads, FDA’s The Real Cost vaping prevention addiction ads, or control videos. Participants assigned to the health harms arm viewed three ads that emphasized the health harms of vaping, such as lung damage. Participants in the addiction arm viewed three ads that focused on the consequences of nicotine addiction from vaping, such as loss of autonomy. Finally, participants in the control arm viewed three investigator-created neutral videos about vaping consisting of black text on a white screen with a narrator reading the text. These videos focused on neutral content about e-cigarettes and vaping, including vaping product definitions, farming practices related to nicotine used in vapes, and manufacturing practices of vaping devices adapted from Wikipedia and other sources. Each ad across all three experimental conditions was approximately 30 seconds long (see Supplementary Table S2 for both The Real Cost and control ads used in this trial).

Figure 1. Flow diagram.

Sample sizes for allocation to the trial arm are from Visit 1, and for trial completion, are from Visit 4. For Visits 2 and 3, 453 and 450 adolescents were in the Health Harms arm, 457 and 461 in the Addiction arm, and 461 and 466 in the Control arm, respectively.

Tobacco use was assessed at all visits before showing ads. At Visit 1, after viewing each ad, participants completed measures of PME (effects perceptions followed by message perceptions), susceptibility to vaping, and demographics. For the subsequent weeks of the trial (Visits 2 and 3), participants viewed the same three ads from their trial arm and completed surveys with the same measures of PME and susceptibility to vaping. At Visit 4, participants completed surveys assessing susceptibility to vaping, but did not view ads or answer questions about PME. Across Visits 1–4, the surveys also included other measures, such as addiction and health harm risk beliefs, which are reported elsewhere (Noar, Gottfredson, Kieu, et al., 2022).

Before starting the surveys, parents or legal guardians provided consent online and adolescents provided assent online. The RCT was approved by the University Institutional Review Board and was registered at clinicaltrials.gov, identifier NCT04836455. The main trial outcomes were reported elsewhere (Noar, Gottfredson, Kieu, et al., 2022). The PME analyses reported here were pre-registered at aspredicted.org/w6ar5.pdf.

Measures

Susceptibility to vaping

We measured susceptibility to vaping with a 3-item scale (Pierce et al., 1996). This scale assessed the extent to which adolescents are open to vaping. The questions are as follows: “Do you think you might use an e-cigarette or vape soon?” “Do you think you might use an e-cigarette or vape in the next year?” and “If one of your best friends were to offer you an e-cigarette or vape, would you use it?” The 4-point response scale ranged from “Definitely not” (coded as 1) to “Definitely yes” (4). We calculated a susceptibility score for each visit by averaging the three items, with higher scores representing higher susceptibility (α = .93, .93, .93, and .95 for Visits 1–4, respectively).

Effects perceptions

A brief 3-item scale assessed effects perceptions (Noar, Gottfredson, Vereen, et al., 2021). This measure was developed using systematic scale development procedures, including cognitive interviews with youth (Kurtzman et al., 2022). The items are as follows: “How much does this ad make you … :” “worry about what vaping will do to you?” “make you think vaping is a bad idea?” and “discourage you from vaping?” The 5-point response scale ranged from “Not at all” (1) to “A great deal” (5). We calculated an effects perceptions score by averaging the three items, with higher scores representing higher effects perceptions (α = .92, .91, and .92 for Visits 1–3, respectively).

Message perceptions

A 6-item scale assessed message perceptions (Davis, Nonnemaker, Duke, et al., 2013). This measure is widely used in FDA message testing studies for youth tobacco prevention ads (Duke et al., 2015; Zhao et al., 2022). The items are as follows: “This ad … ” “grabs my attention,” “is meaningful,” “is informative,” “is convincing,” “is worth remembering,” and “is powerful.” The 5-point response scale ranged from “Strongly disagree” (1) to “Strongly agree” (5). We calculated a message perceptions score by averaging the six items, with higher scores representing higher message perceptions (α = .89, .90, and .91 for Visits 1–3, respectively).

We also collected data about participants’ demographic characteristics (i.e., age, gender, race, ethnicity, and parental education). Together with trial arm and prior ad exposure, these demographics were treated as covariates in prediction and mediation analyses, as specified in the trial pre-registration.

Data analysis

To evaluate experimental correspondence (H1), we first conducted independent between-subjects t-tests to compare the two The Real Cost trial arms (health harms and addiction) on PME at Visit 1 (baseline) and susceptibility to vaping at Visit 4. We also combined the two The Real Cost arms and conducted t-tests to compare The Real Cost trial arms (combined) vs. control on PME at Visit 1 and susceptibility to vaping at Visit 4. We compared the sign of the effect and pattern of statistical significance between PME and susceptibility to vaping. We also reported standardized effect sizes using Cohen’s d. Given that we found no significant difference between The Real Cost trial arms (health harms and addiction) on susceptibility to vaping at Visit 4 (Supplementary Table S1), we combined the two treatment arms together for the prediction and mediation analyses.

To evaluate prediction (H2 & H2a), we used a linear mixed model to estimate the association of PME at Visit 1 with susceptibility to vaping at Visits 1, 2, 3, and 4, controlling for trial arm, prior ad exposure, and demographics (i.e., age, gender, race, ethnicity, and parental education). We coded Visit as a nominal variable (Visit 1 was the reference group) to avoid assuming linear change since we saw evidence of nonlinear change in descriptive analyses; we compared the fit of this model to a model treating time as a linear effect measured in visits. We tested Visit-by-PME interactions to determine whether the effect of PME on susceptibility changed over time. We also compared the association between effects perceptions and susceptibility to vaping with the association between message perceptions and susceptibility to vaping.

The linear mixed model approach increased statistical power and allowed us to model changes in susceptibility to vaping across the trial period. The model accounts for non-independence of repeated measures by treating individual differences as a random effect. We evaluated model fit using Akaike information criterion (AIC) and Schwarz’s Bayesian information criterion (BIC); smaller AIC and BIC values indicate a better model fit. We compared the standardized effects and 95% confidence intervals for the impacts of effects perceptions and message perceptions on susceptibility to vaping. We also calculated multilevel fixed effect R² for each model compared to the null model (with only fixed and random effect intercepts) and the change of fixed effect R² at the person level (level 2) for each model compared to the base model (with fixed and random effect intercepts, the linear effect of time, and covariates) (Snijders & Bosker, 2011).

To evaluate mediation (H3 & H3a), we examined whether PME at Visit 1¹ mediated the impact of the trial arm on susceptibility to vaping at Visits 2, 3, and 4, adjusting for all covariates used in testing H2 to account for the potential for time-varying confounding (VanderWeele, 2015). In these mediation analyses, trial arm and PME were person-level predictors, and susceptibility to vaping was a time-varying outcome. Path coefficients were standardized for comparison. We conducted mediation modeling for effects and message perceptions individually to examine how each measure by itself mediated the trial arm-susceptibility relationship. We also compared the strength of the indirect effect (i.e., the % of the mediated effect) for effects perceptions with the strength of the indirect effect for message perceptions. The mediation analyses used the structural equation modeling R package lavaan (Rosseel, 2012).

Results

Experimental correspondence (H1)

The Real Cost vaping prevention health harms ads led to higher perceived message effectiveness ratings than the addiction ads (Supplementary Table S1). Participants reported higher effects perceptions (M = 3.99 vs. M = 3.80; p < .001, d = 0.21) and higher message perceptions (M = 4.32 vs. M = 4.23; p = .031, d = 0.14) at Visit 1 after viewing the health harm ads compared to the addiction ads. While health harms ads led to lower susceptibility to vaping at Visit 4 than the addiction ads, this difference was not statistically significant (M = 2.16 vs. M = 2.21; p = .445, d = − 0.05).

The Real Cost vaping prevention ads (combined) led to higher perceived and actual message effectiveness (Figure 2). Participants reported higher effects perceptions (M = 3.89 vs. M = 2.91; p < .001, d = 0.89) and higher message perceptions (M = 4.28 vs. M = 4.03; p < .001, d = 0.34) at Visit 1 after viewing The Real Cost ads compared to control ads. The same pattern held, in terms of the direction of effect and pattern of statistical significance, for susceptibility to vaping at Visit 4 (M = 2.18 vs. M = 2.36; p = .003, d = − 0.17).

Figure 2. Experimental comparison of two types of perceived message effectiveness at Visit 1 and susceptibility to vaping at Visit 4.

**p < .01, ***p < .001.

Numbers show means and standard errors.

Prediction (H2 & H2a)

Effects perceptions at Visit 1 were negatively associated with susceptibility to vaping across visits (β = −.38, 95% CI = [−.42, −.33], p < .001), as expected. The same was true for message perceptions (β = −.17, 95% CI = [−.24, −.10], p < .001). Effects perceptions were more strongly related to susceptibility to vaping than message perceptions, p < .001. Both effects perceptions and message perceptions models had a better model fit than the base model (i.e., smaller AIC and BIC values). Compared to the null model, the effects perceptions model explained 25% of the between-subjects variance, whereas the message perceptions model explained 14%. Compared to the base model, the model that included effects perceptions and visit-by-effects perceptions interactions explained an additional 14% of the between-subjects variance not explained in the base model, whereas the model that included message perceptions and the visit-by-message perceptions interactions explained an additional 2% (see Table 2).

Table 2. Perceived message effectiveness predicting susceptibility to vaping over time.

	Base Model			Model 1			Model 2
	B	95% CI	p	B	95% CI	p	B	95% CI	p
Fixed effects
Intercept	1.16	.56, 1.75	<.001	2.24	1.67, 2.81	<.001	1.81	1.16, 2.46	<.001
Visit 2	.09	.05, .12	<.001	−.34	−.47, −.22	<.001	−.13	−.37, .10	.270
Visit 3	−.09	−.12, −.05	<.001	−.49	−.62, −.37	<.001	−.33	−.57, −.10	.006
Visit 4	−.29	−.32, −.25	<.001	−.68	−.81, −.56	<.001	−.64	−.87, −.40	<.001
Effects perceptions				−.38	−.42, −.33	<.001
Visit 2 * Effects perceptions				.12	.09, .15	<.001
Visit 3 * Effects perceptions				.11	.08, .15	<.001
Visit 4 * Effects perceptions				.11	.08, .14	<.001
Message perceptions							−.17	−.24, −.10	<.001
Visit 2 * Message perceptions							.05	−.003, .11	.066
Visit 3 * Message perceptions							.06	.003, .11	.038
Visit 4 * Message perceptions							.08	.03, .14	.003
Random effects
Intercept variance	.63			.54			.62
Residual variance	.27			.27			.27
Model fit
Message arm level 2 R^2	.13			.25			.14
Message arm Δ level 2 R^2				.14			.02

Analyses adjusted for prior ad exposure, Visit 1 trial arm, gender, age, race/ethnicity, and parental education; CI = confidence interval; B = standardized coefficient; SD = standard deviation.

Mediation (H3 & H3a)

Effects perceptions fully mediated the impact of The Real Cost ads (vs. control) on susceptibility to vaping, but message perceptions only partially mediated the effect with a much smaller effect size, p < .001 (Figure 3). The total effect of the trial arm on susceptibility to vaping was β = −.28 (SE = .05, p < .001). For the effects perceptions mediation model, the residual direct effect of the trial arm on susceptibility to vaping was not statistically significant β = .02 (SE = .05, p = .65), with a significant indirect (mediation) effect of the trial arm on susceptibility to vaping through effects perceptions of β = −.30 (SE = .03, p < .001; 100% mediated). For the message perceptions mediation model, the residual direct effect of the trial arm on susceptibility to vaping was statistically significant β = −.24 (SE = .05, p < .001), with a significant indirect (mediation) effect of the trial arm on susceptibility through message perceptions of β = −.04 (SE = .01, p = .001; 17% mediated).

Figure 3. Effects perceptions and message perceptions as mediators of the impact of The Real Cost ads (versus control) on susceptibility to vaping.

***p < .001.

These diagrams depict the 2-2-1 multilevel mediation model with path coefficients and standard errors (in parentheses). The effects perceptions mediation model is above the dashed line, and the message perceptions model is below the dashed line.

Discussion

The current randomized trial examined the relationship of PME – both effects and message perceptions – with AME in a large sample of US adolescents. Using data from a large RCT examining the impact of vaping prevention ads on susceptibility to vaping, we found that (1) differences between trial arms in PME ratings at Visit 1 corresponded to differences in susceptibility to vaping at Visit 4; (2) PME at Visit 1 predicted susceptibility to vaping at Visits 1–4; and (3) PME at Visit 1 mediated the effect of the trial arm on susceptibility to vaping at Visits 2–4, with effects perceptions fully mediating this effect. Overall, the results provide evidence for a relationship between PME and AME and suggest that effects perceptions may be a useful proxy for certain AME outcomes.

The analyses reported here provide complementary findings that support the relationship between PME and AME with adolescents in the context of vaping prevention. First, using an experimental comparison, we found correspondence between PME and AME, consistent with findings of previous PME research using experimental correspondence approaches conducted at a single time point (Noar, Rohde, et al., 2020; Rayala et al., 2022; Rohde et al., 2021). Specifically, we found that FDA’s The Real Cost vaping prevention ads decreased susceptibility to vaping among adolescents compared to the control arm and that both effects and message perceptions ratings mirrored this impact. Second, regression analyses showed that PME at Visit 1 was associated with changes in susceptibility to vaping at Visits 1–4, which is consistent with findings of previous PME research that used the prediction approach in examining associations between PME and AME (Alvaro et al., 2013; Davis & Duke, 2018; Dillard, Shen, et al., 2007). Furthermore, mediation analyses revealed that effects perceptions fully mediated the impact of The Real Cost ads (vs. control) on susceptibility to vaping, while message perceptions only partially mediated this relationship. These findings are also consistent with emerging research on the mediating role of PME (Baig et al., 2021b; Brewer et al., 2019; Popova & Li, 2022).

Our trial is, to the best of our knowledge, the first to examine the mediating role of effects perceptions and message perceptions among adolescents in the context of vaping prevention. Baig et al. (2021b) investigated the mediating effect of PME in the context of smoking behaviors among adults, finding support for a mediational role of effects perceptions. However, one limitation of that trial was that it measured both PME and behavioral outcomes at the same time point at the last trial visit. In the current trial, using PME at Visit 1 as the mediator and susceptibility to vaping at Visits 2–4 as the outcome separated the two measures in time. This provided a mediation test, with a temporal ordering of the strength of ads (assessed by PME) being associated with susceptibility to vaping, which was assessed at later points in time.

We found support for PME–particularly effects perceptions–as a mediator of intervention effects, but it remains unclear whether PME is a true causal mediator of message impact or simply a proxy for the impact that ads are likely to have. In other words, the theoretical significance of PME as a mediator requires additional clarification. Baig et al. (2021b) suggested that effects perceptions may be a proxy for an orientation response that engages message receivers and leads to behavioral change. In that regard, PME may not be a mediator in the sense that the goal of messages is to change PME on the route to changing behavior, but rather PME may be a proxy measure that captures a combination of elements that together make an ad effective (e.g., persuasive arguments, powerful visual elements, emotionally evocative portrayals, and high production value). Further theoretical development and empirical testing of PME as a mediator in future research are warranted.

A critical question concerns whether our results suggest that effects perceptions have more diagnostic value in message pre-testing than message perceptions. In particular, some scholars have questioned entirely the diagnostic value of any measure of PME as a proxy for AME (O’Keefe, 2018, 2020; Pechmann et al., 2003), especially given the correlational nature of PME-AME data in most studies and the fact that reverse causation could be at play. We interpret the results of the current trial in the context of the broader literature that suggests that PME–especially effects perceptions – may be diagnostic of AME (Table 3). Perhaps the best complementary evidence to the current trial is a series of studies by Cappella and colleagues, showing that effects perceptions computed at the aggregate level are predictive of AME at the individual level (Bigsby et al., 2013; Morgan et al., 2020). In these studies, reverse causation is not possible because the PME ratings do not come from the individual but rather from the group. This allays the concern that reverse causation is at play and bolsters our conclusion that effects perceptions appear to be a robust AME proxy that can be usefully applied in message pre-testing.

Table 3. Different types of evidence supporting the relationship between PME and AME.

Approach	Study Findings	Prior Studies	Current Study
Experimental Correspondence	Differences on PME generally align with differences on AME in experiments	(Noar, Gottfredson, Vereen, et al. 2021; Noar, Rohde, et al. 2020; Rayala et al. 2022; Rohde et al. 2021)	Supported
Prediction – individual-level PME	PME ratings at the individual level predict AME in both cross-sectional and longitudinal studies	(Alvaro et al. 2013; Brennan et al. 2014; Davis and Duke 2018; Davis et al. 2017; Dillard and Ye 2008; Dillard, Shen, et al. 2007; Dillard, Weber, et al. 2007; Noar, Barker, et al. 2020)	Supported
Prediction – aggregate-level PME	PME ratings at the aggregate level predict AME	(Bigsby et al. 2013; Morgan et al. 2020)	n/a
Mediation	PME ratings at the individual level mediate the impact of messages on AME	(Baig et al. 2021b; Brewer et al. 2019; Dillard and Peck 2000; Popova and Li 2022)	Supported

Finally, our trial extends the PME literature in many ways. First, existing research has tended to use single message exposure experiments conducted at a single point in time. By contrast, we conducted a longitudinal RCT with multiple message exposures to examine how PME may predict AME. This trial design thus provided stronger evidence of the temporal ordering from PME to AME and better simulated a real-world campaign in which participants may be exposed to messages several times before AME is assessed. Second, most previous work has relied on a single approach to examine the PME-AME relationship, whereas we examined the relationship using three approaches: experimental correspondence, prediction, and mediation. Therefore, the current trial provided complementary tests of PME in the context of a single study. Third, we pitted effects perceptions against message perceptions in evaluating the relationship between PME and AME, providing a comparison of two reliable PME measures with different theoretical underpinnings rather than a test of a single measure.

Strengths and limitations

This trial has several strengths, including longitudinal exposure to messages, use of high-quality stimuli from a national campaign, recruitment of a large adolescent sample at risk of vaping, high retention, use of control messages, and inclusion of multiple PME measures and multiple analysis approaches. This trial also had limitations. First, we were unable to examine the effectiveness of PME for individual messages. Instead, we treated the three messages that participants saw as a group. Some research has suggested individual message-level validation as a strategy (O’Keefe, 2018), which poses challenges, including the potential lack of a single message to change AME. Still, evidence for the value of PME could be bolstered by study designs that tie PME for individual messages to AME, given that PME is typically used to select individual messages. Second, the same participants who rated PME were also evaluated on AME, and given that PME measures aim to be conceptually similar to their AME counterparts, it is possible that some association between PME and AME is due to a methodological artifact rather than PME diagnosing a message’s ability to change AME. Some researchers have suggested using different samples to test PME and AME (O’Keefe, 2018) or other strategies such as aggregate PME scores to control for personal factors that could confound individuals’ evaluation of messages. We were, however, unable to apply those approaches in this trial given that they require a different study design than that used in our RCT. Finally, we examined the PME-AME relationship with only one population, set of messages, exposure channel, and behavior, and future work should apply PME to other areas. While much of the work in PME has been in the tobacco context, there are an increasing number of studies applying PME to other health behaviors (Sigala et al., 2022; Taillie et al., 2022).

Conclusion

Findings of the current trial advance our understanding of the relationship between PME and AME and support for the PME-AME relationship (particularly effects perceptions) in vaping prevention among adolescents. Although effects and message perceptions focus on somewhat different aspects of message responses, results of our trial suggest prioritizing effects perceptions over message perceptions for diagnosing a message’s potential for behavior change. Findings from our trial should enhance confidence in the PME-AME relationship and use of PME ratings as a tool that may provide message designers with meaningful information about the potential of messages to change AME.

Disclosure statement

Seth Noar has served as a paid expert witness in litigation against tobacco and e-cigarette companies. Jacob Rohde has served as a paid expert consultant in litigation against tobacco companies. The other authors declared no conflicts of interest. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the FDA.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/10410236.2023.2222459.

Additional information

Funding

This project was supported by grant number [R01CA246600] from the National Cancer Institute and FDA Center for Tobacco Products (CTP).

Notes

1. In the trial pre-registration, we proposed that we would examine whether PME at Visit w mediates the impact of the trial arm on susceptibility to vaping at Visit w +1, where w refers to the time when participants did the survey, which ranges from Visit 1 to 3. We amended this to PME at Visit 1 only since during message pre-testing PME is typically only assessed once after first message exposure.