An Efficient Message Evaluation Protocol: Two Empirical Analyses on Positional Effects and Optimal Sample Size

Abstract

Reliable and valid message evaluation has a central role in effective health communication and message effects research. The authors have employed a message testing protocol to efficiently acquire valid and reliable message evaluation results: (a) use multiple messages, (b) recruit evaluators from the target population, (c) use valid and reliable effectiveness measures, (d) expose an evaluator to multiple messages, and (e) ensure enough evaluations per message. Two secondary analyses of anti-tobacco message evaluation studies provide evidence for reliability and validity regarding points (d) and (e). Seven studies where adult smokers evaluated the effectiveness of various anti-smoking campaign messages were examined. The first analysis shows that the position in which a message appears has little or no impact on its evaluation, supporting the validity of multiple-exposure design. The second analysis suggests having 25 evaluations per message can achieve a fair balance between accuracy and efficiency.

Supplementary data

Supplemental data for this article can be accessed here.

Notes

¹ The four studies tested 32, 60, 40, and 68 PSAs respectively. One PSA (“critics-cinema” by Truth Initiative, formerly American Legacy Foundation) was included in two studies (PSA2 and PSA4); the two incidents were treated separately in the study, resulting in 200 messages in total.

² Textual arguments were short paragraphs with on average of 27.08 words (SD = 10.51). One example based on the Truth campaign reads “Sodium hydroxide, a caustic chemical found in cigarettes, is found in many hair removal products. Tell others the facts about smoking.”

³ Although PSA1 used a shorter version of the PME measurement, which only included two statements, this measure also showed very high correlation in other studies when compared to the full measure (rs > .80).

⁴ Covariates included age, gender, race, education, need for cognition, nicotine dependence, and stage of change. Because White and Black participants composed the majority of the total sample, race was categorized into three groups: Whites, Blacks, and Other. Level of education was categorized into four groups (1 = ~grade 8, 2 = grade 9 ~ 12/high school/GED, 3 = some college, 4 = college or higher). For most studies, stage of change was assessed according to the contemplation ladder in smoking cessation (Prochaska & DiClemente, 1982), where the response options ranged from 0 (“I have not had thoughts about quitting smoking”) to 10 (“I am taking action to quit smoking”). However, in ARG1, the wording was slightly different, where participants were asked “On a scale of 0 to 10, how interested are you in quitting smoking?” The response to this question ranged from 0 (“Not at all interested”) to 10 (“Very interested”). Nicotine dependence was measured using the responses to the Fagerström test for nicotine dependence scale (Heatherton, Kozlowski, Frecker, & Fagerström, 1991). ARG1 did not include these items, so the analysis of ARG1 did not include nicotine dependence as control variable. Inclusion or exclusion of these covariates did not change the overall results. See Table 1 for descriptive statistics of the covariates.

Additional information

Funding

This work was supported by the National Cancer Institute [P20 CA095856, R01 CA160226].