Comparison Conditions in Research on Persuasive Message Effects: Aligning Evidence and Claims About Persuasiveness

ABSTRACT

In persuasion message effects research, two kinds of research design are common. One compares the persuasiveness of two different advocacy messages on the same topic. The other compares the persuasiveness of an advocacy message against a no-advocacy-message control condition. Because these two designs contain different comparisons they underwrite different claims, but the designs – and their corresponding claims – are prone to misunderstanding and confusion. And when a study combines the two designs, especially complex issues can arise. This article aims to sort out the relevant issues in the service of better alignment between evidence and claims in persuasive message effects research.

Disclosure statement

I have no known conflict of interest to disclose. Thanks to the journal’s reviewers and to Hans Hoeken for useful commentary.

Notes

1. Such random assignment is essential to permit appropriate interpretation of experimental results.

2. Imagine a two-advocacy-message design in which message A’s mean on the outcome variable was higher than message B’s mean (i.e., from the advocate’s perspective, message A looked better—that is, more effective). Absent additional information—and, specifically, as discussed below, absent a no-advocacy-message comparison condition—one cannot rule out the possibility that one or both of the messages produced a backfire effect.

3. Some designs expose participants to multiple advocacy messages advocating opposing views (i.e., a participant sees both a pro-X message and an anti-X message). In some cases this design is used to explore message order effects (“primacy-recency” effects, e.g., Insko, 1964); in others it is used to simulate a competitive message environment (e.g., Chong and Druckman, 2010; Dillard et al., 2023).

4. Even when a collection of studies all appear to have the same message contrast, there may be hidden complexities. For example, the contrast between one-sided messages (that contain only arguments supporting the advocated view) and two-sided messages (that present supportive arguments but also discuss opposing arguments) is straightforward enough. But Allen(1991) pointed out that one can distinguish two different message contrasts here on the basis of exactly how the two-sided message discusses opposing arguments—by simply acknowledging them or by refuting them.

5. In particular, some appropriate two-advocacy-message design would be useful, a design aimed at isolating potential active ingredients.

6. Looking at you, reviewer 2.

7. Researchers will also want to bear in mind larger potential problems with pretest assessments, such as pretest sensitization effects. The classic discussion is Campbell and Stanley, (1963, e.g., p. 18); see also Shadish et al., (2002, e.g., p. 260).

8. The results of de Hoog et al. (2007) were originally reported using d as the effect size index; the results have been converted to r for easy comparison with the results of Walter et al. (2018).

Additional information

Notes on contributors

Daniel J. O’Keefe

Daniel J. O’Keefe is the Owen L. Coon Professor Emeritus in the Department of Communication Studies at Northwestern University