Testing an additive model for the effectiveness of evidence on the persuasiveness of a message

Abstract

Meta-analyses provide support for the improved persuasiveness of a message including: (a) evidence source qualification, (b) assertive evidence, and (c) statistical over narrative evidence. The current empirical examination seeks to determine whether the persuasiveness of a message represents the additive effect of the individual message elements. The data in this investigation support this view of message persuasiveness.

Keywords:

• Persuasion
• Additive model
• Source qualification
• Statistical evidence
• Assertive evidence

The various meta-analyses dealing with the influence of evidence provide a consistent body of results indicating that inclusion of evidence improves the persuasiveness of the message. The impact of the various research efforts indicate that a persuader would be wise to generate a message using evidence to maximize the positive impact on an audience. The available social scientific evidence supports a view of message construction advice included in almost every public speaking book. However, should a communicator, to maximize persuasion, combine in a single message all the message elements believed to enhance message persuasion? While finding all the available means of persuasion may be a wise consideration in message preparation, the actual employment of all the available means in one message may not provide an improvement in the persuasiveness of the message. This investigation provides a test of whether some effects found individually can be added with other individual effects to improve the persuasiveness of a message. The challenge for a more sophisticated view of persuasion is to go beyond the analysis of simple message design questions (e.g., inclusion or non-inclusion of evidence) to a more sophisticated and complex view of the message task faced by the communicator. The first section examines outcomes of various meta-analyses to establish expected outcomes. All meta-analyses use the Hunter and Schmidt form of analysis, which Anker, Reinhart, and Feeley (2010) point out to provide a more accurate estimate of effect than other random models.

Established effects of evidence from meta-analyses including assertive evidence

Probably the simplest addition of support for a claim in a message is the addition of a quote or reference to support a conclusion by an expert (Reinard, 1988). Essentially the conclusion offered in the message that is supported by additional expert or authoritative persons constitutes a simple version of evidence. The question is whether the inclusion of this material improves the effectiveness of the message for a persuader. The simplest way of employing this evidence form would be the use of a quotation from a recognized expert or source in support of a position. The function of this form of evidence is to support the conclusion offered by the communicator by indicating that another respected source/expert endorses the conclusion offered in the message. The impact of such material is to provide an endorsement for the reasoning of the communicator and indicate that the message conclusion should be accepted as reasonable. Reinard's meta-analysis (1998) provides evidence that the use of testimonial assertion improves the effectiveness of a message,  = .25. The sample effect is homogeneous, indicating that no evidence for a moderating variable exists. The finding indicates a very simple conclusion: A communicator, by including a simple testimonial assertion in support of a conclusion, increases the persuasiveness of the message.

The resulting advice to a person generating a persuasive message is simple: Find an expert or source that endorses the conclusion offered by the communicator. For example, a person quoting the need for extension of the Bush tax cuts might choose to quote Senator Finglegruber: “Extension of the tax cuts will provide a basis for economic expansion by reducing the burden on families and small businesses and increasing demand for goods.” As long as the audience would view the hypothetical senator as someone worthy of consideration, then the support for the conclusion indicates agreement with some other existing institutional source of knowledge.

Statistical or narrative evidence use

The consideration in this issue for the use of evidence involves the form of support for a message conclusion that provides superior opportunity to persuade an audience. The existing meta-analysis by Allen and Preiss (1997) supports the improved effectiveness of statistical over narrative evidence,  = .10.1

Statistical evidence constitutes the use of number or a counting of examples in an effort to provide support for a conclusion. An example of this would be a statement such as “90% of economists support the action of extending the Bush tax cuts.” The systematic collection of examples summarized in numerical form is an attempt to demonstrate consensus or a magnitude of effect (Allen & Preiss, 1997). The goal is the establishment of a generalized conclusion across a number of circumstances or conditions.

The narrative form of evidence provides support usually in the form of an example or story that illustrates the conclusion. The advantage of an example is the “human” face that provides a reference point that message receivers can examine and feel the application. Numerical data can be considered abstract and removed from the reality of experience. An example of narrative data would be a person talking about how “a local neighborhood store owned by the Smiths operated for years until excessive government taxation drove the store out of business. The Smiths commented on how the lowering of taxes would have saved their store had it come sooner. We need to extend the tax cuts to protect all the Smiths out there.” The evidence argues for acceptance of a conclusion based on the representation of a story used to illustrate the value of that evidence.

Research indicates that a combination of statistical and narrative evidence is the most persuasive message (Allen et al., 2000). Further, Hoeken and Hustinx (2009) suggest that conditions may exist when one would prefer anecdotal or statistical evidence. The question about whether such relations continue to hold across generalized situations continues to receive attention, particularly when considering messages involving public health or medical issues (DeWit, Das, & Vet, 2008; Feeley, Marshall, & Reinhart, 2006; Ubel, Jepsom, & Baron, 2001). In particular, Greene, Campo, and Banerjee (2010) found that, compared to either statistical or anecdotal evidence, normative evidence was more persuasive with college students about the risks of tanning bed use. The preference could be related to whether the particular topic involved an ongoing behavior of a person and the reasons for that behavior (Slater & Rouner, 2002).

Evidence source qualification

This issue deals not with the use of evidence or support for an argument but instead the question of whether a message sender should provide qualifications for the source of the evidence. The case for the use of qualifications is similar to the assumptions of source credibility, that sources with higher credibility produce more persuasive outcomes—this of course may not be universally true in the case of a sleeper effect (e.g., Hovland, Lumsdaine, & Shefield, 1949) or reactance (e.g., Brehm, 1966). In this case, using evidence from sources that are more believable should improve the persuasiveness of the message. Such advice is routinely provided for persuaders to find and use a credible source for the evidence support for a claim.

The question of the impact of the source of evidence in a message simply goes to whether or not to accept the credibility of the information provided. Providing a statistic or opinion is important to buttress and support the position of the message source. However, understanding the qualification and basis of the claim of the source of the evidence provides an additional level of support for the conclusion. O’Keefe (1998) finds that providing source qualification improves persuasiveness in a meta-analysis of such investigations,  = .06.

Such efforts abound in, for example, political debates where both members of Congress and the Executive Branch will point to reports from the Congressional Budget Office (CBO) or the General Accounting Office (GAO) because the agencies are represented as nonpartisan experts. The sense of fairness or authoritativeness for such evidence remains an important element in employing evidence effectively in a message. Table 1 displays the results of the three different meta-analyses conducted on the topics of interest to this investigation (Allen & Preiss, 1997; O’Keefe, 1998; Reinard, 1998).

Table 1. Summary of results of the meta-analyses

Author(s)	Year	Finding	N participants	N effects
Reinard	1998	Improvement of persuasiveness when including testimonial assertive evidence	1920	16	.25
Allen & Preiss	1997	Improvement of persuasiveness when including statistical evidence versus narrative evidence	1760	15	.10
O’Keefe	1998	Improvement of persuasiveness when including cited information source	5358	23	.06

An additive model

An unanswered question from the previous meta-analyses is whether the individual effects examined in the meta-analyses are additive or not. If each average effect were treated as the best estimate of the predicted effect, then the estimated effect of combining the various elements would be the sum of the average effects. Consider that for testimonial assertive evidence,  = .25. A message including none of such evidence could be considered the control group (i.e., the zero point) and the inclusion of the testimonial assertive evidence an improvement by the magnitude of  = .25. Similarly each meta-analysis demonstrates the average effect by using the specified evidence; statistical versus narrative evidence and source qualification. If the effect indicates the magnitude of the expected increase, the effectiveness of any combination can be calculated by adding the mean effect size estimates. This calculation is provided in Table 2 and gives a theoretical ordering of the effectiveness for each type of message.

Table 2. Predicted and observed effects

Condition	Assertive evidence	Statistical evidence	Source citation	Predicted effect (r)	Observed
					M	SD	n	Dunnett's t	Effect (r)
1				.00	7.86	2.93	144		.00
2			•	.06	8.49	3.18	150	1.84	.11
3		•		.10	8.29	2.68	139	1.24	.07
4		•	•	.16	9.05	2.99	136	3.37*	.20
5	•			.25	8.41	3.00	138	1.57	.09
6	•		•	.31	8.51	2.91	145	1.87	.11
7	•	•		.35	9.11	2.74	153	3.64*	.21
8	•	•	•	.41	9.35	3.17	136	4.23*	.25
•indicates the presence of the designated message features. *p < .01.

A similar issue to the impact of evidence arises in the examination of fear appeals suggested by the extended parallel process model (EPPM; Witte & Allen, 2000). The results demonstrate that the individual elements are largely additive for a fear message (i.e., severity of threat, vulnerability to threat, efficacy of solution, personal efficacy of solution implementation). What happens is that, rather than relying on a few message characteristics, the most effective message will combine the characteristics to be maximally effective at persuasion. EPPM suggests that the failure to incorporate all elements can also reduce the effectiveness of some of the elements because the unity of the elements are not only mutually reinforcing but in some cases required for the other elements to become effective.

Public speaking books (e.g., Lucas, 2010) provide advice on effective communication and the need for evidence to support arguments. The advice is often offered piecemeal, and the elements are individual without a full consideration of the message as a kind of comprehensive or holistic system.

Method

Participants

The participants came from a variety of student sources most of whom received extra credit for participation in the investigation. A total of 1140 useable surveys were included in this analysis.

Message construction

The topic of the message was global warming, with a message conclusion that global warming was a myth. The message was constructed using evidence from websites from organizations and scientists that believed global warming data to be overstated and that many counterexamples existed. The message had a total of eight versions in a 2 (evidence source qualifications provided, not provided) × 2 (statistical, narrative evidence) × 2 (testimonial evidence included, not included) factorial design. Full examination of the various issues in message construction is displayed in Table 2.

The inclusion of source information for evidence was accomplished by including or omitting the qualifications for the particular source of the information used in the message. For example, in the source information presented, a quote came from “Dr. Nils-Axel Mörner, former president of INQUA, an International Fraternal Association for the Study of Sea Changes, and the chair of the Commission on Sea-Level Changes and Coastal Evolution from 1999 to 2003.” Similarly, one source was identified in the source information and presented as “Jame Peden, a member of the American Institute of Aeronautics and Astronautics and served as an Atmospheric Physicist at the Space Research and Coordination Center.” The lack of source qualification condition was presented by giving the name of the source without the affiliations. The assumption is that the better the qualification presented for the source information, the more persuasive the message.

The narrative versus statistical evidence condition was manipulated by using one of two paragraphs. One paragraph pointed to the explanation for the diminishing glaciers on Mt. Kilimanjaro. The narrative explanation provides a story of how not global warming but deforestation at the base of the mountain has caused the glaciers to retreat. The focus is on the single example provided as a basis for understanding the global impact. The statistical presentation provides an examination of 50 glaciers internationally and the rate of advance or retreat. The analysis of the lack of glacial decline is based on a NASA satellite survey of more than 347 million radar altimeter measurements that reports the decline or increase in number of feet each year. The focus of this proof is based on a comprehensive set of statistical analyses conducted across the world. The analysis provides no story or explanation and focuses on no single example. To manipulate the narrative and statistical messages, only one of the appropriate paragraphs was used for the message.

Finally, the use of testimonial assertive evidence was the inclusion or non-inclusion of a paragraph about the International Climate Conference where a group of more than 500 experts issued a statement claiming that no consensus exists among the scientific community about the existence of global warming. The quotation and paragraph present a concluding statement from the group without a presentation of reasons or evidence about the problem. The paragraph was either included or not included for the use of evidence, and no specific source was attributed to this particular evidence so this form of evidence did not have the source qualifications manipulated because the evidence was either included or not included.

Measure

To measure attitude, three 5-point Likert items (1 = strongly disagree, 5 = strongly disagree) were included to examine the impact of the message. The three items were “I agree that global warming is overstated,” “Global warming represents a major problem (R),” and “Global warming is a proven scientific fact (R).” The items established an acceptable internal consistency with standardized α = .85. Accordingly, the three items were summed across to form a composite index of attitude toward global warming.

Statistical analysis

The statistical analysis considers two different approaches: (a) test of the overall additive model consistent with the theory, and (b) a traditional examination using a full factorial ANOVA. The test of the theoretical model is accomplished using an effects-coded procedure, which specifies the order of the eight different messages as provided in Table 2. The result provides a t-test statistic representing the accuracy of the model. A significantly positive t-value indicates the ordering of eight cell means is consistent with the hypothesis.

The full factorial ANOVA provides an alternative view of the results and considers the possibility of interactions existing that could account for the theoretical model. The full model also provides consideration or evidence of the potential for other alternative models to exist. The full factorial model, to be consistent with the theoretical model, should demonstrate three main effects for each form of evidence and no higher-order interaction effects.

The difference between the full factorial model and theoretical model is that the theoretical model provides information about the requirements of the size of the main effects, while the full factorial model does not require a particular relative size of the main effect.

Results

Test of the theoretical additive model

The additive model received support, albeit weak in magnitude, t(1133) = 3.99, p < .05. Essentially the ordering of the means predicted on the basis of effect sizes associated with the particular meta-analysis is supported. For a closer examination of the pattern of the current data and its proximity to the predicted data pattern, the mean of each treatment condition (i.e., Condition 2 through Condition 8) was compared against the mean of the control condition (i.e., Condition 1). Thus obtained Dunnett's t-values were converted into product moment correlation rs for comparison with the predicted effect sizes (see Table 2 for descriptive statistics, t-values, and predicted/observed effect sizes). The observed data points roughly approximate the predicted linear pattern up until Condition 4, but begin to depart starting Condition 5, largely because the observed main effect for inclusion of assertive evidence was substantially smaller than predicted. The following F-tests results further substantiate this reasoning.

Full factorial ANOVA model test

The full factorial model examines several aspects of the results. First, the main effects for each of the manipulated variables are examined to see if each of the forms of evidence adds to the persuasive effect of the message. The second part of the analysis considers whether any interactions exist to create forms of messages producing a greater persuasion than predicted by an additive model.

The main effect for the source citation provided to evidence is significant and in the expected direction, F(1, 1133) = 6.10, p < .05. Arguments presented by cited sources (M = 8.84, SD = 3.08, n = 567) were rated as more persuasive than arguments by sources with no affiliations (M = 8.43, SD = 2.87, n = 574). The observed effect, r = .07, is about the size normally expected for this kind of manipulation as indicated in the meta-analysis,  = .06.

The main effect for statistical versus narrative evidence is significant and consistent with predictions, F(1, 1133) = 13.10, p < .01. Participants found messages including statistical evidence (M = 8.95, SD = 2.91, n = 564) more persuasive than messages including narrative evidence (M = 8.32, SD = 3.01, n = 577). The obtained effect, r = .11, is comparable to the effect predicted from the previous meta-analysis,  = .10.

The main effect for inclusion of assertive evidence was also significant, F(1, 1133) = 5.81, p < .05; participants rated messages containing assertive evidence (M = 8.85, SD = 2.97, n = 572) as more persuasive than messages without assertion (M = 8.42, SD = 2.98, n = 569). However, the obtained effect, r = .07, was substantially smaller then predicted by the meta-analysis,  = .25.

No significant higher-order interactions existed. The lack of significant interactions indicates that no combination of effects was exhibiting a nonlinear set of impact on the persuasiveness of the message. The lack of interaction provides further support for an additive model.

Discussion

The additive model fit was significant and the theoretical model received weak support. The predicted ordering of effects was unapparent in the current data. The presence of three main effects in the absence of interactions supports an additive model for the elements tested. The conclusion and advice to persuaders would be to combine these elements of persuasion to maximize message effectiveness. This finding indicates that instructors of public speaking and persuasive messages are warranted in training speakers to incorporate all three elements in a message.

The challenge of providing advice to those constructing persuasive messages requires understanding the relationship among message components. The normal set of experimental designs usually involves some simple combination of message elements that are treated as separate and individual, not as a holistic set of components that work together with a sense of integration for maximum persuasion. Persuasion theories predict the impact of a message as a whole, but the experimental manipulation of messages is an attempt to reduce a message (i.e., a holistic combination of circumstances) into component parts, which might be unrealistic. One problem associated with this investigation is that assertive evidence was either included or not and source qualifications were not manipulated (that would have doubled the number of messages required), and this represents another possible issue to be considered in future investigations. What this means is that the source qualification manipulation only applies when considering the narrative versus statistical evidence comparison.

Diminishing returns represent an issue not fully addressed in persuasion theories. A fundamental problem with the component approach is that the effects at some point cannot be fully additive because, given the infinite number of potential issues, the continued addition of new elements would mean a more persuasive yet lengthy message. At some point even very effective strategies, when all incorporated into a message, may fail to induce persuasion proportionately. The current investigation, using a message a little over one page long, about 240 words, did not begin to exhaust what might be considered the impact of diminishing returns. The challenge of developing and measuring a model based on a message of diminishing returns is warranted.

One possible limitation is the consideration of cultural influences in how evidence is evaluated within a message. Hornikx and Hoeken (2007) indicate that French and Dutch message recipients differ on how statistical, expert, causal, and anecdotal evidence is considered. The finding is explained in terms of power distance dimensions (Hofstede, 2001) consistent with the findings of Pornpitakpan and Francis (2001) about the persuasiveness of experts. In their study Pornpitakpan and Francis demonstrate that Canadians were more persuaded on the basis of the argument in the message, whereas Thais responded more to the expertise of the source of the message. Other research supports the influence of cultural variations on the persuasiveness of a message (Hoeken, Starren, Nickerson, Crijns, & van Brandt, 2007; Jung & Kellaris, 2006). The findings of this model and for the forms of evidence require empirical testing with a variety of cultures to examine whether the model will continue to hold. One possibility is that the values for the particular components could change across cultures, while the additive property among the components remain culturally invariant (see Das, Kerkhof, & Kuiper, 2008, for interaction with positive/negative frames and evidence). What this would indicate is that the expectation of additivity would remain but the contribution of the various elements would change as the particular evaluations change.

This cultural hypothesis may find some support in the present results. The actual size of the effects observed for each of the components did not match that found in the meta-analysis. However, the overall additive model was maintained. Given the lag of several decades from the initial research to the current investigation, and the fact that additional aspects of the message were manipulated and examined, a change is likely in the amount of contribution that each component makes toward the overall persuasion. Nevertheless, the underlying assumption of additivity persisted. The ability to examine this further provides a basis for additional investigation and theorizing.

This investigation examines message elements and not the psychological states of the receiver (O’Keefe, 2003). The question left unanswered is whether message elements change or create particular psychological states (e.g., credibility, expertise, trustworthiness, argument quality, positive/negative message elaboration, etc.) in the minds of the message recipients. The investigation provided here indicates that the expectation of how message features will generate outcomes but does not provide the mechanism in terms of message processing as to what psychological differences are triggered as a result of changes in those message features. Additional research needs to provide insight into whether these elements work singly or in combination to generate such outcomes.

One aspect of the message that may intentionally influence the results may be the length considerations of the message. A message with some of the material, like source qualifications for evidence, will be longer than a message without such material. The potential impact of length differences in the message thus requires some consideration. This is particularly an issue when considering that the measure of message effect was immediate, and no longitudinal effects were examined. While this is consistent with almost all previous research in this area, issues like the “sleeper effect” are related to perceptions of source credibility and the impact of both decay as well as improvement in observed effects over time (Allen & Stiff, 1989). Future research examining the potential impact of these issues is warranted.

Everyone expects that providing quotations, a persuasive form of evidence, would reach a limit to effectiveness. However, efforts to establish the boundaries of effectiveness do not exist in the current literature. That is, if additiveness remains true for many message effects, the use of more strategies and better content may provide a stronger basis for improving message effectiveness. The question is at what point an audience becomes bored, inattentive, or convinced. Similarly, one could also conjecture whether reactance or a boomerang would produce diminishing or negative returns from efforts at persuasion based on redundant evidentiary proofs. The question of boundary conditions has not been addressed. The only example of that exists in the examination of fear appeals based on manipulation size. Results demonstrate a linear relationship: More fear leads to larger attitude change (Boster & Mongeau, 1984, Witte & Allen, 2000). The additional exploration of the additive model and the prospect of predicting diminishing returns requires further exploration.

Notes

¹ This effectiveness, however, is qualified by subsequent research which demonstrates that using both forms of evidence (statistical and narrative) is an even more effective strategy then using one form of evidence alone (Allen et al., 2000).