Unchecked vs. Uncheckable: How Opinion-Based Claims Can Impede Corrections of Misinformation

ABSTRACT

Although the prominence of fact-checking in political journalism has grown dramatically in recent years, empirical investigations regarding the effectiveness of fact-checking in correcting misperceptions have yielded mixed results. One understudied factor that likely influences the success of fact-checking initiatives is the presence of opinion statements in fact-checked messages. Recent work suggests that people may have difficulty differentiating opinion- from fact-based claims, especially when they are congruent with preexisting beliefs. In three experiments, we investigated the consequences of opinion-based claims to the efficacy of fact-checking in correcting misinformation regarding gun policy. Study 1 (N = 152) demonstrated that fact-checking is less effective when it attempts to correct statements that include both fact- and opinion-based claims. Study 2 (N = 561) replicated and expanded these findings showing that correction is contingent on people’s ability to accurately distinguish facts from opinions. Study 3 (N = 389) illustrated that the observed effects are governed by motivated reasoning rather than actual inability to ascertain fact-based claims. Together these results suggest that distinguishing facts from opinions is a major hurdle to effective fact-checking.

Supplementary material

Supplemental data for this article can be accessed on the publisher’s website.

Notes

¹ All three studies were approved by the Institutional Review Board at Northwestern University (Study number: STU00209397).

² Given concerns over a potential pretesting effect, we decided to use participants’ political affiliation, rather than directly measuring participants support for gun-related policy, to compute the ideological-congruence variable. To ensure that political affiliation can accurately predict participants’ views on gun-related policy, the final part of the questionnaire directly measured participants’ views on guns. Specifically, participants were asked to “Choose the option that best depicts how [they] feel about gun laws,” ranging from (1) “Gun laws should be more restrictive (harder to get and hold guns)” to (7) “Gun laws should be less restrictive (easier to get and hold guns).”The results of an independent samples t-test indicated a strong effect of political affiliation on support for gun-related policy (t(150) = 5.58, p < .001, d = 0.86), with Republicans much more likely to support gun rights (M = 5.37, SD = 2.82), compared to Democrats (M = 3.00, SD = 2.68).

³ Half of the participants in the condition that included a message-checkability detection task received feedback on their performance and half did not receive feedback. Considering the fact that performance feedback did not have direct or moderated effects on any of the research outcomes, including on message accuracy (b = −.59, SE = .56, p = .29, 95% CI [−1.70, .51], ΔR² = .01) and speaker’s credibility (b = −.72, SE = .56, p = .20, 95% CI [−1.82, .38], ΔR² = .01), the two conditions were combined.

⁴ Correcting for the increase in the number of experimental conditions, the same power analysis was utilized to calculate the required sample size for Study 2 (d = 0.40, α = .05, 1-β = .80) resulting in N = 520.

⁵ This was done to examine a potential confound, whereby the mere inclusion of the task might independently influence the research outcomes. As the MANOVA with message-checkability detection task (with/without) indicated, inclusion of the task did not significantly influence the research outcomes (Wilk’s λ = .99, F (3, 826) = 1.19, p = .31, η_p² = .004).

⁶ The specific instructions were: “The original article published in The News-Item included an interview with the political activist, Erica Taylor. The interview included both factual statements that can be objectively verified, as well as opinion-statements that cannot be verified. The following questions test your ability to classify statements as either factual or opinion. Regardless of how knowledgeable you are about each topic, would you consider each statement to be a factual statement (whether you think it is accurate or not) or an opinion statement (whether you agree with it or not)?”.

Additional information

Funding

This work was supported by the School of Communication at Northwestern University.

Notes on contributors

Nathan Walter

Nathan Walter (Ph.D. University of Southern California, 2018) is an assistant professor at the Department of Communication Studies, Northwestern University. His research focuses on cognitive, metacognitive, and emotional processes at the heart of misinformation and its correction.

Nikita A. Salovich

Nikita A. Salovich is a Ph.D. candidate in Cognitive Psychology in the Department of Psychology at Northwestern University. Her research examines when and why people are influenced by false information, and methods that support the evaluation of information.