Are Experts (News)Worthy? Balance, Conflict, and Mass Media Coverage of Expert Consensus

ABSTRACT

Overlooked in analyses of why the public often rejects expert consensus is the role of the news media. News coverage of expert consensus on general matters of policy is likely limited as a result of journalists’ emphasis in news production on novelty and drama at the expense of thematic context. News content is also biased toward balance and conflict, which may weaken the persuasiveness of expert consensus. This study presents an automated and manual analysis of over 280,000 news stories on 10 issues where there are important elements of agreement among scientists or economists. The analyses show that news content typically emphasizes arguments aligned with positions of expert consensus, rather than providing balance, and only occasionally cites contrarian experts. More troubling is that expert messages related to important areas of agreement are infrequent even in relevant news content, and cues signaling the existence of consensus are rarer still.

KEYWORDS:

• expert consensus
• journalism
• mass media
• media bias
• science communication
• text analysis

Acknowledgments

Thanks to my dissertation committee, Paul Quirk, Richard Johnston, and Fred Cutler, as well as my external examiner John Bullock for the useful comments and suggestions. Also, grateful for helpful feedback from Dominik Stecula and Frédéric Bastien. Thanks as well to Kala Bryson for her exceptional research assistance and to UBC's Department of Political Science for their financial support.

Disclosure statement

No potential conflict of interest was reported by the author.

Data availability

Data and replication materials are available with the Open Science Foundation (https://osf.io/rks8b/)

Open Scholarship

This article has earned the Center for Open Science badge for Open Data and Open Materials. The materials are openly accessible at DOI:10.17605/OSF.IO/TPA6U.

Supplementary material

Supplementary data for this article can be accessed on the publisher’s website at https://doi.org/10.1080/10584609.2020.1713269.

Notes

1. The Dallas Morning News and the San Diego Union-Tribune both endorsed Republican presidential candidates in every cycle from 1980 to 2012. The Houston Chronicle endorsed the Republicans in 8 of 9 presidential election cycles. In contrast, the New York Times and Washington Post endorsed Democrats in every cycle in that period, with one exception. Newspaper endorsement histories can be found here: https://noahveltman.com/endorsements/.

2. Here and throughout a random number generator was used to select samples of news articles from a relevant corpus. An undergraduate research assistant coded a random sample of 200 articles I had classified for relevance. Our coding was in 90% agreement, with a Krippendorf’s Alpha of 0.79. More details on validation can be found in Appendix G of the supplementary materials.

3. Accuracy tells us how often the algorithm got it right ([True Positives + True Negatives]/Total). But, the purpose of this analysis is to identify a sample of articles for manual coding. As such, the more important measure of performance is how many true positives we have relative to all articles that were coded as relevant. This is what the precision score gets us (True Positives/[True Positives + False Positives]). The best estimate for false positives (i.e. precision) in the sample is 10%.

4. That is, out of a corpus of articles with discussion relevant to the pertinent expert consensus and that had an expert citation, I used a random number generator to identify a sample of 100 articles within each media subset.

5. The research assistant was given a coding manual that was developed iteratively over the course of four practice rounds where I evaluated coding quality and provided feedback. The final coding manual is shown in Appendix F of the supplementary materials. After the training, they coded 250 additional articles that were then used to inform the intercoder reliability scores.

6. These scores can be evaluated based on whether the cumulative probability of being in a category or higher is greater than 0.9, which is, in part, dependent on the standard errors. Our percentage agreement scores are all rate as “very good” according to Altman’s (1991) benchmarking scale (0.8 to 1) recommended by Gwet (2014). Our Gwet’s AC scores all rate as “good” (0.6 to 0.8). Our Krippendorf’s Alpha scores all rate as “good” as well, with the exception of false balance, which is rated as “moderate” (with a score of 0.6, benchmark of 0.4 to 0.6). The low score on false balance for the Krippendorf’s Alpha is in large part due to the infrequency of false balance in our sample. More details on the validation tests can be found in Appendix G of the supplementary materials.

7. Republican-leaning outlets (e.g. Fox News) were less likely than Democratic-leaning outlets (e.g. CNN) to feature expert citations or messages on issues where Republican and conservative elites resist expert consensus. The reverse is true on issues where Democratic-elites resist expert consensus. Results displayed in panel A of Figure H1 of the supplementary materials.

8. Democratic-leaning outlets are more likely than Republican-leaning outlets to emphasize arguments aligned with the expert consensus on issues where Republican and conservative elites resist expert consensus. The reverse is true on issues where Democratic-elites resist expert consensus. Results displayed in panel B of Figure H1 of the supplementary materials.

9. There is evidence of partisan or ideological bias across outlets in the prevalence of false balance and polarizing opponents. Results displayed in panel C of Figure H1 of the supplementary materials.

Additional information

Funding

This work was supported by the Social Sciences and Humanities Research Council of Canada [752-2015-2504] and the University of British Columbia's Department of Political Science.

Notes on contributors

Eric Merkley

Eric Merkley (PhD, UBC) is a postdoctoral fellow in the Munk School of Global Affairs and Public Policy at the University of Toronto. He studies political communication and mass behavior using experimental methods and quantitative text analysis with a focus on public opinion related to topics with scientific and expert consensus.