If You Have Choices, Why Not Choose (and Share) All of Them? A Multiverse Approach to Understanding News Engagement on Social Media

Abstract

Social sciences are facing a crisis of replicability, and concerns about the confidence in quantitative findings have resulted in an increasing interest in open science practices across many fields. In this article we introduce scholars of (digital) journalism studies and communication science to multiverse analysis while addressing the possible reasons of heterogeneity in the findings of research on engagement with news on social media. Using the question of which news article characteristics predict news engagement on social media, this illustration of the multiverse approach shows how different measurement, data processing, and modelling choices lead to divergent conclusions. In particular, we show how the selection of widely used automated text analysis tools and preprocessing steps influence the conclusions drawn from the analysis. We also use this illustration to guide interested scholars through the different steps of doing a multiverse analysis. More broadly, we demonstrate how multiverse analysis can be an open and transparent research approach in a field that is increasingly faced with a wide range of analytical choices.

Keywords:

• Multiverse analysis
• online news engagement
• news values
• open science
• text analysis
• sentiment analysis

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Notes

1 Preregistration material and replication code available on the OSF project site at https://osf.io/68cje/

2 The number of articles included in the study varies between countries. In some countries (e.g. Germany, Australia), a large amount of news articles is shared exclusively by news sites’ regional or thematic subpages which we did not include. We provide a breakdown of the sample sizes per news site Facebook page in section A3 in the Supporting Information.

3 For an example see the replication code of the foundational multiverse paper by Steegen et al. (2016), available at: https://osf.io/zj68b/

4 https://osf.io/68cje/

5 www.github.com/cpipal/multiverse-tutorial

Additional information

Funding

Christian Pipal received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme under grant agreement No 759079, POLEMIC.