Possible Solution to Publication Bias Through Bayesian Statistics, Including Proper Null Hypothesis Testing

Abstract

The present paper argues that an important cause of publication bias resides in traditional frequentist statistics forcing binary decisions. An alternative approach through Bayesian statistics provides various degrees of support for any hypothesis allowing balanced decisions and proper null hypothesis testing, which may prevent publication bias. Testing a null hypothesis becomes increasingly relevant in mediated communication and virtual environments. To illustrate our arguments, we re-analyzed three data sets of previously published data --media violence effects, mediated communication, and visuospatial abilities across genders. Results are discussed in view of possible Bayesian interpretations, which are more open to a content-related argumentation of varying levels of support. Finally, we discuss potential pitfalls of a Bayesian approach such as BF-hacking (cf., “God would love a Bayes Factor of 3.01 nearly as much as a BF of 2.99”). Especially when BF values are small, replication studies and Bayesian updating are still necessary to draw conclusions.

Acknowledgments

The authors of the re-analyzed data reported in this paper are greatly acknowledged for generously providing their data for our analyses. Please note that we do not evaluate the methods or quality of the research as such; we used these data sets for the current analyses only as interesting cases for our argument. The Netherland Institute for Advanced Studies (NIAS/KNAW) is acknowledged for granting the first author a fellowship allowing time to work on the current paper. In addition, we are grateful to the reviewers of this paper for their insightful and sharp feedback.

Notes

¹ We refer in particular to the commonly used Fisher’s p-value (or Null Hypothesis Significance Testing, NHST). In fact, Fisher’s NHST does not have an alternative hypothesis (Ha) only the null hypothesis (H0). It only tests the strength of evidence by calculating the probability of the observed value (or more extreme) than the observed value, based on the assumption that H0 is true. The method in fact does not test H0 against Ha.

² The pre-set α significance levels to which the obtained p-value is tested may also set to be more strict (e.g., p < .01), varying among disciplines, research designs, sample sizes, or measurement levels. Then, similar reasoning regarding publication bias still holds.

³ A full introduction to Bayesian statistics is beyond the scope of the current paper and we refer to Van De Schoot and Depaoli (2014) as a highly accessible start: http://www.ehps.net/ehp/index.php/contents/article/view/ehp.v16.i2.p75/26).

⁴ The default setting of the BIEMS software is a .5/.5 prior probability and cannot be changed, otherwise the method to calculate the BF is not valid (see Klugkist, Laudy, & Hoijtink, 2005).

⁵ Note that we do not want to claim that BF values have frequentist properties or that small BF values can be justified with such a simulation study. Furthermore, the sensitivity analyses are conducted given the acquired data, the statistical model and tested hypothesis. In addition to scrutinizing the sensitivity of BF for variations in sample size, we also varied the mean difference between the groups and the variance. Due to space limitations, the latter results are not reported, yet, they do not alter the picture as described here. These results can be requested by sending an email to the corresponding author.