An a priori solution to the replication crisis

ABSTRACT

Possibly, the replication crisis constitutes the most important problem in psychology. It calls into question whether psychology is a science. Existing conceptualizations of replicability depend on effect sizes; the larger the population effect size, the greater the probability of replication. This is problematic and contributes to the replication crisis. A different conceptualization, not dependent on population effect sizes, is desirable. The proposed solution features the closeness of sample means to their corresponding population means, in both the original and replication experiments. If the researcher has specified the sampling precision desired, it is possible to calculate the probability of replication, prior to data collection, and without dependence on the population effect size or expected population effect size. In addition, it is not necessary to know population means or standard deviations, nor sample means or standard deviations, to employ the proposed a priori way of thinking about replicability.

KEYWORDS:

• A priori procedure
• closeness
• confidence
• precision
• probability of replication
• replication crisis

Acknowledgments

I thank Uli Widmaier and two anonymous reviewers for their helpful comments.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. Some additional sources are as follows: Asendorpf and colleagues (2013), Brandt and colleauges (2013), Earp (2016), Earp and Everett (2015), Earp et al. (2014), Earp and Trafimow (2015), Ioannidis (2005), Makel and Plucker (2014), McBee and Matthews (2014), Open Science Collaboration (2015), and Prinz, Schlange, and Asadullah (2011).

2. The invalidity of significance testing was a major theme at the American Statistical Association Symposium on Statistical Inference, October, 2017. Also, see Hubbard (2016) and Ziliak and McCloskey (2016) for recent reviews.

3. Michelson eventually received a Nobel Prize in 1907.

4. Of course, it also is highly debatable whether null hypothesis significance tests validly allow one to reject null hypotheses either.

5. This interpretation is in contradiction to what physicists believe.

6. The following equation renders Cohen’s f for k means (Cohen, 1988, pp. 274–275): , where .

7. This is one obvious way in which the a priori procedure to be proposed differs from any procedure that depends on effect sizes (such as Cohen’s d or Cohen’s f). To use any other procedure, one at least needs to have obtained the sample means. In fact, for statistical significance or telescoping, one needs means from two experiments. In contrast, in the procedure to be proposed, replicability can be evaluated without knowing these, as will become clear later.

8. The derivation of equation (1) that Trafimow (2017b) provided makes use of the familiar notion of confidence intervals and can be transformed into a significance test. Nevertheless, Trafimow argued strenuously against using confidence intervals or significance tests to come to conclusions about hypotheses. In that spirit, the present argument features neither traditional confidence intervals nor significance tests.

9. The reader may wonder why it is not necessary to know or estimate the standard deviation to use equation (1). The reason is, as Trafimow (2017b) showed, the standard deviation cancels out in the derivation of equation (1).

10. These computations were based on an online power calculator: https://www.dssresearch.com/KnowledgeCenter/toolkitcalculators/samplesizecalculators.aspx.

11. There may be times when the researcher is interested in the difference in means as opposed to the means themselves. The mathematics for differences between means turns out to be more complicated than for the means themselves, and will be the topic of an upcoming paper. Nevertheless, the basic philosophical issues continue to apply.

12. Because theories contain nonobservational terms, it is necessary to use auxiliary assumptions to bridge the gap from these to the observational terms in empirical hypotheses. As Earp and Trafimow (2015) emphasized, differences in at least some auxiliary assumptions across two iterations of an experiment are inevitable.

13. See Trafimow and Rice (2009) for a quick and accessible description.

14. The American Statistical Association came out with a recent statement admitting that p-values do not justify conclusions about hypotheses (Wasserstein & Lazar, 2016).

15. A counter to this last is that theories are not necessarily supposed to work in many contexts. From the point of view that theories invoke idealized (and simplified) universes, it might be that they only should work when auxiliary assumptions are used that eliminate the complications. For example, Newton’s theory does not take friction into account, and so some Newtonian predictions only work in special contexts where friction is removed (the experiment is done in a vacuum or outer space) or when friction is accounted for by equations designed for that purpose.

16. Aficionados of Reichenbach (1938) or Whewell (1996) [1840]) could apply a slightly analogous distinction between the context of discovery and the context of justification, though that distinction will not be explored further here.