A Bayes Factor for Replications of ANOVA Results

References

• Albarracín, D., Handley, I. M., Noguchi, K., McCulloch, K. C., Li, H., Leeper, J., Brown, R. D., Earl, A., and Hart, W. P. (2008), “Increasing and Decreasing Motor and Cognitive Output: A Model of General Action and Inaction Goals,” Journal of Personality and Social Psychology, 95, 510–523. DOI: 10.1037/a0012833.
   PubMed Web of Science ®Google Scholar
• Anderson, C. J., pan Bahnik, t., Barnett-Cowan, M., Bosco, F. A., Chandler, J., Chartier, C. R., Cheung, F., Christopherson, C. D., Cordes, A., Cremata, E. J., Della Penna, N., Estel, V., Fedor, A., Fitneva, S. A., Frank, M. C., Grange, J. A., Hartshorne, J. K., Hasselman, F., Henninger, F., van der Hulst, M., Jonas, K. J., Lai, C. K., Levitan, C. A., Miller, J. K., Moore, K. S., Meixner, J. M., Munafo, M. R., Neijenhuijs, K. I., Nilsonne, G., Nosek, B. A., Plessow, F., Prenoveau, J. M., Ricker, A. A., Schmidt, K., Spies, J. R., Stieger, S., Strohminger, N., Sullivan, G. B., van Aert, R. C. M., van Assen, M. A. L. M., Vanpaemel, W., Vianello, M., Voracek, M., and Zuni, K. (2016), “Response to Comment on “Estimating the Reproducibility of Psychological Science,” Science, 351, 1037–1037. DOI: 10.1126/science.aad9163.
   PubMed Web of Science ®Google Scholar
• Anderson, S. F., and Maxwell, S. E. (2016), “There’s more than One way to Conduct a Replication Study: Beyond Statistical Significance,” Psychological Methods, 21, 1–12. DOI: 10.1037/met0000051.
   PubMed Web of Science ®Google Scholar
• Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J. A., Fiedler, K., Fiedler, S., Funder, D. C., Kliegl, R., Nosek, B. A., Perugini, M., Roberts, B. W., Schmitt, M., van Aken, M. A. G., Weber, H., and Wicherts, J. M. (2013), “Recommendations for Increasing Replicability in Psychology,” European Journal of Personality, 27, 108–119. DOI: 10.1002/per.1919.
   Web of Science ®Google Scholar
• Bakan, D. (1966), “The Test of Significance in Psychological Research,” Psychological Bulletin, 66, 423–437.
   PubMed Web of Science ®Google Scholar
• Bayarri, M. J., Benjamin, D. J., Berger, J. O., and Sellke, T. M. (2015), “Rejection Odds and Rejection Ratios: A Proposal for Statistical Practice in Testing Hypotheses,” Journal of Mathematical Psychology, 72, 90–103. DOI: 10.1016/j.jmp.2015.12.007.
   Web of Science ®Google Scholar
• Belia, S., Fidler, F., Williams, J., and Cumming, G. (2005), “Researchers Misunderstand Confidence Intervals and Standard Error Bars,” Psychological Methods, 10, 389–396. DOI: 10.1037/1082-989X.10.4.389.
   PubMed Web of Science ®Google Scholar
• Boekel, W., Wagenmakers, E.-J., Belay, L., Verhagen, J., Brown, S., and Forstmann, B. U. (2015), “A Purely Confirmatory Replication Study of Structural Brain-Behavior Correlations,” Cortex, 66, 115–133. DOI: 10.1016/j.cortex.2014.11.019.
   PubMed Web of Science ®Google Scholar
• Bos, C. S. (2002), “A Comparison of Marginal Likelihood Computation Methods,” in Compstat, Heidelberg: Physica-Verlag HD, pp. 111–116.
   Google Scholar
• Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., and van ’t Veer, A. (2014), “The Replication Recipe: What Makes for a Convincing Replication?” Journal of Experimental Social Psychology, 50, 217–224. DOI: 10.1016/j.jesp.2013.10.005.
   Web of Science ®Google Scholar
• Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., and Munafò, M. R. (2013), “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience,” Nature Reviews Neuroscience, 14, 365–376. DOI: 10.1038/nrn3475.
   PubMed Web of Science ®Google Scholar
• Chib, S., and Greenberg, E. (1995), “Understanding the Metropolis-Hastings Algorithm,” The American Statistician, 49, 327. DOI: 10.1080/00031305.1995.10476177.
   Web of Science ®Google Scholar
• Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences, (2nd Ed.), Hillsdale, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Cumming, G. (2008), “Replication and p Intervals: p Values Predict the Future Only Vaguely, but Confidence Intervals Do Much Better,” Perspectives on Psychological Science, 3, 286–300. DOI: 10.1111/j.1745-6924.2008.00079.x.
   Web of Science ®Google Scholar
• Cumming, G. (2012), Understanding The New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis, New York: Routledge.
   Google Scholar
• Cumming, G., and Finch, S. (2001), “A Primer on the Understanding, Use, and Calculation of Confidence Intervals that are Based on Central and Noncentral Distributions,” Educational and Psychological Measurement, 61, 532–574. DOI: 10.1177/0013164401614002.
   Web of Science ®Google Scholar
• Dienes, Z. (2016), “How Bayes Factors Change Scientific Practice,” Journal of Mathematical Psychology, 72, 78–89. DOI: 10.1016/j.jmp.2015.10.003.
   Web of Science ®Google Scholar
• Edwards, W., Lindman, H., and Savage, L. J. (1963), “Bayesian Statistical Inference for Psychological Research,” Psychological Review, 70, 193–242. DOI: 10.1037/h0044139.
   Web of Science ®Google Scholar
• Etz, A., Gronau, Q. F., Dablander, F., Edelsbrunner, P. A., and Baribault, B. (2018), “How to become a Bayesian in Eight Easy Steps: An Annotated Reading List,” Psychonomic Bulletin & Review, 25, 219–234. DOI: 10.3758/s13423-017-1317-5.
   PubMed Web of Science ®Google Scholar
• Etz, A. and Vandekerckhove, J. (2016), “A Bayesian Perspective on the Reproducibility Project: Psychology,” PLOS ONE, 11, e0149794. DOI: 10.1371/journal.pone.0149794.
   PubMed Web of Science ®Google Scholar
• Gamerman, D. and Lopes, H. F. (2006), Markov Chain Monte Carlo: Stochastic Simulation for Bayesian Inference, Boca Raton, FL: Chapman & Hall/CRC.
   Google Scholar
• Gelman, A. (2014), ““Edlin’s rule” for Routinely Scaling Down Published Estimates – Statistical Modeling, Causal Inference, and Social Science,” available at http://andrewgelman.com/2014/02/24/edlins-rule-routinely-scaling-published-estimates/.
   Google Scholar
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. B. (2013), Bayesian Data Analysis, (3rd ed.), Boca Raton, FL: CRC Press.
   Google Scholar
• Gelman, A., Lee, D., and Guo, J. (2015), “Stan: A Probabilistic Programming Language for Bayesian Inference and Optimization,” Journal of Educational and Behavioral Statistics, 40, 530–543. DOI: 10.3102/1076998615606113.
   Web of Science ®Google Scholar
• Gelman, A. and Stern, H. (2006), “The Difference Between ‘Significant’ and ‘Not Significant’ is not Itself Statistically Significant,” The American Statistician, 60, 328–331. DOI: 10.1198/000313006X152649.
   Web of Science ®Google Scholar
• Gilbert, D. T., King, G., Pettigrew, S., and Wilson, T. D. (2016), Comment on “Estimating the Reproducibility of Psychological Science,” Science, 351, 1037–1037. DOI: 10.1126/science.aad7243.
   PubMed Web of Science ®Google Scholar
• Gronau, Q. F., Sarafoglou, A., Matzke, D., Ly, A., Boehm, U., Marsman, M., Leslie, D. S., Forster, J. J., Wagenmakers, E.-J., and Steingroever, H. (2017), “A Tutorial on Bridge Sampling,” Journal of Mathematical Psychology, 81, 80–97. DOI: 10.1016/j.jmp.2017.09.005.
   PubMed Web of Science ®Google Scholar
• Harms, C. (2018), “A Bayes Factor for Replications of ANOVA Results,” available at https://doi.org/10.17605/OSF.IO/JV39H. DOI: 10.1080/00031305.2018.1518787.
   Google Scholar
• Ionides, E. L., Giessing, A., Ritov, Y., and Page, S. E. (2017), “Response to the ASA’s Statement on p -Values: Context, Process, and Purpose,” The American Statistician, 71, 88–89. DOI: 10.1080/00031305.2016.1234977.
   Web of Science ®Google Scholar
• JASP Team (2018), “JASP (Version 0.8.6) [Computer software].”
   Google Scholar
• Jeffreys, H. (1961), Theory of Probability, (3rd ed.), Oxford, UK: Clarendon Press.
   Google Scholar
• Joy-Gaba, J. A., Clay, R., and Cleary, H. (2016), “Replication of Williams & Bargh (2008, PS, Study 4),” available at https://osf.io/vnsqg.
   Google Scholar
• Kass, R. E. and Raftery, A. E. (1995), “Bayes Factors,” Journal of the American Statistical Association, 90, 773–795. DOI: 10.1080/01621459.1995.10476572.
   Web of Science ®Google Scholar
• Kruschke, J. K. (2015), “Introduction: Credibility, Models, and Parameters,” in Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan (2nd ed.), Waltham, MA: Academic Press/Elsevier, chap. 2, pp. 15–32.
   Google Scholar
• Lecoutre, B. (1999), “Two Useful Distributions for Bayesian Predictive Procedures under Normal Models,” Journal of Statistical Planning and Inference, 79, 93–105. DOI: 10.1016/S0378-3758(98)00231-6.
   Web of Science ®Google Scholar
• Lindley, D. V. (1993), “The Analysis of Experimental Data: The Appreciation of Tea and Wine,” Teaching Statistics, 15, 22–25. DOI: 10.1111/j.1467-9639.1993.tb00252.x.
   Google Scholar
• Ly, A., Etz, A., Marsman, M., and Wagenmakers, E.-J. (2018), “Replication Bayes Factors from Evidence Updating,” Behavior Research Methods, available at https://doi.org/10.3758/s13428-018-1092-x.
   Google Scholar
• Marsman, M., Schönbrodt, F. D., Morey, R. D., Yao, Y., Gelman, A., and Wagenmakers, E.-J. (2017), “A Bayesian Bird’s Eye View of ‘Replications of Important Results in Social Psychology’,” Royal Society Open Science, 4.
   Web of Science ®Google Scholar
• Maxwell, S. E., Lau, M. Y., and Howard, G. S. (2015), “Is Psychology Suffering from a Replication Crisis? What does ‘Failure to Replicate’ Really Mean?” American Psychologist, 70, 487–498. DOI: 10.1037/a0039400.
   PubMed Web of Science ®Google Scholar
• McElreath, R. (2016), Statistical Rethinking: A Bayesian Course with Examples in R and Stan, Boca Raton, FL: CRC Press.
   Google Scholar
• Meng, X. and Wing, H. (1996), “Simulating Ratios of Normalizing Constants via a Simple Identity: A Theoretical Exploration,” Statistica Sinica, 6, 831–860.
   Web of Science ®Google Scholar
• Morey, R. D. and Lakens, D. (2016), “Why Most of Psychology is Statistically Unfalsifiable,” available at http://doi.org/10.5281/zenodo.838685.
   Google Scholar
• Morey, R. D., Romeijn, J.-W., and Rouder, J. N. (2016), “The Philosophy of Bayes Factors and the Quantification of Statistical Evidence,” Journal of Mathematical Psychology. DOI: 10.1016/j.jmp.2015.11.001.
   Web of Science ®Google Scholar
• Nickerson, R. S. (2000), “Null Hypothesis Significance Testing: A Review of an old and Continuing Controversy,” Psychological Methods, 5, 241–301. DOI: 10.1037/1082-989X.5.2.241.
   PubMed Web of Science ®Google Scholar
• Open Science Collaboration (2015), “Estimating the Reproducibility of Psychological Science,” Science, 349, aac4716–aac4716.
   PubMed Web of Science ®Google Scholar
• Pashler, H., and Wagenmakers, E. (2012), “Editors’ Introduction to the Special Section on Replicability in Psychological Science,” Perspectives on Psychological Science, 7, 528–530. DOI: 10.1177/1745691612465253.
   Web of Science ®Google Scholar
• Rosenthal, R., Rosnow, R. L., and Rubin, D. B. (2000), Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach, Cambridge University Press.
   Google Scholar
• Rouder, J. N., and Morey, R. D. (2012), “Default Bayes Factors for Model Selection in Regression,” Multivariate Behavioral Research, 47, 877–903. DOI: 10.1080/00273171.2012.734737.
   PubMed Web of Science ®Google Scholar
• Rouder, J. N., Morey, R. D., Speckman, P. L., and Province, J. M. (2012), “Default Bayes factors for ANOVA designs,” Journal of Mathematical Psychology, 56, 356–374. DOI: 10.1016/j.jmp.2012.08.001.
   Web of Science ®Google Scholar
• Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., and Iverson, G. (2009), “Bayesian t Tests for Accepting and Rejecting the Null Hypothesis,” Psychonomic Bulletin & Review, 16, 225–237. DOI: 10.3758/PBR.16.2.225.
   PubMed Web of Science ®Google Scholar
• Schönbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., and Perugini, M. (2017), “Sequential Hypothesis Testing With Bayes Factors: Efficiently Testing Mean Differences,” Psychological Methods, 22, 322–339. DOI: 10.1037/met0000061.
   PubMed Web of Science ®Google Scholar
• Sedlmeier, P., and Gigerenzer, G. (1989), “Do Studies of Statistical Power have an Effect on the Power of Studies?” Psychological Bulletin, 105, 309–316. DOI: 10.1037/0033-2909.105.2.309.
   Web of Science ®Google Scholar
• Simonsohn, U. (2015), “Small Telescopes: Detectability and the Evaluation of Replication Results,” Psychological Science, 26, 559–569. DOI: 10.1177/0956797614567341.
   PubMed Web of Science ®Google Scholar
• Steiger, J. H. (2004), “Beyond the F Test: Effect Size Confidence Intervals and Tests of Close Fit in the Analysis of Variance and Contrast Analysis,” Psychological Methods, 9, 164–182. DOI: 10.1037/1082-989X.9.2.164.
   PubMed Web of Science ®Google Scholar
• Szucs, D., and Ioannidis, J. P. A. (2017), “Empirical Assessment of Published Effect Sizes and Power in the Recent Cognitive Neuroscience and Psychology Literature,” PLOS Biology, 15, e2000797. DOI: 10.1371/journal.pbio.2000797.
   PubMed Web of Science ®Google Scholar
• Verhagen, J., and Wagenmakers, E.-J. (2014), “Bayesian Tests to Quantify the Result of a Replication Attempt,” Journal of Experimental Psychology: General, 143, 1457–1475. DOI: 10.1037/a0036731.
   PubMed Web of Science ®Google Scholar
• Voracek, M., and Sonnleitner, C. (2016), “Replication of D Albarracn, IM Handley, K Noguchi, KC McCulloch, H Li, J Leeper, RD Brown, A Earl, WP Hart (2008, JPSP 95(3), Exp. 7),”.
   Google Scholar
• Wasserstein, R. L., and Lazar, N. A. (2016), “The ASA’s Statement on p -Values: Context, Process, and Purpose,” The American Statistician, 70, 129–133. DOI: 10.1080/00031305.2016.1154108.
   Web of Science ®Google Scholar
• Williams, L. E., and Bargh, J. a. (2008), “Keeping One’s Distance: The Influence of Spatial Distance Cues on Affect and Evaluation,” Psychological Science, 19, 302–308. DOI: 10.1111/j.1467-9280.2008.02084.x.
   PubMed Web of Science ®Google Scholar