Variation and Covariation in Large-Scale Replication Projects: An Evaluation of Replicability

References

• Baribault, B., Donkin, C., Little, D. R., Trueblood, J., Oravecz, Z., van Ravenzwaaij, D., White, C., De Boeck, P., and Vandekerckhove, J. (2018), “Meta-Studies for Robust Tests of Theory,” Proceedings of the National Academy of Sciences, 115, 2607–2612. DOI: 10.1073/pnas.1708285114.
   PubMed Web of Science ®Google Scholar
• Begley, C. G., and Ellis, L. M. (2012), “Raise Standards for Preclinical Cancer Research,” Nature, 483, 531–533. DOI: 10.1038/483531a.
   PubMed Web of Science ®Google Scholar
• Berkey, C. S., Hoaglin, D. C., Antczak-Bouckoms, A., Mosteller, F., and Colditz, G. A. (1998), “Meta-Analysis of Multiple Outcomes by Regression with Random Effects,” Statistics in Medicine, 17, 2537–2550, 1998. DOI: 10.1002/(SICI)1097-0258(19981130)17:22<2537::AID-SIM953>3.0.CO;2-C.
   PubMed Web of Science ®Google Scholar
• Bourassa, D., McManus, I., and Bryden, M. (1996), “Handedness and Eye-Dominance: A Meta-Analysis of their Relationship,” Laterality, 1, 5–34. DOI: 10.1080/713754206.
   PubMedGoogle Scholar
• Box, G. E. P., and Cox, D. R. (1964), “An Analysis of Transformations,” Journal of the Royal Statistical Society, Series B, 26, 211–252. DOI: 10.1111/j.2517-6161.1964.tb00553.x.
   Google Scholar
• Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., and Wu, H. (2016), “Evaluating Replicability of Laboratory Experiments in Economics,” Science, 351, 1433–1436. DOI: 10.1126/science.aaf0918.
   PubMed Web of Science ®Google Scholar
• Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikenstein, E., Hummer, L., Imai, T., Isaksson, S., Manfredi, D., Rose, J., Wagenmakers, E.-J., and Wu, H. (2018), “Evaluating the Replicability of Social Science Experiments in Nature and Science between 2010 and 2015,” Nature Human Behaviour, 2, 637–644. DOI: 10.1038/s41562-018-0399-z.
   PubMed Web of Science ®Google Scholar
• Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., and Riddell, A. (2017), “Stan: A Probabilistic Programming Language,” Journal of Statistical Software, 76, 1–32. DOI: 10.18637/jss.v076.i01.
   PubMed Web of Science ®Google Scholar
• Colling, L. J., Szűcs, D., De Marco, D., Cipora, K., Ulrich, R., Nuerk, H.-C., Soltanlou, M., Bryce, D., Chen, S.-C., Schroeder, P. A., Henare, D. T., Chrystall, C. K., Corballis, P. M., Ansari, D., Goffin, C., Sokolowski, H. M., Hancock, P. J. B., Millen, A. E., Langton, S. R. H., Holmes, K. J., Saviano, M. S., Tummino, T. A., Lindemann, O., Zwaan, R. A., Lukavský, J., Beckov, A., Vranka, M. A., Cutini, S., Mammarella, I. C., Mulatti, C., Bell, R., Buchner, A., Mieth, L., Rer, J. P., Klein, E., Huber, S., Moeller, K., Ocampo, B., Lupiáñez, J., Ortiz-Tudela, J., de la Fuente, J., Santiago, J., Ouellet, M., Hubbard, E. M., Toomarian, E. Y., Job, R., Treccani, B., McShane, B. B. (2020), “Registered Replication Report on Fischer, Castel, Dodd, and Pratt (2003),” Advances in Methods and Practices in Psychological Science, 3, 143–162. DOI: 10.1177/2515245920903079.
   Google Scholar
• DeKay, M. L., Rubinchik, N., Li, Z., and DeBoeck, P. (2022), “Accelerating Psychological Science with Metastudies: A Demonstration Using the Rrisky-Choice Framing Effect,” Perspectives on Psychological Science, forthcoming.
   PubMed Web of Science ®Google Scholar
• Erosheva, E. A., and Curtis, S. M. (2017), “Dealing with Reflection Invariance in Bayesian Factor Analysis,” Psychometrika, 82, 295–307. DOI: 10.1007/s11336-017-9564-y.
   PubMed Web of Science ®Google Scholar
• Fabrigar, L. R., and Wegener, D. T. (2016), “Conceptualizing and Evaluating the Replication of Research Results,” Journal of Experimental Social Psychology, 66, 68–80. DOI: 10.1016/j.jesp.2015.07.009.
   Web of Science ®Google Scholar
• Fischer, M. H., and Brugger, P. (2011), “When Digits Help Digits: Spatial–Numerical Associations Point to Finger Counting as Prime Example of Embodied Cognition,” Frontiers in Psychology, 2, 260. DOI: 10.3389/fpsyg.2011.00260.
   PubMed Web of Science ®Google Scholar
• Fischer, M. H., Castel, A. D., Dodd, M. D., and Pratt, J. (2003), “Perceiving Numbers Causes Spatial Shifts of Attention,” Nature Neuroscience, 6, 555–556. DOI: 10.1038/nn1066.
   PubMed Web of Science ®Google Scholar
• Fodor, J. A. (1975), The Language of Thought (Vol. 5), Cambridge, MA: Harvard University Press.
   Google Scholar
• Galton, F. (1880), “Visualized Numerals,” Nature, 21, 252–256. DOI: 10.1038/021252a0.
   Google Scholar
• Gelman, A., Van Dyk, D. A., Huang, Z., and Boscardin, J. W. (2008), “Using Redundant Parameterizations to Fit Hierarchical Models,” Journal of Computational and Graphical Statistics, 17, 95–122. DOI: 10.1198/106186008X287337.
   Web of Science ®Google Scholar
• Gładziejewski, P., and Miłkowski, M. (2017), “Structural Representations: Causally Relevant and Different from Detectors,” Biology & Philosophy, 32, 337–355.
   PubMed Web of Science ®Google Scholar
• Kalaian, H. A., and Raudenbush, S. W. (1996), “A Multivariate Mixed Linear Model for Meta-Analysis,” Psychological Methods, 1, 227–235. DOI: 10.1037/1082-989X.1.3.227.
   Web of Science ®Google Scholar
• Kay, A. C., and Jost, J. T. (2003), “Complementary Justice: Effects of “Poor but Happy” and “Poor but Honest” Stereotype Exemplars on System Justification and Implicit Activation of the Justice Motive,” Journal of Personality and Social Psychology, 85, 823–837.
   PubMed Web of Science ®Google Scholar
• Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., Hasselman, F., Hicks, J. A., Hovermale, J. F., Hunt, S. J., Huntsinger, J. R., IJzerman, H., John, M.-S., Joy-Gaba, J. A., Barry Kappes, H., Krueger, L. E., Kurtz, J., Levitan, C. A., Mallett, R. K., Morris, W. L., Nelson, A. J., Nier, J. A., Packard, G., Pilati, R., Rutchick, A. M., Schmidt, K., Skorinko, J. L., Smith, R., Steiner, T. G., Storbeck, J., Van Swol, L. M., Thompson, D., van ’t Veer, A. E., Vaughn, L. A., Vranka, M., Wichman, A. L., Woodzicka, J. A., and Nosek, B. A. (2014), “Investigating Variation in Replicability: A “Many Labs” Replication Project,” Social Psychology, 45, 142–152. DOI: 10.1027/1864-9335/a000178.
   Web of Science ®Google Scholar
• McCulloch, R., and Rossi, P. E. (1994), “An Exact Likelihood Analysis of the Multinomial Probit Model,” Journal of Econometrics, 64, 207–240. DOI: 10.1016/0304-4076(94)90064-7.
   Web of Science ®Google Scholar
• McShane, B. B., and Böckenholt, U. (2018), “Multilevel Multivariate Meta-Analysis with Application to Choice Overload,” Psychometrika, 83, 255–271. DOI: 10.1007/s11336-017-9571-z.
   PubMed Web of Science ®Google Scholar
• McShane, B. B., Tackett, J. L., Böckenholt, U., and Gelman, A. (2019), “Large Scale Replication Projects in Contemporary Psychological Research,” The American Statistician, 73, 99–105. DOI: 10.1080/00031305.2018.1505655.
   Web of Science ®Google Scholar
• Muthén, B. O. (1994), “Multilevel Covariance Structure Analysis,” Sociological Methods & Research, 22, 376–398.
   Web of Science ®Google Scholar
• Newell, A., and Simon, H. A. (1976), “Computer Science as Empirical Enquiry: Symbols and Search,” Communications of the ACM, 19, 113–126. DOI: 10.1145/360018.360022.
   Web of Science ®Google Scholar
• Open Science Collaboration. (2015), “Estimating the Reproducibility of Psychological Science,” Science, 349, aac4716.
   PubMed Web of Science ®Google Scholar
• Prinz, F., Schlange, T., and Asadullah, K. (2011), “Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets,” Nature Reviews Drug Discover, 10, 712. DOI: 10.1038/nrd3439–c1.
   PubMed Web of Science ®Google Scholar
• Rabe-Hesketh, S., Skrondal, A., and Pickles, A. (2004), “Generalized Multilevel Structural Equation Modeling,” Psychometrika, 69, 167–190. DOI: 10.1007/BF02295939.
   Web of Science ®Google Scholar
• Stan Development Team (2020), “Stan user’s guide (version 2.24).” Available at http://mc-stan.org/.
   Google Scholar
• Williams, D., and Colling, L. (2018), “From Symbols to Icons: The Return of Resemblance in the Cognitive Neuroscience Revolution,” Synthese, 195, 1941–1967. DOI: 10.1007/s11229-017-1578-6.
   Web of Science ®Google Scholar
• Wilson, M. (2002), “Psychonomic Bulletin & Review,” Six views of Embodied Cognition, 9, 625–636.
   Google Scholar