Large-Scale Replication Projects in Contemporary Psychological Research

References

• Bargh, J. A., Chen, M., and Burrows, L. (1996), “Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action,” Journal of Personality and Social Psychology, 71, 230–244.
   PubMed Web of Science ®Google Scholar
• Bargh, J. A., Gollwitzer, P. M., Lee-Chai, A., Barndollar, K., and Trotschel, R. (2001), “The Automated Will: Nonconscious Activation and Pursuit of Behavioral Goals,” Journal of Personality and Social Psychology, 81, 1014–1027.
   PubMed Web of Science ®Google Scholar
• Baribault, B., Donkin, C., Little, D. R., Trueblood, J., Oravecz, Z., van Ravenzwaaij, D., White, C., De Boeck, P., and Vandekerckhove, J. (2018), “Meta-Studies for Robust Tests of Theory,” Proceedings of the National Academy of Sciences, 115, 2607–2612.
   PubMed Web of Science ®Google Scholar
• Bem, D. J. (2011), “Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect,” Journal of Personality and Social Psychology, 100, 407–425.
   PubMed Web of Science ®Google Scholar
• Bouwmeester, S., Verkoeijen, P. P., Aczel, B., Barbosa, F., Bègue, L., Brañas-Garza, P., Chmura, T. G., Cornelissen, G., Døssing, F. S., and Espín, A. M. (2017), “Registered Replication Report: Rand, Greene, and Nowak (2012),” Perspectives on Psychological Science, 12, 527–542.
   Web of Science ®Google Scholar
• Brandt, M. J., IJzerman, H., Dijksterhuis, A., Farach, F. J., Geller, J., Giner-Sorolla, R., Grange, J. A., Perugini, M., Spies, J. R., and Van’t Veer, A. (2014), “The Replication Recipe: What makes for a Convincing Replication?” Journal of Experimental Social Psychology, 50, 217–224.
   Web of Science ®Google Scholar
• Brown, S. D., Furrow, D., Hill, D. F., Gable, J. C., Power, L. P., and Jacobs, W. J. (2014), “A Duty to Describe: Better the Devil You Know than the Devil You Don’t,” Perspectives on Psychological Science, 9, 626–640.
   Web of Science ®Google Scholar
• Carney, D. R., Cuddy, A. J., and Yap, A. J. (2010), “Power Posing Brief Nonverbal Displays Affect Neuroendocrine Levels and Risk Tolerance,” Psychological Science, 21, 1363–1368.
   PubMed Web of Science ®Google Scholar
• Chernev, A., Böckenholt, U., and Goodman, J. (2015), “Choice Overload: A Conceptual Review and Meta-Analysis,” Journal of Consumer Psychology, 25, 333–358.
   Web of Science ®Google Scholar
• Cheung, I., Campbell, L., LeBel, E. P., Ackerman, R., Aykutoğlu, B., Bahník, Š., Bowen, J., Bredow, C., Bromberg, C., and Caprariello, P. (2016), “Registered Replication Report: Study 1 From Finkel, Rusbult, Kumashiro, & Hannon (2002),” Perspectives on Psychological Science, 11, 750–764.
   Web of Science ®Google Scholar
• Chung, Y., Rabe-Hesketh, S., and Choi, I.-H. (2013a), “Avoiding Zero Between-Study Variance Estimates in Random-Effects Meta-Analysis,” Statistics in Medicine, 32, 4071–4089.
   PubMed Web of Science ®Google Scholar
• Chung, Y., Rabe-Hesketh, S., Dorie, V., Gelman, A., and Liu, J. (2013b), “A Non-Degenerate Estimator for Hierarchical Variance Parameters via Penalized Likelihood Estimation,” Psychometrika, 78, 685–709.
   PubMed Web of Science ®Google Scholar
• Eerland, A., Sherrill, A. M., Magliano, J. P., Zwaan, R. A., Arnal, J., Aucoin, P., Berger, S., Birt, A., Capezza, N., and Carlucci, M. (2016), “Registered Replication Report: Hart & Albarracín (2011),” Perspectives on Psychological Science, 11, 158–171.
   Web of Science ®Google Scholar
• Ehrenberg, A. S. and Bound, J. A. (1993), “Predictability and Prediction,” Journal of the Royal Statistical Society, Series A, 156, 167–206.
   Google Scholar
• Fabrigar, L. R. and Wegener, D. T. (2016), “Conceptualizing and Evaluating the Replication of Research Results,” Journal of Experimental Social Psychology, 66, 68–80.
   Web of Science ®Google Scholar
• Gilovich, T., Griffin, D., and Kahneman, D. (2002), Heuristics and Biases: The Psychology of Intuitive Judgment, Cambridge, UK: Cambridge University Press.
   Google Scholar
• Hagger, M. S., Chatzisarantis, N. L., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., Brand, R., Brandt, M. J., Brewer, G., and Bruyneel, S. (2016), “A Multilab Preregistered Replication of the Ego-Depletion Effect,” Perspectives on Psychological Science, 11, 546–573.
   Web of Science ®Google Scholar
• Higgins, J. P. and Thompson, S. G. (2002), “Quantifying Heterogeneity in a Meta-Analysis,” Statistics in Medicine, 21, 1539–1558.
   PubMed Web of Science ®Google Scholar
• Huedo-Medina, T. B., Sánchez-Meca, J., Marín-Martínez, F., and Botella, J. (2006), “Assessing Heterogeneity in Meta-Analysis: Q Statistic or I2 Index?” Psychological Methods, 11, 193–206.
   PubMed Web of Science ®Google Scholar
• Ioannidis, J. P., Patsopoulos, N. A., and Evangelou, E. (2007), “Uncertainty in Heterogeneity Estimates in Meta-Analyses,” BMJ, 335, 914–916.
   PubMed Web of Science ®Google Scholar
• Jacowitz, K. E. and Kahneman, D. (1995), “Measures of Anchoring in Estimation Tasks,” Personality and Social Psychology Bulletin, 21, 1161–1166.
   Web of Science ®Google Scholar
• Kennedy, L. and Gelman, A. (2018), “How the Recognition of Constraints on Generality can Provide a Direction for Generalizing Research Findings,” Columbia University Technical Report.
   Google Scholar
• Klein, R. A., Ratliff, K., Nosek, B. A., Vianello, M., Pilati, R., Devos, T., Galliani, E. M., Brandt, M., van ’t Veer, A., Rutchick, A. M., Schmidt, K., Bahnik, S., Vranka, M., IJzerman, H., Hasselman, F., Joy-Gaba, J., Chandler, J. J., Vaughn, L. A., Brumbaugh, C., van swol, L., Wichman, A., Packard, G., Brooks, B., Cemalcilar, Z., Storbeck, J., Bocian, K., Levitan, C., Bernstein, M. J., Krueger, L. E., Eisner, M., Davis, W. E., Nier, J. A., Nelson, A. J., Steiner, T. G., Mallett, R., Thompson, D., Huntsinger, J. R., Morris, W., Skorinko, J., and Kappes, H. (2014), “Investigating Variation in replicability: A “Many Labs” Replication Project,” Social Psychology, 45, 142–152.
   Web of Science ®Google Scholar
• Lindsay, R. M. and Ehrenberg, A. S. (1993), “The Design of Replicated Studies,” The American Statistician, 47, 217–228.
   Web of Science ®Google Scholar
• McShane, B. B. and Böckenholt, U. (2014), “You Cannot Step Into the Same River Twice: When Power Analyses Are Optimistic,” Perspectives on Psychological Science, 9, 612–625.
   Web of Science ®Google Scholar
• ——— (2017), “Single Paper Meta-analysis: Benefits for Study Summary, Theory-testing, and Replicability,” Journal of Consumer Research, 43, 1048–1063.
   Web of Science ®Google Scholar
• ——— (2018), “Multilevel Multivariate Meta-analysis with Application to Choice Overload,” Psychometrika, 83, 255–271.
   PubMed Web of Science ®Google Scholar
• McShane, B. B., Böckenholt, U., and Hansen, K. T. (2016), “Adjusting for Publication Bias in Meta-Analysis: An Evaluation of Selection Methods and Some Cautionary Notes,” Perspectives on Psychological Science, 11, 730–749.
   Web of Science ®Google Scholar
• Open Science Collaboration (2015), “Estimating the Reproducibility of Psychological Science,” Science, 349, aac4716.
   PubMed Web of Science ®Google Scholar
• Park, D. K., Gelman, A., and Bafumi, J. (2004), “Bayesian Multilevel Estimation with Poststratification: State-Level Estimates from National Polls,” Political Analysis, 12, 375–385.
   Web of Science ®Google Scholar
• Pigott, T. (2012), Advances in Meta-Analysis, New York: Springer.
   Google Scholar
• Rosenthal, R. (1991), Replication Research in the Social Sciences, Newbury Park, CA: Sage, chap. Replication in Behavioral Sciences, pp. 1–30.
   Google Scholar
• Simons, D. J., Holcombe, A. O., and Spellman, B. A. (2014), “An Introduction to Registered Replication Reports at Perspectives on Psychological Science,” Perspectives on Psychological Science, 9, 552–555.
   Web of Science ®Google Scholar
• Simons, D. J., Shoda, Y., and Lindsay, D. S. (2017), “Constraints on Generality (COG): A Proposed Addition to all Empirical Papers,” Perspectives on Psychological Science, 12, 1123–1128.
   Web of Science ®Google Scholar
• Stanley, T., Carter, E. C., and Doucouliagos, H. (2017), “What Meta-Analyses Reveal about the Replicability of Psychological Research.” Tech. Rep., Deakin Laboratory for the Meta-Analysis of Research.
   Google Scholar
• Stroebe, W. and Strack, F. (2014), “The alleged crisis and the Illusion of Exact Replication,” Perspectives on Psychological Science, 9, 59–71.
   Web of Science ®Google Scholar
• Tackett, J. L., Lilienfeld, S. O., Patrick, C. J., Johnson, S. L., Krueger, R. F., Miller, J. D., Oltmanns, T. F., and Shrout, P. E. (2017), “It’s Time to Broaden the Replicability Conversation: Thoughts for and from Clinical Psychological Science,” Perspectives on Psychological Science, 12, 742–756.
   Web of Science ®Google Scholar
• Tsang, E. W. and Kwan, K.-M. (1999), “Replication and Theory Development in Organizational Science: A Critical Realist Perspective,” Academy of Management Review, 24, 759–780.
   Web of Science ®Google Scholar
• van Erp, S., Verhagen, J., Grasman, R. P. P. P., and Wagenmakers, E.-J. (2017), “Estimates of Between-Study Heterogeneity for 705 Meta-Analyses Reported in Psychological Bulletin From 1990–2013,” Journal of Open Psychology Data, 5.
   Google Scholar
• Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams Jr, R., Albohn, D., Allard, E., Benning, S., and Blouin-Hudon, E.-M. (2016), “Registered Replication Report: Strack, Martin, & Stepper (1988),” Perspectives on Psychological Science, 11, 917–928.
   Web of Science ®Google Scholar