What Can We Learn from Many Labs Replications?

References

• Alogna, V. K., Attaya, M. K., Aucoin, P., Bahník, Š., Birch, S., Birt, A. R., … Zwaan, R. A. (2014). Registered replication report: Schooler and Engstler-Schooler (1990). Perspectives on Psychological Science, 9(5), 556–578. doi:10.1177/1745691614545653
   Web of Science ®Google Scholar
• Aronson, E., & Mills, J. (1959). The effect of severity of initiation on liking for a group. Journal of Abnormal and Social Psychology, 19, 177–181. doi:10.1037/h0047195
   Web of Science ®Google Scholar
• Bargh, J. A., Chen, M., & Burrows, L. (1996). Automaticity of social behaviour: Direct effects of trait construct and stereotype activation. Journal of Personality and Social Psychology, 71(2), 230–244. doi:10.1037//0022-3514.71.2.230
   PubMed Web of Science ®Google Scholar
• Bouwmeester, S., Verkoeijen, P. P. J. L., Aczel, B., Barbosa, F., Bègue, L., Brañas-Garza, P., … Wollbrant, C. E. (2017). Registered replication report: Rand, Greene and Nowak (2012). Perspectives on Psychological Science, 12(3), 527–542. doi:10.1177/1745691617693624
   Web of Science ®Google Scholar
• Broad, W. J., & Wade, N. (1982). Betrayers of the truth. New York, NY: Simon & Schuster.
   Google Scholar
• Cacioppo, J. E., Petty, R. E., & Morris, K. J. (1983). Effects of need for cognition on message evaluation, recall, and persuasion. Journal of Personality and Social Psychology, 45(4), 805–818. doi:10.1037//0022-3514.45.4.805
   Web of Science ®Google Scholar
• Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., … Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644. doi:10.1038/s41562-018-0399-z
   PubMed Web of Science ®Google Scholar
• Chang, J., Zhang, M., Hitchman, G., Qiu, J., & Liu, Y. (2014). When you smile, you become happy: Evidence from resting state task-based fMRI. Biological Psychology, 103, 100–106. doi:10.1016/j.biopsycho.2014.08.003
   PubMed Web of Science ®Google Scholar
• Cheung, I., Campbell, L., LeBel, E. P., Ackerman, R. A., Aykutog˘lu, B., Bahník, Š., … Yong, J. C. (2016). Registered replication report: Study 1 from Finkel, Rusbult, Kumashiro & Hannon (2002). Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 11(5), 750–764. doi:10.1177/1745691616664694
   PubMedGoogle Scholar
• Crandall, C. S., & Sherman, J. (2016). On the scientific superiority of conceptual replications for scientific progress. Journal of Experimental Social Psychology, 66, 91–99. doi:10.1073/pnas.1521897113
   Web of Science ®Google Scholar
• Crocker, J., & Cooper, L. (2011). Editorial: Addressing scientific fraud. Science, 334(6060), 1182. Retrieved from http://www.sciencemag.org.proxy.library.uu.nl/content/334/6060/1182.full doi:10.1126/science.1216775
   PubMed Web of Science ®Google Scholar
• Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145–153.
   PubMed Web of Science ®Google Scholar
• Darwin, C. (1872). The expression of emotions in man and animals. London, UK: John Murray.
   Google Scholar
• DeCoster, J., & Claypool, H. M. (2004). A meta-analysis of priming effects on impression formation supporting a general model of informational bias. Personality and Social Psychology Review, 8(1), 2–27. doi:10.1207/S15327957PSPR0801_1
   PubMed Web of Science ®Google Scholar
• Deutsche Forschungsgemeinschaft (2017). Stellungsnahme zur Replizierbarkeit von Forschungsergebnissen (Position paper on the replicability of research findings). Retrieved from http://www.dfg.de/download/pdf/dfg_im_profil/reden_stellungnahmen/2017/170425_stellungnahme_replizierbarkeit_forschungsergebnisse_de.pdf
   Google Scholar
• Dijksterhuis, A. (2018). Reflection on the professor-priming replication report. Perspectives on Psychological Science, 13, 295–296. doi:10.1177/1745691618755705
   Web of Science ®Google Scholar
• Dijksterhuis, A., & van Knippenberg, A. (1998). The relation between perception and behavior, or how to win a game of Trivial Pursuit. Journal of Personality and Social Psychology, 74(4), 865–877.
   PubMed Web of Science ®Google Scholar
• Dijksterhuis, A., van Knippenberg, A., & Holland, R. (2014). Evaluating behavior priming research: Three observations and a recommendation. Social Cognition, 32(Supplement), 196–208. doi:10.1521/soco.2014.32.supp.196
   Web of Science ®Google Scholar
• Doyen, S., Klein, O., Pichon, C., & Cleeremans, A. (2012). Behavioral priming: It's all in the mind, but whose mind? PLoS One, 7(1), e29081. doi:10.1371/journal.pone.0029081
   PubMed Web of Science ®Google Scholar
• Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6, 621. doi:10.3389/fpsyg.2015.00621
   PubMed Web of Science ®Google Scholar
• Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., … Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68–82. doi:10.1016/j.jesp.2015.10.012
   Web of Science ®Google Scholar
• Ebersole, C. R., Alaei, R., Atherton, O. E., Bernstein, M. J., Brown, M., Chartier, C. R., … Nosek, B. A. (2017). Observe, hypothesize, test, repeat: Luttrell, Petty and Xu (2017) demonstrating good science. Journal of Experimental Social Psychology, 69, 184–186. doi:10.1016/j.jesp.2016.12.005
   Web of Science ®Google Scholar
• Eerland, A., Sherrill, A. M., Magliano, J. P., Zwaan, R. A., Arnal, J. D., Aucoin, P., … Prenoveau, J. M. (2016). Registered replication report: Hart & Albarracín (2011)). Perspectives on Psychological Science : A Journal of the Association for Psychological Science, 11(1), 158–171. doi:10.1177/1745691615605826
   PubMedGoogle Scholar
• Erdfelder, E., & Ulrich, R. (2018). Zur Methodologie von Replikationsstudien. (The methodology of replication studies). Psychologische Rundschau, 69(1), 3–21. doi:10.1026/0033-3042/a000387
   Web of Science ®Google Scholar
• Fiedler, K., & Prager, J. (2018). The regression trap and other pitfalls of replication science illustrated by the report of the Open Science Collaboration. Basic and Applied Social Psychology, 40(3), 115–125. doi:10.1080/01973533.2017.1421953
   Web of Science ®Google Scholar
• Fiedler, K., & Schwarz, N. (2016). Questionable research practices revisited. Social Psychological and Personality Science, 7(1), 45–52. doi:10.1177/1948550615612150
   Web of Science ®Google Scholar
• Fraley, R. C., & Vazire, S. (2014). The N-Pact factor: evaluating the quality of empirical journals with respect to sample size and statistical power. PLos One, 9(10), e109019doi:10.1371/journal.pone.0109019
   PubMed Web of Science ®Google Scholar
• Gadenne, V. (1984). Theorie und Erfahrung in der psychologischen Forschung (Theory and experience in psychological research). Tübingen, Germany: Mohr & Siebeck.
   Google Scholar
• Gerard, H. B., & Mathewson, G. C. (1966). Effects of the severity of initiation for a group – replication. Journal of Experimental Social Psychology, 2(3), 278–287. doi:10.1016/0022-1031(66)90084-9
   Web of Science ®Google Scholar
• Gergen, K. J. (1973). Social psychology as history. Journal of Personality and Social Psychology, 26(2), 309–320. doi:10.1037/h0034436
   Web of Science ®Google Scholar
• Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comments on “Estimating the reproducibility of psychological science”. Science, 351(6277), 1037. doi:10.1126/science.aad7243
   PubMed Web of Science ®Google Scholar
• Hagger, M. S., Chatzisarantis, N. L. D., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., … Zwienenberg, M. (2016). A multilab pregegistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11(4), 546–573. doi:10.1177/1745691616652873
   Web of Science ®Google Scholar
• Hennenlotter, A., Dresel, C., Castrop, F., Ceballos-Baumann, A. O., Wohlschläger, A. M., & Haslinger, B. (2009). The link between facial feedback and neural activity within central circuitries of emotion – new insights from botulin toxin induced denervation of frown muscles. Cerebral Cortex, 19(3), 537–542. doi:10.1093/cercor/bhn104
   PubMed Web of Science ®Google Scholar
• Inbar, Y. (2016). Association between contextual dependence and replicability in psychology may be spurious. Proceedings of the National Academy of Sciences, 113(34), E4933–E4934. doi:10.1073/pnas.1608676113
   PubMed Web of Science ®Google Scholar
• John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. doi:10.1177/0956797611430953
   PubMed Web of Science ®Google Scholar
• Klein, R. A. et al. (2018). Many Labs 2: Investigating variation in replicability across samples and setting. Advances in Methods and Practices in Psychological Science. 1(4), 443–490. doi:10.31234/0sf.io
   Google Scholar
• Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B., Bahník, Š., Bernstein, M. J., … Nosek, B. A. (2014). Investigating variation in replicability. A “Many Labs” replication project. Social Psychology, 45(3), 142–152. doi:10.1027/1864-9335/a000178
   Web of Science ®Google Scholar
• Levelt, N. & Drenth Committees (2012). Flawed science: The fraudulent research practices of social psychologist Diederik Stapel. Retrieved from https://www.tilburguniversity.edu/upload/3ff904d7-547b-40ae-85fe-bea38e05a34a_Final%20report%20Flawed%20Science.pdf
   Google Scholar
• Luttrell, A., Petty, R. E., & Xu, M. (2017). Replication and fixing failed replications: The case of need for cognition and argument quality. Journal of Experimental Social Psychology, 69, 178–183. doi:10.1016/j.jesp.2016.09.006
   Web of Science ®Google Scholar
• Marsh, A. A., Rhoads, S. A., & Ryan, R. (2018). A multi-semester classroom demonstration yields evidence in support of the facial feedback effect. Emotion. doi:10.1037/emo0000532
   Web of Science ®Google Scholar
• Noah, T., Schul, Y., & Mayo, R. (2018). When both the original study and its failed replication are correct: Feelings observed cancels the facial-feedback effect. Journal of Personality and Social Psychology, 114(5), 657–664. doi:10.1037/pspa0000121
   PubMed Web of Science ®Google Scholar
• Nosek, B. A., & Lakens, D. (2014). A method to increase the credibility of published results. Social Psychology, 45(3), 137–141.
   Web of Science ®Google Scholar
• O’Donnell, M. (2018). Registered replication report: Dijksterhuis and van Knippenberg (1998). Perspectives on Psychological Science, 13, 268–294. doi:10.1177/1745691618755704
   Web of Science ®Google Scholar
• Open Science Collaboration (OSC; (2015). Estimating the reproducibility of psychological science. Science, 349(6251)
   Web of Science ®Google Scholar
• Payne, B. K., Burkley, M. A., & Stokes, M. B. (2008). Why do implicit and explicit attitude tests diverge? The role of structural fit. Journal of Personality and Social Psychology, 94(1), 16–31. doi:10.1037/0022-3514.94.1.16
   PubMed Web of Science ®Google Scholar
• Petty, R. E., & Cacioppo, J. T. (1986). Communication and persuasion: Central and peripheral routes to attitude change. New York, NY: Springer.
   Google Scholar
• Petty, R. E., & Cacioppo, J. T. (2016). Methodological choices have predictable consequences in replicating studies on motivation to think: Commentary on Ebersole et al. (2016). Journal of Experimental Social Psychology, 67, 86–87. doi:10.1016/j.jesp.2015.12.007
   Web of Science ®Google Scholar
• Petty, R. E., Cacioppo, J. T., & Goldman, R. (1981). Personal involvement as a determinant of argument-based persuasion. Journal of Personality and Social Psychology, 41(5), 847–855. doi:10.1037//0022-3514.41.5.847
   Web of Science ®Google Scholar
• Popper, K. R. (1959). The logic of scientific discovery. London, UK: Hutchinson.
   Google Scholar
• Price, T., & Harmon-Jones, E. (2015). Embodied emotion: The influence of manipulated facial and bodily states on emotive responses. Wiley Interdisciplinary Reviews: Cognitive Science, 6(6), 461–473. doi:10.1002/wcs.1370
   PubMed Web of Science ®Google Scholar
• Risen, J. L., & Gilovich, T. (2008). Why people are reluctant to tempt fate. Journal of Personality and Social Psychology, 95(2), 293–307. doi:10.1037/0022-3514.95.2.293
   PubMed Web of Science ®Google Scholar
• Roediger, H. L. (2012). Psychology’s woes and a partial cure: The value of replication. Observer. Retrieved from http://www.psychologicalscience.org/index.php/publications/observer/2012/february-12/psychologys-woes-and-a-partial-cure-the-value-of-replication.html
   Google Scholar
• Schweinsberg, M., Madan, N., Vianello, M., Sommer, S. A., Jordan, J., Tierney, W., … Uhlmann, E. L. (2016). The pipeline project: Prepublication independent replications of a single laboratory’s research pipeline. Journal of Experimental Social Psychology, 66, 55–67. doi:10.1016/j.jesp.2015.10.001
   Web of Science ®Google Scholar
• Shnabel, N., & Nadler, A. (2008). A needs-based model of reconciliation: Satisfying the differential emotional needs of victim and perpetrator as a key to promoting reconciliation. Journal of Personality and Social Psychology, 94(1), 116–132. doi:10.1037/0022-3514.94.1.116
   PubMed Web of Science ®Google Scholar
• Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on power studies?. Psychological Bulletin, 105(2), 309–316. doi:10.1037//0033-2909.105.2.309
   Web of Science ®Google Scholar
• Strack, F. (2016). Reflection on the smiling registered replication report. Perspectives on Psychological Science, 11(6), 929–930. doi:10.1177/1745691616674460
   Web of Science ®Google Scholar
• Strack, F. (2017). From data to truth in psychological science. A Personal Perspective. Frontiers in Psychology, 8, 702.
   PubMed Web of Science ®Google Scholar
• Strack, F., Martin, L. I., & Stepper, S. (1988). Inhibiting and facilitating conditions of the human smile: A nonobtrusive test of the facial feedback hypothesis. Journal of Personality and Social Psychology, 54(5), 768–777. doi:10.1037/0022-3514.54.5.768
   PubMed Web of Science ®Google Scholar
• Strack, F., & Stroebe, W. (2017). What have we learned?. What Can we Learn? Behavioral and Brain Sciences, 41, e151.
   Web of Science ®Google Scholar
• Stroebe, W. (2012). The truth about, But Nobody Seems to Care. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 7(1), 54–57. doi:10.1177/1745691611427306
   PubMedGoogle Scholar
• Stroebe, W. (2016a). Are most published social psychological findings false? Journal of Experimental Social Psychology, 66, 134–144. doi:10.1016/j.jesp.2015.09.017
   Web of Science ®Google Scholar
• Stroebe, W. (2016b). Populist science reporting at the Volkskrant. Retrieved from https://dub.uu.nl/nl/opinie/populist-science-reporting-volkskrant
   Google Scholar
• Stroebe, W., Gadenne, V., & Nijstad, B. A. (2018). Do our laws only apply to students: The problem with external validity. Basic and Applied Social Psychology, 40, 384–395. doi:10.1080/01973533.2018.1513362
   Web of Science ®Google Scholar
• Stroebe, W., Postmes, T., & Spears, R. (2012). Scientific misconduct and the myth of self-correction in science. Perspectives on Psychological Science: A Journal of the Association for Psychological Science, 7(6), 670–688. doi:10.1177/1745691612460687
   PubMedGoogle Scholar
• Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9(1), 59–71. doi:10.1177/1745691613514450
   Web of Science ®Google Scholar
• Trafimow, D. (2012). The role of auxiliary assumptions for the validity of manipulations and measures. Theory & Psychology, 22, 486–498. doi:10.1177/0959354311429996
   Web of Science ®Google Scholar
• Trafimow, D. (2018). An a priori solution to the replication crisis. Philosophical Psychology, 31(8), 1188–1214. doi:10.1080/09515089.2018.1490707
   Web of Science ®Google Scholar
• Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences, USA, 113(23), 6454–6459.
   PubMed Web of Science ®Google Scholar
• Wagenmakers, E.-J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams, R. B., … Zwaan, R. A. (2016). Registered replication report: Strack, Martin, and Stepper (1988). Perspectives on Psychological Science, 11(6), 917–928. doi:10.1177/1745691616674458
   Web of Science ®Google Scholar
• Yong, E. (2012). Nobel Laureate challenges psychologists to clean up their act. Nature. Retrieved from https://www.nature.com/news/nobel-laureate-challenges-psychologists-to-clean-up-their-act-1.11535
   Google Scholar