A call for replications of addiction research: which studies should we replicate and what constitutes a ‘successful’ replication?

References

• Alexander BK, Coambs RB, Hadaway PF. 1978. The effect of housing and gender on morphine self-administration in rats. Psychopharmacology. 58(2):175–179.
   PubMed Web of Science ®Google Scholar
• Allen C, Mehler DMA. 2019. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 17(5):e3000246.
   PubMed Web of Science ®Google Scholar
• Anvari F, Lakens D. 2019. (pre-print). Using anchor-based methods to determine the smallest effect size of interest.
   Google Scholar
• Bener A, Yildirim E, Torun P, Çatan F, Bolat E, Alıç S, Akyel S, Griffiths MD. 2019. Internet addiction, fatigue, and sleep problems among adolescent students: A large-scale study. Int J Ment Health Addiction. 17(4):959–969.
   Web of Science ®Google Scholar
• Blaszczynski A, Gainsbury SM. 2019. Editor’s note: replication crisis in the social sciences. Int Gambl Stud. 19(3):359–361.
   Web of Science ®Google Scholar
• Brown NJL, Heathers JAJ. 2017. The GRIM Test: a simple technique detects numerous anomalies in the reporting of results in psychology. Soc Psychol Personal Sci. 8(4):363–369.
   Web of Science ®Google Scholar
• Camerer CF, Dreber A, Holzmeister F, Ho TH, Huber J, Johannesson M, Kirchler M, Nave G, Nosek BA, Pfeiffer T. 2018. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav. 2(9):637–644.
   PubMed Web of Science ®Google Scholar
• Chambers C. 2019. What’s next for registered reports? Nature. 573(7773):187–189.
   PubMed Web of Science ®Google Scholar
• Coles NA, Tiokhin L, Scheel AM, Isager PM, Lakens D. 2018. The costs and benefits of replication studies. Behav Brain Sci. 41:e124.
   PubMed Web of Science ®Google Scholar
• Cumming G. 2008. Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspect Psychol Sci. 3(4):286–300.
   PubMedGoogle Scholar
• Cumming G. 2014. The new statistics: why and how. Psychol Sci. 25(1):7–29.
   PubMed Web of Science ®Google Scholar
• Cumming G, Fidler F. 2009. Confidence intervals: Better answers to better questions. Zeitschrift Für Psychologie/J Psychol. 217(1):15–26.
   Web of Science ®Google Scholar
• Cumming G, Maillardet R. 2006. Confidence intervals and replication: where will the next mean fall?. Psychol Methods. 11(3):217–227.
   PubMed Web of Science ®Google Scholar
• Dienes Z. 2014. Using Bayes to get the most out of non-significant results. Front Psychol. 5:781–781.
   PubMed Web of Science ®Google Scholar
• Dienes Z. 2016. How Bayes factors change scientific practice. Bayes Factors for Testing Hypotheses in Psychological Research: Practical Relevance and New Developments. 72:78–89.
   Google Scholar
• Fanelli D. 2010. Do pressures to publish increase scientists’ bias? An empirical support from US States Data. PLoS One. 5(4):e10271–e10271.
   PubMed Web of Science ®Google Scholar
• Field SM, Hoekstra R, Bringmann L, Van Ravenzwaaij D. 2019. When and why to replicate: as easy as 1, 2, 3? Collabra: Psychol. 5(1):46.
   Google Scholar
• Fraley RC, Vazire S. 2014. The N-Pact Factor: evaluating the quality of empirical journals with respect to sample size and statistical power. PLoS One. 9(10):e109019.
   PubMed Web of Science ®Google Scholar
• Fried EI, van Borkulo CD, Cramer AOJ, Boschloo L, Schoevers RA, Borsboom D. 2017. Mental disorders as networks of problems: a review of recent insights. Soc Psychiatry Psychiatr Epidemiol. 52(1):1–10.
   PubMed Web of Science ®Google Scholar
• Gage SH, Sumnall HR. 2019. Rat Park: how a rat paradise changed the narrative of addiction. Addiction. 114(5):917–922.
   PubMed Web of Science ®Google Scholar
• Gelman A. 2018. Don’t characterize replications as successes or failures. Behav Brain Sci. 41:e128.
   PubMed Web of Science ®Google Scholar
• Grønbaek M, Nielsen B. 2007. A randomized controlled trial of Minnesota day clinic treatment of alcoholics. Addiction. 102(3):381–388.
   PubMed Web of Science ®Google Scholar
• Hartgerink CHJ, Wicherts JM, Van Assen MALM. 2017. Too good to be false: nonsignificant results revisited. Collabra: Psychology. 3(1):9.
   Google Scholar
• Hedges LV, Schauer JM. 2019. Statistical analyses for studying replication: meta-analytic perspectives. Psychol Methods. 24(5):557–570.
   PubMed Web of Science ®Google Scholar
• Imperio CG, McFalls AJ, Hadad N, Blanco-Berdugo L, Masser DR, Colechio EM, Coffey AA, Bixler GV, Stanford DR, Vrana KE. 2018. Exposure to environmental enrichment attenuates addiction-like behavior and alters molecular effects of heroin self-administration in rats. Neuropharmacology. 139:26–40.
   PubMed Web of Science ®Google Scholar
• Ioannidis JPA. 2005. Why most published research findings are false. PLoS Med. 2(8):e124.
   PubMed Web of Science ®Google Scholar
• Isager PM. 2018. What to replicate? Justifications of study choice from 85 replication studies. Zenodo. DOI:10.23668/psycharchives.2392
   Google Scholar
• Isager PM. 2019. Quantifying Replication Value: A formula-based approach to study selection in replication research. ZPID (Leibniz Institute for Psychology Information).
   Google Scholar
• Kenny DA, Judd CM. 2019. The unappreciated heterogeneity of effect sizes: implications for power, precision, planning of research, and replication. Psychol Methods. 24(5):578–589.
   PubMed Web of Science ®Google Scholar
• Klein RA, Ratliff KA, Vianello M, Adams RB, Bahník Š, Bernstein MJ, Bocian K, Brandt MJ, Brooks B, Brumbaugh CC. 2014. Investigating variation in replicability. Soc Psychol. 45(3):142–152.
   Web of Science ®Google Scholar
• Klein RA, Vianello M, Hasselman F, Adams BG, Adams RB, Alper S, Aveyard M, Axt JR, Babalola MT, Bahník Š. 2018. Many Labs 2: investigating variation in replicability across samples and settings. Adv Methods Practices Psychol Sci. 1(4):443–490.
   Google Scholar
• Kramer ADI, Guillory JE, Hancock JT. 2014. Experimental evidence of massive-scale emotional contagion through social networks. Proc Natl Acad Sci. 111(24):8788–8790.
   PubMed Web of Science ®Google Scholar
• Kruschke JK. 2011. Bayesian assessment of null values via parameter estimation and model comparison. Perspect Psychol Sci. 6(3):299–312.
   PubMedGoogle Scholar
• Kvarven A, Strømland E, Johannesson M. 2019. Comparing meta-analyses and preregistered multiple-laboratory replication projects. Nat Hum Behav. DOI:10.1038/s41562-019-0787-z
   PubMed Web of Science ®Google Scholar
• Lakens D, Etz AJ. 2017. Too true to be bad: When sets of studies with significant and nonsignificant findings are probably true. Soc Psychol Personal Sci. 8(8):875–881.
   PubMed Web of Science ®Google Scholar
• Lakens D, Scheel AM, Isager PM. 2018. Equivalence testing for psychological research: A tutorial. Adv Methods Practices Psychol Sci. 1(2):259–269.
   Google Scholar
• LaPlante DA. 2019. Replication is fundamental, but is it common? A call for scientific self-reflection and contemporary research practices in gambling-related research. Int Gambl Stud. 19(3):362–368.
   Web of Science ®Google Scholar
• Lilienfeld SO. 2018. Direct replication and clinical psychological science. Behav Brain Sci. 41:e140.
   PubMed Web of Science ®Google Scholar
• Lintzeris N, Bell J, Bammer G, Jolley DJ, Rushworth L. 2002. A randomized controlled trial of buprenorphine in the management of short‐term ambulatory heroin withdrawal. Addiction. 97(11):1395–1404.
   PubMed Web of Science ®Google Scholar
• Ly A, Etz A, Marsman M, Wagenmakers E-J. 2019. Replication Bayes factors from evidence updating. Behav Res. 51(6):2498–2508.
   Web of Science ®Google Scholar
• Mackey A. 2012. Why (or why not), when and how to replicate research. In: G. Porte, editor. Replication research in applied linguistics. Cambridge, UK: Cambridge University Press. p. 34–69.
   Google Scholar
• Makel MC, Plucker JA, Hegarty B. 2012. Replications in psychology research: how often do they really occur? Perspect Psychol Sci. 7(6):537–542.
   Google Scholar
• Maxwell SE, Lau MY, Howard GS. 2015. Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychol. 70(6):487–498.
   PubMed Web of Science ®Google Scholar
• Morey RD, Hoekstra R, Rouder JN, Lee MD, Wagenmakers E-J. 2016. The fallacy of placing confidence in confidence intervals. Psychon Bull Rev. 23(1):103–123.
   PubMed Web of Science ®Google Scholar
• Murphy KR, Myors B, Wolach A. 2014. Statistical power analysis: A simple and general model for traditional and modern hypothesis tests. 4th ed. New York, USA: Routledge/Taylor & Francis Group.
   Google Scholar
• Nosek BA, Errington TM. 2017. Making sense of replications. ELife. 6, e23383.6
   PubMed Web of Science ®Google Scholar
• Nuijten MB, Bakker M, Maassen E, Wicherts JM. 2018. Verify original results through reanalysis before replicating. Behav Brain Sci. 41:e143.
   PubMed Web of Science ®Google Scholar
• Nuijten MB, Hartgerink CHJ, van Assen MALM, Epskamp S, Wicherts JM. 2016. The prevalence of statistical reporting errors in psychology (1985-2013). Behav Res. 48(4):1205–1226.
   Web of Science ®Google Scholar
• Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science. 349(6251):aac4716. aac4716.
   PubMed Web of Science ®Google Scholar
• Patil P, Peng RD, Leek JT. 2016. What should researchers expect when they replicate studies? a statistical view of replicability in psychological science. Perspect Psychol Sci. 11(4):539–544.
   PubMedGoogle Scholar
• Pek J, Flora DB. 2018. Reporting effect sizes in original psychological research: A discussion and tutorial. - PubMed - NCBI. Psychol Methods. 23(2):208–225.
   PubMed Web of Science ®Google Scholar
• Petrakis IL, Ralevski E, Gueorguieva R, O'Malley SS, Arias A, Sevarino KA, Jane JS, O'Brien E, Krystal JH. 2018. Mecamylamine treatment for alcohol dependence: a randomized controlled trial. Addiction. 113(1):6–14.
   PubMed Web of Science ®Google Scholar
• Petrie BF. 1996. Environment is not the most important variable in determining oral morphine consumption in wistar rats. Psychol Rep. 78(2):391–400.
   PubMed Web of Science ®Google Scholar
• Pfister R, Janczyk M. 2013. Confidence intervals for two sample means: Calculation, interpretation, and a few simple rules. ACP. 9(2):74–80.
   Google Scholar
• Piper SK, Grittner U, Rex A, Riedel N, Fischer F, Nadon R, Siegerink B, Dirnagl U. 2019. Exact replication: Foundation of science or game of chance? PLoS Biol. 17(4):e3000188–e3000188.
   PubMed Web of Science ®Google Scholar
• Pisklak JM, Yong JJH, Spetch ML. 2019. The near-miss effect in slot machines: a review and experimental analysis over half a century later. J Gambl Stud. https://www.ncbi.nlm.nih.gov/pubmed/31522339.
   PubMed Web of Science ®Google Scholar
• Pooriamehr A, Sabahi P, Miladi-Gorji H. 2017. Effects of environmental enrichment during abstinence in morphine dependent parents on anxiety, depressive-like behaviors and voluntary morphine consumption in rat offspring. Neurosci Lett. 656:37–42.
   PubMed Web of Science ®Google Scholar
• Quintana DS. 2018. Revisiting non-significant effects of intranasal oxytocin using equivalence testing. Psychoneuroendocrinol. 87:127–130.
   PubMed Web of Science ®Google Scholar
• Rhemtulla M, Fried EI, Aggen SH, Tuerlinckx F, Kendler KS, Borsboom D. 2016. Network analysis of substance abuse and dependence symptoms. Drug Alcohol Depend. 161:230–237.
   PubMed Web of Science ®Google Scholar
• Romero F. 2018. Who should do replication labor? Adv Methods Pract Psychol Sci. 1(4):516–537.
   Google Scholar
• Scheel AM, Schijen M, Lakens D. 2020. An excess of positive results: comparing the standard psychology literature with registered reports.
   Google Scholar
• Simmons JP, Nelson LD, Simonsohn U. 2011. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol Sci. 22(11):1359–1366.
   PubMed Web of Science ®Google Scholar
• Simonsohn U. 2015. Small telescopes: detectability and the evaluation of replication results. Psychol Sci. 26(5):559–569.
   PubMed Web of Science ®Google Scholar
• Spence JR, Stanley DJ. 2016. Prediction interval: what to expect when you’re expecting a replication. PLoS One. 11(9):e0162874.
   PubMed Web of Science ®Google Scholar
• Stanley DJ, Spence JR. 2014. Expectations for replications: are yours realistic? Perspect Psychol Sci. 9(3):305–318.
   PubMedGoogle Scholar
• Stanley TD, Carter EC, Doucouliagos H. 2018. What meta-analyses reveal about the replicability of psychological research. Psychol Bull. 144(12):1325–1346.
   PubMed Web of Science ®Google Scholar
• Sullivan GM, Feinn R. 2012. Using effect size-or why the p value is not enough. J Grad Med Educ. 4(3):279–282.
   PubMedGoogle Scholar
• Verhagen J, Wagenmakers E-J. 2014. Bayesian tests to quantify the result of a replication attempt. J Exp Psychol. General. 143(4):1457–1475.
   PubMed Web of Science ®Google Scholar
• Wohl MJA, Tabri N, Zelenski JM. 2019. The need for open science practices and well-conducted replications in the field of gambling studies. Int Gambl Stud. 19(3):369–376.
   Web of Science ®Google Scholar
• Zou GY. 2007. Toward using confidence intervals to compare correlations. Psychol Methods. 12(4):399–413.
   PubMed Web of Science ®Google Scholar
• Zwaan RA, Etz A, Lucas RE, Donnellan MB. 2018. Making replication mainstream. Behav Brain Sci. 41:e120.
   Web of Science ®Google Scholar