How Powerful is the Evidence in Criminology? On Whether We Should Fear a Coming Crisis of Confidence

References

• American Academy of Arts & Sciences (2018). Perceptions of science in America. Cambridge, MA: Author.
   Google Scholar
• Austin, J. (2003). Why criminology is irrelevant. Criminology Public Policy, 2, 557–564.
   Google Scholar
• Bailystok, E., Kroll, J. F., Green, D. W., MacWhinney, B., & Craik, F. I. M. (2015). Publication bias and the validity of evidence: What’s the connection? Psychological Science, 26, 944–946.
   Web of Science ®Google Scholar
• Baker, M. (2016). 1,500 scientists lift the lid on reproducibility: Survey sheds light on the ‘crisis’ rocking research. Nature, 533, 452–454.
   PubMed Web of Science ®Google Scholar
• Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7, 543–554.
   Web of Science ®Google Scholar
• Bartels, L. (2017). Swift, certain and fair: Does Project HOPE provide a therapeutic paradigm for managing offenders? Basingstoke: Palgrave MacMillan.
   Google Scholar
• Britt, C. L., & Weisburd, D. (2010). Statistical power. In Piquero, A. R. and Weisburd, D. (eds.), Handbook of quantitative criminology (pp. 313–332). New York, NY: Springer.
   Google Scholar
• Brown, S. E. (1989). Statistical power and criminal justice research. Journal of Criminal Justice, 17, 115–122.
   Web of Science ®Google Scholar
• Bushway, S. D., Sweeten, G., & Wilson, D. B. (2006). Size matters: Standard errors in the application of null hypothesis significance testing in criminology and criminal justice. Journal of Experimental Criminology, 2, 1–22.
   Google Scholar
• Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376.
   PubMed Web of Science ®Google Scholar
• Carter, E. C., & McCullough, M. E. (2014). Publication bias and the limited strength model of self-control: Has the evidence for ego depletion been overestimated? Frontiers in Psychology, 5, 1–11.
   PubMed Web of Science ®Google Scholar
• Clear, T. (2010). Policy and evidence: The challenge to the American Society of Criminology: 2009 Presidential Address to the American Society of Criminology. Criminology, 48, 1–25.
   Web of Science ®Google Scholar
• Clemens, M. A. (2017). The meaning of failed replications: A review and proposal. Journal of Economic Surveys, 31, 326–342.
   Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York, NY: Lawrence Erlbaum Associates.
   Google Scholar
• Colquhoun, D. (2014). An investigation of the false discovery rate and the misinterpretation of p-values. Royal Society Open Science, 1, 1–16.
   Web of Science ®Google Scholar
• Cressey, D. R. (1978). Criminological theory, social science, and the repression of crime. Criminology, 16, 171–191.
   Web of Science ®Google Scholar
• Cullen, F. T., Pratt, T. C., & Turanovic, J. J. (2016). It’s hopeless: Beyond zero-tolerance supervision. Criminology and Public Policy, 15, 1215–1227.
   Web of Science ®Google Scholar
• Duncan, L. A., & Keller, M. C. (2011). A critical review of the first 10 years of candidate gene-by-environment interaction research in psychiatry. American Journal of Psychiatry, 168, 1041–1049.
   PubMed Web of Science ®Google Scholar
• Duvendack, M., Palmer-Jones, R., & Reed, W. R. (2017). What is meant by "replication" and why does it encounter resistance in economics? American Economic Review, 107(5), 46–51.
   Web of Science ®Google Scholar
• Earp, B. D., & Trafimow, D. (2015). Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology, 6, 1–11.
   PubMed Web of Science ®Google Scholar
• Farrington, D. P., Gaffney, H., & Ttofi, M. M. (2016). Systematic reviews of explanatory risk factors for violence, offending, and delinquency. Aggression and Violent Behavior, 33, 24–36.
   Web of Science ®Google Scholar
• Ferguson, C. J. (2013). Spanking, corporal punishment and negative long-term outcomes: A meta-analytic review of longitudinal studies. Clinical Psychology Review, 33, 196–208.
   PubMed Web of Science ®Google Scholar
• Ferguson, C. J., & Heene, M. (2012). A vast graveyard of undead theories: Publication bias and psychological science's aversion to the null. Perspectives on Psychological Science : A Journal of the Association for Psychological Science, 7, 555–561.
   PubMedGoogle Scholar
• Fisher, R. A. (1925). /1973). Statistical Methods for Research Workers. New York, NY: Hafner.
   Google Scholar
• Franco, A., Malhotra, N., & Simonovits, G. (2014). Social science. Publication bias in the social sciences: unlocking the file drawer. Science (New York, N.Y.), 345, 1502–1505.
   PubMed Web of Science ®Google Scholar
• Gelman, A., & Carlin, J. (2014). Beyond power calculations: Assessing type s (sign) and type m (magnitude) errors. Perspectives on Psychological Science, 9, 641–651.
   Web of Science ®Google Scholar
• Gelman, A., & Loken, E. (2014a). The AAA tranche of subprime science. Chance, 27, 51–56.
   Google Scholar
• Gelman, A., & Loken, E. (2014b). Data-dependent analysis—a “garden of forking paths”—explains why many statistically significant comparisons don’t hold up. American Scientist, 102, 460–465.
   Web of Science ®Google Scholar
• Gelman, A., Skardhamar, T., & Aaltonen, M. (2018). Type M error can explain Weisburd’s paradox. Journal of Quantitative Criminology.
   Google Scholar
• Gilbert, D. T., King, G., Pettigrew, S., & Wilson, T. D. (2016). Comment on “Estimating the reproducibility of psychological science”. Science, 351, 1037.
   PubMed Web of Science ®Google Scholar
• Gill, J. (2014). Bayesian methods: A social and behavioral sciences approach. New York, NY: Chapman & Hall/CRC.
   Google Scholar
• Gobeil, R., Blanchette, K., & Stewart, L. (2016). A meta-analytic review of correctional interventions for women offenders: Gender-neutral versus gender-informed approaches. Criminal Justice and Behavior, 43, 301–322.
   Google Scholar
• Gorman, D. M. (2015). “Everything works”: The need to address confirmation bias in evaluations of drug misuse prevention interventions for adolescents. Addiction, 110, 1539–1540.
   PubMed Web of Science ®Google Scholar
• Hagan, J. (1973). Labeling and deviance: A case study in the “sociology of the interesting”. Social Problems, 20, 447–458.
   Web of Science ®Google Scholar
• Hamilton, Z., Campbell, C. M., van Wormer, J., Kirgl, A., & Posey, B. (2016). Impact of swift and certain sanctions: Evaluation of Washington State’s policy for offenders on community supervision. Criminology and Public Policy, 15, 1009–1072.
   Web of Science ®Google Scholar
• Hawken, A., & Kleiman, M. A. R. (2009). Managing drug involved probationers with swift and certain sanctions: Evaluating Hawaii’s HOPE. Washington, DC: National Institute of Justice.
   Google Scholar
• Higdon, J. V., & Frei, B. (2006). Coffee and health: A review of recent human research. Critical Reviews in Food Science and Nutrition, 46, 101–123.
   PubMed Web of Science ®Google Scholar
• Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124
   PubMed Web of Science ®Google Scholar
• Ioannidis, J. P. A., Stanley, T. D., & Doucouliagos, H. (2015). The power of bias in economics research. Working Paper, https://www.deakin.edu.au/__data/assets/pdf_file/0007/477763/2016_1.pdf
   Google Scholar
• Johnson, V. E., Payne, R. D., Wang, T., Asher, A., & Mandal, S. (2017). On the reproducibility of psychological science. Journal of the American Statistical Association, 112(517), 1–10.
   PubMed Web of Science ®Google Scholar
• Jussim, L., Crawford, J. T., Anglin, S. M., Stevens, S. T., & Duarte, J. L. (2016). Interpretations and methods: Towards a more effectively self-correcting social psychology. Journal of Experimental Social Psychology, 66, 116–133.
   Web of Science ®Google Scholar
• Kahneman, D. (2011). Thinking, Fast and Slow. New York, NY: Farrar, Straus and Giroux.
   Google Scholar
• King, G. (1995). Replication, replication. PS: Political Science & Politics, 28, 444–452.
   Web of Science ®Google Scholar
• Kulig, T. C., Pratt, T. C., & Cullen, F. T. (2017). Revisiting the Stanford prison experiment: A case study in organized skepticism. Journal of Criminal Justice Education, 28, 74–111.
   Web of Science ®Google Scholar
• Lajous, M., Bijon, A., Fagherazzi, G., Balkau, B., Boutron-Ruault, M. C., & Chapelon, F. (2015). Egg and cholesterol intake and incident type 2 diabetes among French women. British Journal of Nutrition, 114, 1667–1673.
   PubMed Web of Science ®Google Scholar
• Lansford, J. E., Sharma, C., Malone, P. S., Woodlief, D., Dodge, K. A., Oburu, P., … Di Giunta, L. (2014). Corporal punishment, maternal warmth, and child adjustment: A longitudinal study in eight countries. Journal of Clinical Child and Adolescent Psychology, 43, 670–685.
   PubMed Web of Science ®Google Scholar
• Lattimore, P. K., MacKenzie, D. L., Zajac, G., Dawes, D., Arsenault, E., & Tueller, S. (2016). Outcome findings from the HOPE demonstration field experiment: Is swift, certain, and fair an effective supervision strategy? Criminology and Public Policy, 15, 1103–1141.
   Web of Science ®Google Scholar
• Lee, P. M. (2012). Bayesian statistics: An introduction (4th ed.). West Sussex, UK: Wiley.
   Google Scholar
• Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage.
   Google Scholar
• Maltz, M. D. (1994). Deviating from the mean: The declining significance of significance. Journal of Research in Crime and Delinquency, 31, 434–463.
   Web of Science ®Google Scholar
• Maruna, S. (2015). Qualitative research, theory development, and evidence-based corrections: can success stories be “evidence.” In: Jody Miller and Wilson R. Palacios (Eds.), Qualitative research in criminology—Advances in criminological theory (Vol. 20, pp. 311–337). New Brunswick, NJ: Transaction.
   Google Scholar
• Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate really mean?” The American Psychologist, 70, 487–498.
   PubMed Web of Science ®Google Scholar
• McNeeley, S., & Warner, J. J. (2015). Replication in criminology: A necessary practice. European Journal of Criminology, 12, 581–597.
   Web of Science ®Google Scholar
• McShane, B. B., & Gal, D. (2016). Blinding us to the obvious? The effect of statistical training on the evaluation of evidence. Management Science, 62, 1707–1718.
   Web of Science ®Google Scholar
• Merton, R. K. (1957). Priorities in scientific discovery: A chapter in the sociology of science. American Sociological Review, 22, 635–659.
   Web of Science ®Google Scholar
• Merton, R. K. (1973). The sociology of science: Theoretical and empirical investigations. Chicago, IL: University of Chicago Press.
   Google Scholar
• Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E. J. (2016). The fallacy of placing confidence in confidence intervals. Psychonomic Bulletin & Review, 23, 103–123.
   PubMed Web of Science ®Google Scholar
• Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Du Sert, N. P., … Ioannidis, J. P. A. (2017). A manifesto on reproducible science. Nature Human Behaviour, 1, 0021.
   PubMed Web of Science ®Google Scholar
• Nakamura, Y., Iso, H., Kita, Y., Ueshima, H., & Okada, K. (2006). Egg consumption, serum total cholesterol concentrations and coronary heart disease incidence: Japan Public Health Center-based prospective study. British Journal of Nutrition, 96, 921–928.
   PubMed Web of Science ®Google Scholar
• National Academies of Sciences, Engineering, and Medicine. (2016). Statistical challenges in assessing and fostering the reproducibility of scientific results: Summary of a workshop.
   Google Scholar
• Nelson, M. S., Wooditch, A., & Dario, L. M. (2015). Sample size, effect size, and statistical power: A replication study of Weisburd’s paradox. Journal of Experimental Criminology, 11, 141–163.
   Web of Science ®Google Scholar
• Nickerson, R. S. (1998). Confirmation bias: A ubiquitous phenomenon in many guises. General Review of Psychology, 2, 175–220.
   Google Scholar
• Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348, 1422–1425.
   PubMed Web of Science ®Google Scholar
• Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631.
   Web of Science ®Google Scholar
• Nuijten, M. B., van Assen, M. A. L. M., Augusteijn, H. E. M., Crompvoets, E. A. V., & Wicherts, J. M. (2018). Effect sizes, power, and biases in intelligence research: A meta-meta-analysis. Preprint retrieved from https://psyarxiv.com/ytsvw.
   Google Scholar
• O’Connell, D. J., Brent, J. J., & Visher, C. A. (2016). Decide your time: A randomized trial of a drug testing and graduated sanctions program for probationers. Criminology and Public Policy, 15, 1073–1102.
   Web of Science ®Google Scholar
• Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.
   PubMed Web of Science ®Google Scholar
• Pashler, H., & Wagenmakers, E. J. (2012). Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science : A Journal of the Association for Psychological Science, 7, 528–530.
   PubMedGoogle Scholar
• Petrosino, A., Boruch, R. F., Soydan, H., Duggan, L., & Sanchez-Meca, J. (2001). Meeting the challenges of evidence-based policy: The Campbell collaboration. Annals of the American Academy of Political and Social Science, 578, 14–34.
   Web of Science ®Google Scholar
• Pickett, J. T., & Roche, S. P. (2018). Questionable, objectionable or criminal? Public opinion on data fraud and selective reporting in science. Science and Engineering Ethics, 24, 151–171.
   PubMed Web of Science ®Google Scholar
• Pratt, T. C., Turanovic, J. J., & Cullen, F. T. (2016). Revisiting the criminological consequences of exposure to fetal testosterone: A meta-analysis of the 2D:4D digit ratio. Criminology, 54, 587–620.
   Web of Science ®Google Scholar
• Pratt, T. C., Turanovic, J. J., Fox, K. A., & Wright, K. A. (2014). Self-control and victimization: A meta-analysis. Criminology, 52, 87–116.
   Web of Science ®Google Scholar
• Pridemore, W. A., Makel, M. C., & Plucker, J. A. (2018). Replication in criminology and the social sciences. Annual Review of Criminology, 1, 19–38.
   Web of Science ®Google Scholar
• Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59–82.
   PubMed Web of Science ®Google Scholar
• Rosnow, R. L., & Rosenthal, R. (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284.
   Web of Science ®Google Scholar
• Sampson, R. J. (2010). Gold standard myths: Observations in the experimental turn in quantitative criminology. Journal of Quantitative Criminology, 26, 489–500.
   Web of Science ®Google Scholar
• Schmidt, F. L., & Oh, I. S. (2016). The crisis of confidence in research findings in psychology: Is lack of replication the real problem? Or is it something else? Archives of Scientific Psychology, 4, 32–37.
   Google Scholar
• Sham, P. C., & Purcell, S. M. (2014). Statistical power and significance testing in large-scale genetic studies. Nature Reviews Genetics, 15, 335–346.
   PubMed Web of Science ®Google Scholar
• Sherman, L. W. (2007). The power few: Experimental criminology and the reduction of harm: The 2006 Joan McCord Prize Lecture. Journal of Experimental Criminology, 3, 299–321.
   Google Scholar
• Sherman, L. W., Farrington, D. P., Welsh, B. C., & MacKenzie, D. L. (2002). Evidence-based crime prevention. New York, NY: Routledge.
   Google Scholar
• Sherman, L. W., Gottfredson, D., MacKenzie, D., Eck, J., Reuter, P., & Bushway, S. (1997). Preventing crime: What works, what doesn’t, what’s promising: A report to the United States congress. Washington, DC: National Institute of Justice.
   Google Scholar
• Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366.
   PubMed Web of Science ®Google Scholar
• Stroebe, W., & Strack, F. (2014). The alleged crisis and the illusion of exact replication. Perspectives on Psychological Science, 9, 59–71.
   Web of Science ®Google Scholar
• Vostal, F. (2016). Accelerating academia. The changing structure of academic time. Basingstoke: Palgrave Macmillan.
   Google Scholar
• Wasserstein, R. L., & Lazar, N. A. (2016). The ASA’s statement on p-values: Contest, process, and purpose. The American Statistician, 70, 129–133.
   Web of Science ®Google Scholar
• Weisburd, D., Lum, C. M., & Yang, S. M. (2003). When can we conclude that treatments or programs “don’t work”? The Annals of the American Academy of Political and Social Science, 587, 31–48.
   Web of Science ®Google Scholar
• Weisburd, D., Petrosino, A., & Mason, G. (1993). Design sensitivity in criminal justice experiments. Crime and Justice, 17, 337–379.
   Google Scholar
• Weisburd, D., & Piquero, A. R. (2008). How well do criminologists explain crime? Statistical modeling in published studies. Crime and Justice, 37, 453–502.
   Web of Science ®Google Scholar
• Welsh, B. C., & Farrington, D. P. (2009). Public area CCTV and crime prevention: An updated systematic review and meta-analysis. Justice Quarterly, 26, 716–745.
   Google Scholar
• Woodward, M., & Tunstall-Pedoe, H. (1999). Coffee and tea consumption in the Scottish Heart Health Study follow up: Conflicting relations with coronary risk factors, coronary disease, and all cause mortality. Journal of Epidemiology and Community Health, 53, 481–487.
   PubMed Web of Science ®Google Scholar
• Woodward, V. H., Webb, M. E., Griffin, III, O. H., & Copes, H. (2016). The current state of criminological research in the United States: An examination of research methodologies in criminology and criminal justice journals. Journal of Criminal Justice Education, 27, 340–361.
   Web of Science ®Google Scholar
• Worrall, J. L. (2000). In defense of the “quantoids”: More on the reasons for the quantitative emphasis in criminal justice education and research. Journal of Criminal Justice Education, 11, 353–361.
   Google Scholar