How Powerful is the Evidence in Criminology? On Whether We Should Fear a Coming Crisis of Confidence

Abstract

A crisis of confidence has struck the behavioral and social sciences. A key factor driving the crisis is the low levels of statistical power in many studies. Low power is problematic because it leads to increased rates of false-negative results, inflated false-discovery rates, and over-estimates of effect sizes. To determine whether these issues impact criminology, we computed estimates of statistical power by drawing 322 mean effect sizes and 271 average sample sizes from 81 meta-analyses. The results indicated criminological studies, on average, have a moderate level of power (mean = 0.605), but there is variability. This variability is observed across general studies as well as those designed to test interventions. Studies using macro-level data tend to have lower power than studies using individual-level data. To avoid a crisis of confidence, criminologists must not ignore statistical power and should be skeptical of large effects found in studies with small samples.

Keywords:

• crisis of confidence
• P values
• null hypothesis significance tests
• statistical power

Acknowledgments

The authors thank David P. Farrington, Cory P. Haberman, Jean M. McGloin, David C. Pyrooz, Kyle J. Thomas, Jillian J. Turanovic, Jacob T.N. Young, four anonymous reviewers, and the Editor, Megan C. Kurlychek, for their helpful comments on earlier versions of this paper.

Notes on Contributors

J.C. Barnes is an associate professor in the School of Criminal Justice at the University of Cincinnati. His primary research area focuses on the interplay between genetic and environmental factors in the development of human behavior. He has also recently worked on projects in applied statistical analysis and offender decision-making.

Michael F. TenEyck is an assistant professor with the Department of Criminology and Criminal Justice at the University of Texas at Arlington. His research interests include criminological theory testing, biosocial criminology, drugs and crime, and the psychological correlates of crime. He has published in outlets such as Journal of Quantitative Criminology, Social Networks, PLoS ONE, and Journal of Psychiatric Research.

Travis C. Pratt is a fellow with the University of Cincinnati Corrections Institute. He is the author of Addicted to Incarceration: Corrections Policy and the Politics of Misinformation in the United States. His research focuses on criminological theory and correctional policy, and his work has been published in a variety of journals, including Criminology, Criminology & Public Policy, Journal of Research in Crime and Delinquency, Justice Quarterly, Journal of Quantitative Criminology, and Crime and Justice: A Review of Research. In 2006, he received the Ruth Shonle Cavan Young Scholar Award from the American Society of Criminology.

Francis T. Cullen is Distinguished Research Professor Emeritus and Senior Research Associate in the School of Criminal Justice at the University of Cincinnati. He is a past president of both the Academy of Criminal Justice Sciences and the American Society of Criminology. His recent works include Communities and Crime: An Enduring American Challenge and Criminological Theory: Context and Consequences (7th edition).

Appendix 1. Intervention and nonintervention studies.

Notes. The location of the 25^th percentile value is denoted with the triangle marker on the left, the location of the median value is denoted with the diamond marker, the location of the mean value is denoted with the circle marker, and the location of the 75^th percentile value is denoted with the triangle marker on the right.

Appendix 2. Individual-level & macro-level studies.

Notes. The location of the 25^th percentile value is denoted with the triangle marker on the left, the location of the median value is denoted with the diamond marker, the location of the mean value is denoted with the circle marker, and the location of the 75^th percentile value is denoted with the triangle marker on the right.

Notes

1 We recognize the conceptual distinctions between the terms criminology and criminal justice. For simplicity, we will use criminology to refer to both in this article.

2 These concerns are, practically speaking, impossible to eliminate and one might argue that it is undesirable to eliminate them altogether. Instead, finding small ways to incentivize transparent and accountable research practices is likely the best way forward. As an anonymous reviewer noted, the badge systems used at top psychology journals (e.g., Psychological Science) appear to have had the intended effects.

3 This statement presupposes that the null hypothesis is either true or false. The validity of this perspective has been drawn into question by leading statisticians (see, e.g., Gelman & Loken, 2014a). We will briefly return to this point in the Discussion section, but interested readers are directed to introductory Bayesian texts for more thorough considerations (e.g., Gill, 2014; Lee, 2012).

4 Sherman (2007) raises a fourth factor—heterogeneity in effect size—that should also be considered. Because we seek to offer a broad discussion of statistical power here, we consider heterogeneity of effect sizes to be part of the more general effect size component, but we do recognize its unique contribution to average estimates of statistical power. Interested readers are encouraged to see Sherman’s comments.

5 We thank an anonymous reviewer for pointing out that, in applied research contexts, all else is typically not equal. Funding constraints mean researchers must balance various issues, where maximizing sample size is just one concern of many. It is important to note, then, that we are not implying criminologists have intentionally overlooked statistical power. It is more likely the case that statistical power is but one concern among a landscape of issues that confront scientists in any project. Our goal here is to provide context for why statistical power is and should remain a key concern.

6 But, as an anonymous reviewer noted, it is not always easy for researchers to increase sample sizes given (a) budgetary constraints on primary data collection efforts and (b) the reality that secondary data analysis affords no flexibility on sample size. We agree with these concerns but would also point out that calculating statistical power prior to conducting a study (whether it be primary or secondary data) allows one to gain insight into whether prevailing sample sizes are appropriate for testing the question at hand given an expected effect size. Doing so also affords the researcher the opportunity to estimate the probability of other types of inferential errors prior to carrying out an analysis (Gelman & Carlin, 2014). This, however, should not give the misleading impression that statistical significance is the goal of any study and that one simply needs a “big enough” sample size to achieve P < 0.05. We will return to these points in the Discussion section, particularly when we recommend that criminologists consider adopting alternative approaches.

7 The journals were Journal of Criminal Law & Criminology, Criminology, Journal of Research in Crime and Delinquency, Crime & Delinquency, Journal of Criminal Justice, Journal of Police Science & Administration, Criminal Justice Review, and Criminal Justice and Behavior.

8 But, as an anonymous reviewer pointed out, published meta-analyses do not perfectly represent the complexity of research in criminology. For example, focusing on meta-analyses means we necessarily are restricted to quantitative studies. It is not clear whether findings from our study have anything to say about qualitative research. This point, to our knowledge, has not been raised in other fields that are currently experiencing the crisis of confidence. We thus encourage criminologists to engage in this discussion given the rich pieces of information that have been gleaned from qualitative research in our field.

9 Ideally, we would have collected additional information such as the sample size per cell, manipulations that were carried out, and statistical tests that were estimated. But most of the meta-analyses did not include this information. We encourage scholars to follow-up on our efforts with more nuanced foci like those mentioned here. To do so will most likely necessitate a review of primary studies instead of meta-analyses.

10 An anonymous reviewer noted that meta-analysis “provides an effective solution to the problems of sampling error and low power and precision in individual studies” (Schmidt & Oh, 2016, p.34). This is an important point. Because we rely on the average effect size and the average sample size reported in each meta-analysis, we are able to provide an estimate of the average level of statistical power for each meta-analysis included in our sample. But, to the degree that reliance on meta-analysis has affected our estimates of statistical power, given the comments by Schmidt and Oh, one might be inclined to conclude that our estimates are inflated (i.e., that statistical power is actually lower than we report). Also, by extension, our study does not consider research that has not been meta-analyzed. Based on Schmidt and Oh’s (2016) arguments, we have reason to believe statistical power will be higher in our study than if one were to assess a sample of findings in criminology that have not been meta-analyzed.

11 Specifically, the advanced search codes for the Web of Science search were: “TI = (meta*) AND TI = (crim* OR delinq* OR antisocial OR anti-social OR recidivism) AND WC = (Criminology & Penology) Timespan: 1990–2015.” The codes for the Google Scholar search were: “allintitle: meta-analysis crime, OR delinquency, OR antisocial, OR anti-social, OR recidivism” and the date range was restricted to studies published in 1990–2015.

12 An example of a study that did not meet the inclusion criteria and was, therefore, omitted from the analysis is the meta-analysis on the relationship between self-control and victimization by Pratt et al. (2014). This study met the first two criteria, but it did not meet the third criterion.

13 For example, the meta-analysis by Welsh and Farrington (2009) analyzed the effect of CCTV on local crime rates. Moderator analyses explored whether the effect of CCTV was specific to a particular crime type (violent versus vehicle crime). For the purposes of this study, we recorded information on the overall crime analysis. Another example is the meta-analytic results presented by Gobeil et al. (2016). This study presented the results of interventions on recidivism for female offenders. The authors provide an “overall” effect, which was included in our analysis. Gobeil and colleagues also performed several additional analyses to address questions about whether the type of intervention affected the outcome. These latter estimates were not coded for the present analysis. Again, our point here was to gather broad estimates of effect sizes and average sample sizes across the discipline—not to explore variability of effects due to moderator analyses.

14 Risk ratios were treated as odds ratios during conversion, which is unlikely to have biased the results due to the fact that our analysis generally focuses on rare outcomes. Risk ratios are approximately equal to odds ratios when the outcome is rare, but they will diverge such that the odds ratio will grow larger than the risk ratio as the prevalence of the outcome increases. Only 21 of the total 322 effects sizes were reported as risk ratios and our effect size estimates do not substantively change when we omit the risk ratios from the analysis.

15 We thank an anonymous reviewer for raising this point.

16 We thank an anonymous reviewer for raising these points.