Meta-Analysis in Criminology and Criminal Justice: Challenging the Paradigm and Charting a New Path Forward

Abstract

Meta-analyses are appearing more frequently in the criminological literature. Yet the methods typically used are guided by a methodological paradigm that risks producing meta-analyses of limited value. Here we outline three key methodological issues that meta-analysts face and we present a methodological challenge to the dominant meta-analysis paradigm. We focus specifically on: (1) inclusion criteria, (2) analysis of bivariate versus multivariate effect sizes, and (3) methods for handling statistical dependence. Issues of reproducibility and recommendations for moving forward are discussed.

Keywords:

• Inclusion criteria
• meta-analysis
• partial effect sizes
• reproducibility
• statistical dependence

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 To clarify, these meta-analyses are also those that included just one effect size per study/per sample, meaning that most ran their analyses on a sample of 10 or fewer effect sizes.

2 Weighting effect sizes in a meta-analysis—for example, by study sample size or inverse variance—is intended to mitigate the problem of underpowered research by distributing a greater portion of the degrees of freedom to those effects produced using larger samples (Marin-Martinez & Sanchez-Meca, 2010). Yet doing so does not alter the total number of degrees of freedom available in the meta-analysis. So while weighting may reduce the potential influence of underpowered effect sizes in a meta-analytic sample, such weights cannot overcome or otherwise “cure” the problem of a meta-analysis itself being underpowered (see, e.g., Ioannidis, 2005).

3 There are various formulas available to convert different types of effect sizes from bivariate and multivariate statistical models into a common metric, such as r, to be compared in a meta-analysis (see, e.g., Aloe & Thompson, 2013; Lipsey & Wilson, 2001; Peterson & Brown, 2005; Pratt et al., 2014).

4 We recognize that there can also be instances where the improper inclusion of covariates will result in the overestimation of multivariate effect sizes that are sometimes larger than bivariate associations (Beckstead, 2012).

5 When using the effect size r: for bivariate effect sizes, standard errors are dependent on sample size and can be calculated using $\sigma =\sqrt{1/(n-3)}$ (Lipsey & Wilson, 2001). For multivariate effect sizes, σ can be calculated using$\sigma = r/(b/SE),$ where r is the effect size estimate, b is an unstandardized regression coefficient, and SE the standard error for the unstandardized coefficient (see Pratt et al., 2014; Pyrooz et al., 2016).

6 To be clear, for a meta-analytic dataset that contains two substantive levels, such as effect sizes nested within studies—the “within studies” model can be expressed as: ${\mathit{ESE}}_{ij}= {\zeta }_{ij}+e_{ij}$ , where ${\mathit{ESE}}_{ij}$ is the observed effect size i in study j, ${\zeta }_{ij}$ is an estimate of the “true parameter” value of the effect size i in study j, and $e_{ij}$ is the sampling error of effect size i in study j. The $e_{ij}$ term has a known variance of${\sigma }_{ij}^2,$ and thus ${\mathit{ESE}}_{ij}$ varies between studies as a result of the joint effect of sampling error and “true parameter” variance (see Hox, 1995, p. 70). The “between-studies” model for the parameter ${\zeta }_{ij}$ can be expressed as: ${\zeta }_{ij}= {\gamma }_{00}+u_{0j},$ where $u_{0j}$ is the study-level random error. In combining the equations, we obtain the unconditional model: ${\mathit{ESE}}_{ij}= {\gamma }_{00}+u_{0j}+e_{ij},$ where the intercept ${\gamma }_{00}$ is the mean effect size estimate. This model—although it contains two substantive levels (of effect sizes nested within studies)—can be described as “three levels” where the known variance for each effect size estimate is at level 1, the within-study variation is at level 2, and the between-study variation is at level 3.

Additional information

Notes on contributors

Jillian J. Turanovic

Jillian J. Turanovic is an associate professor in the College of Criminology and Criminal Justice at Florida State University. Her research examines various issues in criminological theory and correctional policy, with a special focus on victimization, violence, and the life course.

Travis C. Pratt

Travis C. Pratt is a fellow at the University of Cincinnati Corrections Institute and the research director at the Harris County Community Supervision and Corrections Department. His work is focused on criminological theory and corrections policy.