Meta‐Analysis in Criminal Justice and Criminology: What It is, When It's Useful, and What to Watch Out for

Abstract

Meta‐analyses are becoming more common in the criminal justice and criminological literature. While much of the published work related to the technique is focused on the more technical aspects of the statistical methods that have been developed for meta‐analysis, less common are broader—and perhaps more critical—discussions concerning various issues associated with the method. Accordingly, this article presents an overview of meta‐analysis in the context of what have become three of the more important issues within the meta‐analysis literature in recent years: (1) the conditions under which meta‐analyses are, and are not, most useful, (2) the dilemma of whether or not to include unpublished work in the sample of studies to be analyzed, and (3) the choice of bivariate versus multivariate effect size estimates to be synthesized. The objective is to take these issues out of what has arguably been debates about technical orthodoxy and instead to place them into a broader research context within criminal justice and criminology.

Notes

1. The mean treatment effect size estimates for the Whitehead and Lab (1989) and Andrews et al. (1990) studies were similar—both revealed a treatment effect of approximately .20. Their disagreement centered primarily around the substantive interpretation of a treatment effect of that size (i.e., whether a 20% reduction in recidivism is “good enough”).

2. Others in this tradition have used similar bivariate effect size estimates such as the Cohen's d, which is the difference between two group means divided by the pooled within‐group standard deviation (Cohen 1977; see also Loeber and Stouthamer‐Loeber 1986). Still other researchers have used the RIOC statistic (relative improvement over chance), which scales down certain descriptive statistics into a 2 × 2 table of whether or not a predictor variable is present and whether or not an individual engaged in delinquency (Loeber and Dishion 1983). Both of these statistics, however, assume, at minimum, a quasi‐experimental research design, and are therefore not applicable to synthesizing correlational research based on statistical control.

3. Hedges and Olkin's critique of using multivariate effect sizes in meta‐analysis was certainly valid when it was raised over two decades ago. Since then, however, advances in both quantitative methods and computer software (in particular, Hierarchical Linear Modeling methods and software) have resulted in new approaches for handling this potential problem in methodologically defensible ways, including the approach taken here.

4. The Level 1 effect size variances are computed using the formula for estimating the standard error of the Fisher r‐to‐z transformation (see, e.g., Hox 1995; Lipsey and Wilson 2001):

The full model is estimated with the equation: