Rigorous evaluation research among US police departments: special cases or a representative sample?

Abstract

This study considers whether police departments’ characteristics relate to the methodology and results of the evaluations that they participate in, and contrasts police departments that have carried out rigorous crime prevention research to comparison groups of US police agencies. Findings include departments that participate in quasi-experimental crime prevention research are larger and more likely to find statistically significant successes than those that participate in experiments, and evaluators differ in composition and practices from most US police departments. This suggests that researchers and policy makers should involve smaller suburban and rural police agencies to increase the generalizability of evaluation research.

Keywords:

• LEMAS
• police innovation
• evaluation research
• experimental methods

Acknowledgments

We thank the faculty members of the Police Research Group at George Mason University (Christopher Koper, Brian Lawton, Cynthia Lum, Stephen Mastrofski, David Weisburd, and James Willis) for their helpful comments.

Notes

1. We recognize that not all policing evaluation research uses quasi-experimental or experimental designs, but chose to focus on these methods. We understand that some agencies in our ‘nonevaluator’ sample may be participating in research, but using other research designs or focusing on noncrime control research questions.

2. See Lum et al. (2011) and http://cebcp.org/evidence-based-policing/the-matrix/ for further description of their inclusion criteria.

3. See http://cebcp.org/evidence-based-policing/the-matrix/results-key/ for more on Matrix results coding.

4. The NYPD was involved in five studies. Three were identified as SMS = 3, and two were SMS = 5. The results of the studies were not significant (3) significant backfire (1) and mixed success (1). The relationship between SMS score and results generally followed the pattern established in this study, with experiments showing nonsignificant or significant backfire effects, and the SMS = 3 studies finding non-significant or mixed results.

5. One of the reviewers suggested that we further divide the sample of Matrix agencies and comparison groups into small (100–500 sworn officers), medium (501–1000 officers), and large agencies (1001 or more officers). The results are available on request from the authors.

6. When NYPD is included, the differences in population served, number of sworn officers, and staffing between the Matrix police departments and the comparison groups are significant at the .001 level. There were no substantial differences in findings in training and education, use of special units, and the use of innovative practices.