Good Questions: Building Evaluation Evidence in a Competitive Policy Environment

Abstract

Criminal justice evaluations are funded and executed in a policy environment increasingly characterized by intense competition that creates a premium on studies utilizing experimental methods. Randomization, however, is not a universally suitable method for answering all questions of criminal justice policy and practice. Experimental designs are particularly ill-suited for addressing key analytical challenges and exploiting important opportunities in justice, including discontinuity effects, interventions that depend on the perceptions and beliefs of individuals, and models of general policy change. Experiments are also rarely able to reliably measure and isolate the effects of very complex justice interventions. Policymakers and practitioners in the justice sector should consider evaluation research as a portfolio of strategic investments in knowledge development. Randomized controlled trials are merely one asset in a broader investment strategy.

Keywords:

• Competition; design; evaluation; policy; randomization; research

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1 The NLSY asks, “Suppose you were arrested for stealing a car, what is the percent chance that you would serve time in jail?”

2 Incarceration could be deemed by some as immoral or unjust. The only concern here, however, is whether more or less incarceration is associated with the amount of crime. In that scenario, it could be logical to expect that larger numbers of incarcerated people will reduce the number of crimes.

3 An experiment is not impossible in this setting, but in practice such research is rarely attempted.

Additional information

Notes on contributors

Jeffrey A. Butts

Jeffrey A. Butts is the director of the Research and Evaluation Center at John Jay College of Criminal Justice, City University of New York.

John K. Roman

John K. Roman is a Senior Fellow at NORC at the University of Chicago.