The Evidence of Effectiveness: Beyond the Methodological Standards

Abstract

There is growing recognition that policymakers can achieve substantially better results by using an evidence-based approach to solve social problems. Nevertheless, there is still considerable debate as to how best to identify evidence-based information, aggregate and process this information, and then disseminate it to non-technical users. This manuscript discusses the sources of this discord within the context of evidence-based registries and propose ways in which to ameliorate it.

Keywords:

• Methodology
• evaluation
• evidence-based
• policy-making
• research to practice

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 SAMHSA Assistant Secretary Elinore F. McCance-Katz suspended the operation of NREPP in 2018.

2 One exception to this general observation is in CrimeSolutions.gov, where studies of comparative effectiveness are excluded from the evidence review.

3 https://www.crimesolutions.gov/ProgramDetails.aspx?ID=585

4 https://www.crimesolutions.gov/ProgramDetails.aspx?ID=586

5 It should be pointed out that this is not exactly a novel concept as some repositories do take this approach. For example, both NREPP (prior to the shutdown) and the What Works Clearinghouse have rated programs by outcomes. In addition, CrimeSolutions.gov rates practices by outcome.

6 https://www.crimesolutions.gov/ProgramDetails.aspx?ID=143

7 https://www.crimesolutions.gov/ProgramDetails.aspx?ID=232

8 https://www.crimesolutions.gov/ProgramDetails.aspx?ID=188

9 Cohen’s U3 looks at describing distribution overlap in that it “describes the percentage of scores in the lower-meaned group that are exceeded by the average score in the higher-meaned group” (Valentine & Cooper, 2006, p. 5). This effect size essentially describes the percentile gains (or losses) that the average unit of the treatment group would experience should it move to the control group.

10 Alley-Gating in Liverpool (England) crimesolutions.gov/ProgramDetails.aspx?ID=135 and Burglary Reduction (Hartlepool, England) crimesolutions.gov/ProgramDetails.aspx?ID = 57

11 Preventing Parolee Crime Program (PPCP) https://www.crimesolutions.gov/ProgramDetails.aspx?ID=72 and Drug Treatment Alternative to Prison (DTAP) https://www.crimesolutions.gov/ProgramDetails.aspx?ID=89

Additional information

Funding

This project was supported by funding from the National Institute of Justice, Office of Justice Programs, U.S. Department of Justice. Points of view in this document are those of the authors and do not necessarily represent the official position or policies of the U.S. Department of Justice.

Notes on contributors

Stephen V. Gies

Stephen Gies, PhD is the Director of Research and Evaluation at Development Services Group, Inc., specializing in research and evaluation, advanced statistical analysis, policy analysis, program development, database management, and translating research to practice. He received his PhD from American University School of Public Affairs. He is the principal designer of OJP's CrimeSolutions.gov – an online database of evidence-based programs and practices encompassing a broad range of criminal justice, juvenile justice, and victims' topics), and formerly the Quality of Research Review Manager and Principal Designer for SAMHSA's restyled National Registry of Evidence-based Programs and Practices (NREPP) – a program and policy development tool for mental health and substance abuse. His research interests focus on the commercial sexual exploitation of children and human trafficking, emerging technology trends and their impact on crime, identity theft and financial abuse, the intersection of sports and crime, and evidence-based policymaking. His research has been published in several book chapters, and peer-reviewed professional and policy journals, including Criminal Justice Studies, Journal of Ethnographic and Qualitative Research, Corrections Today.

Eoin Healy

Eoin Healy, PhD is a Senior Research Scientist at Development Services Group, Inc., specializing in quantitative statistical analysis and methodology, program research and evaluation, risk assessment, evidence review, and best-practice dissemination. Dr. Healy is a Senior Researcher for the CrimeSolutions.gov database for the National Institute of Justice (NIJ), and he also serves as an internal study reviewer for CrimeSolutions.gov. His areas of research include terrorism, reentry programs, reducing unnecessary pretrial detention, risk assessment, technology evaluations, GPS monitoring programs, human trafficking, and post-secondary education programs. His research has been published in book chapters, and peer-reviewed journals, including Criminal Justice Studies.

Rachel Stephenson

Rachel Stephenson is a Research Analyst at Development Services Group, Inc. Ms. Stephenson specializes in evaluating criminal and juvenile justice programs, assessing the quality of program evaluations, managing numerous databases, advising on policy development, and translating complicated research for policymaker and practitioner audiences. She is the Project Director for the National Institute of Justice's CrimeSolutions.gov and the Office of Juvenile Justice and Delinquency Prevention's Model Programs Guide. She has also worked on a number of federal- and state-sponsored juvenile justice research projects, including multiple evaluations of efforts to reduce disproportionate minority contact in the juvenile justice system. She received her B.A. in political science from The George Washington University in Washington, D.C.; and her M. A. in criminology and criminal justice from the University of Maryland, College Park.