Synthetic Control Analysis of the Short-Term Impact of New York State’s Bail Elimination Act on Aggregate Crime

Abstract

We conduct an empirical evaluation of the short-term impact of New York’s bail reform on crime. New York State’s Bail Elimination Act went into effect on January 1, 2020, eliminating money bail and pretrial detention for nearly all misdemeanor and nonviolent felony defendants. Our analysis of effects on aggregate crime rates after the reform informs the understanding of bail reform and general deterrence, rather than specific deterrence via re-arrest rates of the detained/released population. We conduct a synthetic control analysis for a comparative case study of the impact of bail reform. We focus on synthetic control analysis of post-intervention changes in crime for assault, theft, burglary, robbery, and drug crimes, constructing a dataset from publicly reported crime data of 27 large municipalities. Due to the short time frame before the onset of COVID-19 and its far-reaching effects, we restrict attention to a short post-intervention time period. Nonetheless, evaluation of short-term impacts may still inform hypotheses of general deterrence of bail reform policy. Our findings, including placebo checks and other robustness checks, show that for assault, theft, and drug crimes, there is no significant impact of bail reform on aggregate crime. For robbery, we find a statistically significant increase; for burglary, the synthetic control is more variable and our analysis is deemed less conclusive. Since our study assesses the short-term impacts, further work studying long-term impacts of bail reform and on specific deterrence remains necessary. Supplementary materials for this article are available online.

KEYWORDS:

• Bail reform
• Criminal justice
• Program evaluation
• Synthetic control

1 Introduction

The use of cash bail has come under increasing scrutiny in the United States, with several states and jurisdictions adopting or considering reforms to limit its use (Covert 2017). One key question is whether such reforms—which generally lead to fewer pretrial detainees—impact crime rates. We address this question by investigating a real-life case study, New York’s implementation of a sweeping bail reform legislation which took effect on January 1, 2020. We focus on the impact on the incidence of crime of different types in New York City (NYC), using the synthetic control method (Abadie, Diamond, and Hainmueller 2010, 2011) to conduct comparative event-study analysis.

Specifically, our analysis takes advantage of the fact that cities across the United States historically experience similar shifts in crime trends (Barker 2010), which allows us to compare the developments in NYC to a pool of control jurisdictions where no such policy changes were made within the timeframe of our study between January 1, 2018 and March 15, 2020. Our focus is specifically on the effect of New York State’s Bail Elimination Act on New York City crime rates. “Bail reform” is a broad term that is applied to a wide range of policies, from nudges to judicial actors to reduce the use of money bail, to a wide-ranging bill such as the Bail Elimination Act which eliminate the use of money bail for a wide variety of charges without judicial discretion. Our conceptualization of treatment is the specific elimination of money bail in New York State as mandated by the Bail Elimination Act.

This cross-unit comparison aids causal inference by disentangling the true impact of bail reform from other concurrent events and trends, which would not be possible by conducting a within-unit comparison such as interrupted-time series (Degli Esposti et al. 2020), which we also conduct as a reference starting point. To facilitate our analysis, we collected data directly from 27 different cities’ police departments on the incidence rates of different types of crimes. Using the synthetic control method, for each crime type, we construct a weighted combination of these cities to match NYC before 2020. Roughly, comparing the divergence—or lack thereof—in 2020 allows us to assess the impact of the novel treatment in 2020. Our analysis is limited to short-term impacts due to the onset of COVID-19 shortly after the institution of bail reform. In Sections 6 and 7 we discuss limitations in further detail. Overall, following statistical inference and robustness checks, we find little to no evidence of any significant impact of bail reform on total crime rates. The synthetic control explains away observed trends in post-intervention crime rates, assessing the average treatment effect estimate against the variability in the distribution of effect estimates under placebo analyses with known null interventional effect. These results are robust under other robustness checks.

The structure of our article is as follows: we first discuss the background, data, methods, and results, before summarizing limitations of the analysis and offering concluding remarks. We review the context of bail reform and of findings elsewhere as well as the specific changes under the reform passed in New York, and we highlight domain-level considerations that inform methodological choices for our empirical evaluation and our focus on a synthetic control evaluation. We then turn to describing data, methods, and our results: we describe our constructed dataset and pre-processing as well as our main analyses. We focus on synthetic control analysis of post-intervention changes in crime for assault, theft, burglary, robbery, and drug crimes. Our findings, including placebo checks and other robustness checks, show that for assault, theft, and drug crimes, there is no significant impact of bail reform on crime; for burglary and robbery, we similarly have null findings but the synthetic control is also more variable so these are deemed less conclusive. Finally, we discuss important limitations of our study in the operational context of New York before concluding by presenting directions for future research.

2 Background

2.1 Bail Reform

As more U.S. jurisdictions consider reforms to end or sharply reduce the use of cash bail, debates about the impacts of such reforms have gained urgency. Key questions include the effects of such reforms on: (i) pretrial detention rates; (ii) pretrial rearrest rates; and (iii) overall crime rates. The first two questions have received the most attention in the literature. This study makes a unique contribution by focusing on the third question. Specifically, in assessing the effects on aggregate crime rates, we explore the broader consequences of bail reform, including potential consequences for general deterrence.

To assess bail reform’s impacts on crime and rearrest, we investigate the effects of New York State’s Bail Elimination Act, which went into effect on January 1, 2020. Other reports with more descriptive detail and statistics include those of Rempel and Rodriguez (2020) and Rempel and Weill (2021). The reform “eliminates money bail and pretrial detention for nearly all misdemeanor and nonviolent felony defendants,” among other policy changes, such as directing judges to consider ability to pay when setting money bail amounts (Rempel and Rodriguez 2019, p. 1). Prior to its enactment, judges in New York had the option to set bail or remand in all cases–regardless of the charge a defendant was facing. The new law proscribes money bail in almost all cases charged with a misdemeanor (except for sex offenses and domestic violence, which make up about 12% of those charged with misdemeanors) and disallows remand for misdemeanors (Rempel and Rodriguez 2019, p. 2). For felonies, in summary, the new criteria where both money bail and remand are possible “permit bail and detention with nearly all violent felonies but rule it out with nearly all nonviolent felonies” (Rempel and Rodriguez 2019, p. 2). In effect, this meant that pretrial release was mandated in nearly all misdemeanor and nonviolent felony cases on the basis of a single factor: the present charge in a case.

We highlight some descriptive statistics which shed light on magnitudes of differences in charge eligibility under the reform. In 2019, “collectively, judges ordered bail or remand in 7 percent of misdemeanors, 35 percent of nonviolent felonies, and 62 percent of violent felonies” (Rempel and Rodriguez 2019, p. 9). Of the cases made bail-eligible, judges ordered bail or remand in 54% of the cases, and of the cases made ineligible for bail, judges used bail or remand 12% of the time. Where judges ordered bail or remand in 2019, in about 13,000 cases they would be unable to do so under the original reform; the greatest difference is in nonviolent felonies (Rempel and Rodriguez 2019, p. 10).

Partly due to concerns related to public safety, six months after the law went into effect, on July 2, 2020, it was amended to expand the eligibility criteria (New York Senate 2019; New York State 2020). The amendments included a list of newly eligible misdemeanor and nonviolent felony charges, as well as broader eligibility criteria related to a defendant’s criminal history, including whether a defendant had a prior history of felony convictions, a separate pending case, or was on probation or parole. These changes effectively ended New York’s charge-based approach to bail reform, replacing it with a second bail reform regime in which bail eligibility was determined based on the present charge or a defendant’s criminal history.

2.2 Evaluations of Bail Reform

The earliest and best known example of bail reform in the United State is Washington, D.C., which in 1992 replaced cash bail with a risk-based system (Lockwood and Griffin 2020). Although no rigorous evaluations were conducted at the time, more recent descriptive statistics reveal a pretrial rearrest rate of 12%–14%, which is broadly in line with that of other jurisdictions that still use cash bail (Advancing Pretrial Policy and Research 2020). In 2017, New Jersey implemented a raft of pretrial reforms, including: (a) mandating the use of a pretrial risk assessment tool, (b) restricting the use of bail to defendants at high risk of flight, and (c) restricting pretrial detention to defendants at high risk of either flight or new criminal activity. While no rigorous studies have yet been conducted on the impact of the reforms, descriptive analyses suggest that after the reform went into effect pretrial rearrest rates did not subsequently rise (NJ Courts 2020). In 2017, Philadelphia’s District Attorney announced that his office would no longer seek bail for 25 low-level felony and misdemeanor offenses. Using an instrumental difference-in-differences design to compare bail eligible offenses to bail ineligible offenses, researchers found that the policy had no impact on the likelihood of either pretrial detention or pretrial rearrest (Ouss and Stevenson 2020). Also in 2017, the Chief Justice in Cook County, Illinois, issued a court order (G.0. 18.8A), establishing a presumption of release for most defendants. Using multivariate models to compare matched samples of defendants pre- and post-reform, researchers found that the order was associated with a 4 percentage-point decrease in pretrial detention but no change in pretrial rearrest (Stemen and Olson 2020). Similar findings were described in an official report (Office of the Chief Judge 2019), but another study of the same reform found that crime among individuals on pretrial release did indeed increase (Cassell and Fowles 2020). In 2019, in Harris County, Texas, a court-ordered consent decree (Rule 9) mandated the pretrial release of a subset of misdemeanor offenses. A descriptive analysis of 8 pretrial rearrest rates among misdemeanor defendants showed no change following the implementation of the order (Duke University School of Law 2020). In 2014, Mecklenburg, North Carolina, a new automated risk assessment tool was rolled out to replace an older risk assessment instrument. Using an interrupted time series design, researchers found that the implementation was followed by an 11 percentage-point decrease in money bail, and a 2 percentage-point increase in pretrial rearrest (Redcross et al. 2019).

Outside of any reform context, research has also found that bail and pretrial detention are associated with a higher likelihood of rearrest or recidivism (Lowenkamp, VanNostrand, and Holsinger 2013; Gupta, Hansman, and Frenchman 2016; Monaghan, van Holm, and Surprenant 2020). Other research has highlighted pretrial detention’s harmful collateral impacts, including its impact on future employment and residential stability for arrested individuals (Heaton, Mayson, and Stevenson 2017; Dobbie, Goldin, and Yang 2018). Additionally, bail and pretrial detention can also intensify racial disparities at later phases of the criminal legal process (Donnelly and MacDonald 2018).

New York’s bail reform is a difference in kind, not only in degree, from other reforms. In the typology of (Jorgensen and Smith 2021, pp. 5–7)’s landscape analysis of bail reforms, in categorizing state-led reforms, during our study period, New York stands alone in the category of “abolish[ing] cash bail for some or all crimes.” (Other states include Illinois and Maine, whose reforms go into effect strictly after our study period). Among city- and county-led reforms, most such reforms are outside our study period. We discuss in greater detail in Appendix A.2.

Finally, we clarify how our study is relevant to general deterrence by studying aggregate crime. There are two different mechanisms to conceptualize impacts of bail reform on crime: general deterrence, which changes the costs/benefits of arrest for committing crime and hence possibly inducing increases in crime rates among the general population, and specific deterrence, regarding incapacitative impacts of pretrial-detention and bail policy on individuals or reducing pretrial re-arrest and/or recidivism specifically in the pretrial population. Our analysis does not inform specific deterrence.

On the one hand, increasing the released population may generally be expected to increase the absolute magnitude of re-arrests, since bail (via detention) has incapacitative effects. Setting monetary bail may also pose financial incentives to prevent reoffense, leading to specific deterrence. The recent analysis of Albright (2021) studies the question of financial incentives in a different context. Therefore, bail reform could introduce societal costs. If bail reform leads to an increase in crime via specific deterrence or via general deterrence, assessing the absolute magnitude helps articulate tradeoffs against other important arguments for bail reform. Dobbie and Yang (2021) overview these considerations balancing individual rights and public interests. On the other hand, perhaps bail reform does not increase aggregate crime (does not reduce general deterrence), so that there are fewer than posited societal costs of bail reform weighed against individual-level benefits. Our analysis indeed suggests this latter case. Such benefits, not included in our quantitative analysis, include avoiding the distributional consequences on detention for those unable to make money bail, civil liberties, racial and ethnic disparities, constitutional and moral arguments, and other important policy considerations.

This study represents the first attempt to evaluate the effects of the bail reform of New York State’s Bail Elimination Act on aggregate crime rates. Bail reform is difficult to evaluate due to (i) confounders (other changes that take place at the same time leading to cross-unit unobserved confounding); (ii) data limitations and difficulties of model specification; and (iii) the uniqueness of particular case studies (findings that hold true in NYC may not hold true elsewhere and therefore are not representative of bail reform’s generalized effects). We address (i) and (iii) by using synthetic control methodology (Abadie, Diamond, and Hainmueller 2010), which has been used to evaluate policy, including on crime (Saunders et al. 2015; Donohue, Aneja, and Weber 2019; Rees et al. 2019; Ben-Michael, Feller, and Raphael 2021). We address (ii) by aggregating open data from municipal police departments. Synthetic control can be argued to estimate causal effects under shared unobserved confounding, while interrupted time series cannot. Unobserved confounders could include broader, shared shifts in crime incidence such as federal or county-level policy and aforementioned mechanisms discussed in the previous section driving crime rates. In Appendix A, we include a discussion on the criminological theory of crime trends as a national phenomenon. The synthetic control reweights the non-NYC municipal police departments in order to track NYC’s pre-intervention trajectory. Divergence in outcomes between NYC and the synthetic control (weighted composite of other cities) allows for a comparative analysis that uses the synthetic control to account for shared aggregate-level changes in crime rates. We include more detail on dataset construction in Section 3 and the synthetic control method in Section 4.

3 Overview of Data

We construct a dataset by compiling crime data released by police departments across 27 cities across the United States from the period January 1, 2018–March 15, 2020 for the purpose of establishing a synthetic control for NYC crime trends. Although the main analysis is synthetic control, we also conduct an interrupted time series analysis with only New York City’s data starting a year prior, from January 1, 2017, as we are less restricted by ensuring data availability across all municipalities for interrupted time series.

Reported incidents of crime were obtained from police department-reported incidents accessible via data portals of the 30 most populous U.S. cities (table of sources in Appendix B). We omitted Dallas, Atlanta and Fort Worth due to significant reporting discontinuities in the data. We removed Philadelphia, San Francisco, and Houston due to significant bail reform that occurred during the study period. We also removed some cities due to poor pretreatment fit as suggested in (Abadie, Diamond, and Hainmueller 2010). Other locations passed bail reforms before, or very near, the start of our synthetic control timeframe on January 1, 2018. These locations are retained in our analysis under an assumption that their prior treatment effects of local reforms were realized by the beginning of our study period. See Appendix B for more details.

Our use of such incident report data comes with standard disclaimers regarding crime data and potential measurement issues: see Kaplan (2020) for extensive discussion of particularities with this type of data. For our comparative case study, measurement biases that do not differ across units do not affect analysis conclusions.

Each row in this dataset corresponds to an incident of crime. The dataset was constructed by merging the reported incidents from each agency. However, each agency reports crime types in a slightly different way, via different descriptors and categorizations. Our approach based on aggregating temporally fine data (weekly) differs from standard approaches that use FBI Uniform Crime Reporting (UCR) data at a monthly timescale. In our dataset, some agencies, but not all, report UCR codes, but at different granularities. We therefore code crime types and construct a schema that broadly resembles UCR’s index crime designations. The crimes at the second level of the hierarchy, at which level we conducted our analysis are:

• Homicide: Murder and manslaughter (does not include justifiable homicide)

• Robbery: All taking of property through force or threat of force

• Assault: Includes aggravated and simple assaults when possible

• Burglary: Unlawful entry into structure (residential and commercial) to commit theft without use of force

• Theft: Unlawful taking of property (includes motor vehicle theft)

• Drug: All drug-related abuses including cultivation, distribution, sale, purchase, use, possession, transportation, or importation of any controlled drug or narcotic substance.

These definitions are modified from other crime frameworks, such as a previous Bureau of Justice Statistics analysis,¹ FBI offense definitions,² and definitions from City Crime Stats.³ See Table 1 for a summary of what these categories include. This study was conducted before the 2020 UCR data were available. Moreover, UCR data reporting is at the monthly level, which would result in only three post-intervention timepoints at the timescale of our post-intervention, pre-COVID window, muddying any inference. In contrast, constructing the dataset at a daily reporting level allows for greater flexibility and improves the precision of our estimates and inferences. In Appendix E.1 we provide a more granular comparison.

Table 1 Summary of crime classifications.

Assault	Theft	Burglary	Drug	Robbery
	Grand larceny auto
Aggravated assault	Grand larceny	Residential burglary
Simple assault	Theft from auto	Non-residential burglary
Unclassified	Other theft	Unclassified
	Unclassified

Bail reform went into effect on January 1, 2020, which we use as the intervention date. However, the post-intervention period is necessarily abbreviated due to COVID lockdowns and COVID-related changes which introduce a large degree of nonstationarity. The pandemic was accompanied by several developments that could be plausibly linked to crime, such as increased economic hardship (Parolin et al. 2020), decarceration (Porter 2020), an uptick in gun purchases (Schleimer et al. 2021) and gun carrying (Arthur and Asher 2021), and halted non-policing efforts to prevent violence (Paterson 2020; Badger and Bui 2020). Including post-pandemic data in our analysis could therefore introduce treatment effects from the pandemic and differential responses to it across municipalities. In NYC, the lockdown for COVID went into effect on March 15, 2020 which we use as the end of the post-intervention period. A partial rollback of bail reform rollback went into effect in New York on July 2, 2020, altogether outside our post- or pre-treatment windows.

Our treatment of interest is specifically the effect of New York State’s bail reform on New York City, which eliminated money bail and pretrial detention for nearly all misdemeanor and nonviolent felony defendants. We only include other units whose other bail reform packages occurred prior to the synthetic control timeframe, in Appendix A.2 we discuss more precise differences.

4 Methods

Our preferred methodology is synthetic control: we now describe its benefits in this section as compared to other event-study methods. Relative to comparative case study approaches that would require choosing a comparator municipality, using synthetic control allows for flexible construction of a comparison, based on fitting the trajectory of crime rates before the intervention. This is helpful because the cities themselves differ on a number of factors (seasonality patterns, opportunities for different types of crime, etc.) so that manually choosing a control comparator is not self-evident in this setting.

Relative to a within-NYC time series analysis, synthetic control allows for accounting for possible co-movements in underlying crime trends (e.g., changes in federal policy or broader economic trends) with other non-treated units; the possibility of such trends is discussed in Section 2. Further, relative to interrupted time series, comparative interrupted time series, or differences-in-differences analysis, an important consideration in favor of synthetic control is that it compares the post-treatment time period to a longer pretreatment time period than would occur otherwise, under segmented regression used for the previous methods. At a domain-level, the crucial assumption of no-anticipation is violated. In our setting, no-anticipation presumes that outcomes do not change due to anticipation of the actual intervention date. However, anticipation was already quantitatively documented by Center for Court Innovation (2020) with increased rates of supervised release in the weeks and months preceding the January 1 implementation of the reform; judges avoided processing individuals for bail or detention who would have been released under the new regime soon afterwards. To conclude, synthetic control is a more appropriate methodology that will additionally handle (a) shared unobserved confounders (such as documented crime trends) and (b) documented violations of a no-anticipation assumption and is preferred to a comparative case study.

The original synthetic control method (Abadie, Diamond, and Hainmueller 2010) uses outcome predictors in order to balance a weighted distance between predictors of treated unit and predictors of control units. We first introduce the terminology and notation of the synthetic control, before describing the variants we introduce following the literature.

Index the units (here, municipalities) as $j=1,2,\dots ,J+1$ and associate the first index, j = 1, to the treated unit (NYC). The possible “donor pool” is a collection of untreated units, $j=2,\dots ,J+1$, not affected by the intervention (bail reform). The data comprises T time periods, with T₀ of these periods occurring before the intervention. For each unit and time period, an outcome Y_jt is measured (aggregate crime level within a categroy). Let $Y_{jt}(1)$ denote the potential response under the treatment (bail reform intervention) and $Y_{jt}(0)$ otherwise, that is, under control. The effect of the intervention of interest for the treated unit at some time $t>T_0$ is $${\tau }_{1t}=Y_{1t}(1)-Y_{1t}(0).$$

Of course, the fundamental problem of causal inference is that for any time $t>T_0$ we only observe $Y_{1t}(1)$, not the post-intervention outcome without the intervention.

The synthetic control method constructs an estimate for $Y_{1t}(0)$ as a reweighted average of donor units’ outcomes, $Y_{jt}(0)$, for whom we do observe outcomes under nonintervention after the intervention time. The synthetic control estimate is specified via a weight vector $W=[w_2,\dots ,w_{J+1}]$ that is used to construct a weighted average of donor units (the synthetic control). We optimize for the synthetic control weights minimizing the predicted mean-square error for pre-intervention outcomes Y plus a ridge regularization: (1) $$w^{*}\in \arg \min \left\{\frac{1}{T_0}\sum _{t\le T_0}{\left(Y_{1t}-\sum _{j=2}^{J+1}{w}_j{Y}_{jt}\right)}^2+\lambda \Vert w{\Vert }_2:\sum _{j=2}^{J+1}{w}_j=1\right\}.$$(1)

The above program is easily solvable as a convex quadratic program. Then the synthetic control estimates of the counterfactual and treatment effect are, respectively: $${\widehat{Y}}_{1t}(0)=\sum _{j=2}^{J+1}{w}_j^{*}{Y}_{jt},\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}{\widehat{\tau }}_{1t}=Y_{1t}(1)-{\widehat{Y}}_{1t}(0).$$

Our specification differs somewhat from Abadie, Diamond, and Hainmueller (2011) and but follows common choices used in the literature. We implement synthetic control with unlagged outcomes and without covariates. (Including covariates would be either too noisy or redundant (Kaul et al. 2015).) For better interpretability and to avoid hyperparameter tuning, we do not consider re-weighting mean-squared errors as in the original paper, that is, a reweighting hyperparameter across donor units. We include a ridge regression penalty, following a suggestion of Abadie, Diamond, and Hainmueller (2015), and allow negative weights. Doudchenko and Imbens (2016) discusses some of these relaxations.

Due to the widely varying absolute magnitudes of crime in different cities—since NYC is the most populated municipality—we normalize the crime series in each city by the city’s population (not metropolitan statistical area). Similarly, Abadie, Diamond, and Hainmueller (2010) considers per-capita smoking rates. We also demean the trends in order to build the synthetic control weights, subtracting the pretreatment average for each city. This normalization is crucial for our permutation-based placebo checks down the line, as we assess the magnitude of the treatment effect for NYC relative to the magnitude of the treatment effect assessed under placebo checks (more details in Section 5).

5 Results

Our primary analysis is with synthetic control. We assess uncertainty via placebo checks and other robustness checks that are common in the literature. In Appendix C, for completeness we also include an interrupted time series analysis.

5.1 Implementation and Details

We first describe data pre-processing steps used to ensure quality of the synthetic control fit, as well as describe our use of placebo checks common in the synthetic control literature to quantify the uncertainty in the treatment effect estimate, before describing the inferential results.

We consider estimates of an average treatment effect by averaging post-intervention outcomes to estimate the treated outcome and estimating the control outcome for NYC by reweighting with the synthetic control weights: (2) $$\widehat{\tau }=\frac{1}{T-T_0}\left(\sum _{t=T_0}^T{Y}_{1t}-\sum _{j=1}^{J+1}{w}_j^{*}{Y}_{jt}\right).$$(2)

We analyze Assault, Theft, Drug, Robbery, and Burglary crimes. Assault and Robbery are categorized as violent crimes while Burglary and Theft are categorized as property crimes. The number of homicides is quite small so that the analysis for synthetic control is too noisy on this timescale. For example, Tables 2 and 3 show it is three order of magnitudes more sparse than burglary and robbery. Due to data sparsity and noisy outcomes, we aggregate different crime types to different temporal aggregations by summing weekly counts over longer time intervals. Assault and Theft crimes remain on a weekly aggregation, Burglary and Robbery crimes are aggregated to 5 weeks, while Robbery crimes are aggregated to 2 weeks and Drug crimes to 3 weeks.

Table 2 Data descriptives: crime time series pre-intervention, daily counts.

Statistic	Mean	St. Dev.	Min	Pctl(25)	Pctl(75)	Max
Assault	246.566	29.849	135	226	267	359
Theft	373.182	50.718	150	339	410	517
Burglary	31.537	7.571	12	26	37	57
Drug	45.642	21.013	1	30	60	105
Robbery	36.747	8.082	18	31	42	71
Homicide	0.841	1.054	0	0	1	9

Table 3 Data descriptives: crime time series post-intervention, daily counts.

Statistic	Mean	St. Dev.	Min	Pctl(25)	Pctl(75)	Max
Assault	250.068	21.694	197	234.2	265	318
Theft	381.851	35.118	302	358.2	404.5	462
Burglary	36.135	7.206	19	31	41.5	52
Drug	39.149	17.633	8	22.2	51.8	73
Robbery	40.351	7.023	19	36	44	58
Homicide	0.797	1.033	0	0	1	4

The data-pre-processing steps are fairly standard in the synthetic control literature, and include tuning the ridge penalty by pretreatment prediction quality (RMSE, root mean-squared error), determining aggregation level for outcome series by pretreatment fit, omitting cities based on pretreatment fit more than 7.5 times that of the original synthetic control (as suggested in Abadie, Diamond, and Hainmueller 2011). In the interest of brevity, we discuss data-driven or other justifications in Appendix B.

5.2 Results: Linear Regression with Ridge Penalty Specification

Our preferred specification is based on synthetic control via linear regression with a ridge penalty. We include a visual comparison of the NYC time series, as well as the imputed synthetic control fit; visual overlay of the placebo fits and the NYC-treated fit, a summary table of ATE estimate and p-values, as well as summary statistics on pretreatment fit, testing at level $\alpha =0.1$ adjusted for multiple-tests via Holm-Šidák correction, as well as the output of synthetic control weights. Then we include tables of placebo checks: the in-time placebo checks and early roll-in of intervention placebo checks. We walk through this series of analyses for our preferred specification, and summarize the main inferential conclusions across specifications.

In Figure 1, we show results for the synthetic control analysis for Assault and Property Theft outcomes, computed on a weekly aggregation, Burglary and Robbery, after aggregating event counts to a five-weekly coarsening. The other crime types are much lower frequency (ranging from 20% to 5% of the average frequencies as Assault). Therefore, aggregating at the temporal level is required to reduce noise of the synthetic control estimation. For purposes of display, we graphically display loess-smoothed time series (using 7% of the data to estimate each value). In the solid black line, we display the time series of observed cases. The vertical dark lines indicate the time of intervention and COVID lockdown (end of study period), respectively. The dashed black line indicates the predicted NYC time series. The average treatment effect estimate differences the average of the post- and pre-treatment difference between these time series. The three crimes, assault, theft, and drug crimes, have good pretreatment fit: the predicted time series closely matches the observed time series. The burglary and robbery time series are higher-variance and pretreatment fit is not as close, but matches overall trends. For assault, theft, and drug, the post-treatment difference visually appears fairly close. For burglary and robbery, the post-treatment difference visually appears to increase, but due to the noise in pretreatment fit, it’s not clear whether this can be statistically distinguished from noise. Next we describe the treatment effects quantitatively.

Fig. 1 Synthetic control plots, ridge (Loess-smoothed for display).

In Table 4, we summarize the main numerical results of the synthetic control analysis, for each crime type. We report the averaged ATE estimate (on the event scale, normalized to be a rate per 1000 individuals). That is, the synthetic control comparison suggests that the increase in average weekly violent assaults is 0.0083 events per 1000 individuals, relative to the synthetic control. The “p (ATE)” column reports the p-value from the unit-level placebo distribution of the ATE estimate, while the “p (RMSE)” column reports the p-value from the RMSE test statistic. We also report measures of pre-intervention predictive fit of the synthetic control in predicting observed outcomes: “Pre R²” is the R² value of the pretreatment fit for the main synthetic control analysis where NYC is the treated unit. We also compare the pre-intervention RMSE (“Pre RMSE”) and average placebo pre-intervention RMSE (“Plac. RMSE”). The last comparison, pre-intervention RMSE versus average placebo pre-intervention RMSE, provides a sense of the predictive fits for the placebo synthetic controls. Finally we report the number of placebo units (“Num. plac.”) that remain after the RMSE prescreening step.

Table 4 Synthetic control (ridge) tabular results.

	ATE (/1000)	p (ATE)	p (RMSE)	Pre R^2	Pre RMSE	Plac. RMSE	Num. plac.
Assault	0.0083	0.35	0.53	0.76	$8.19\times {10}^{-6}$	$2.38\times {10}^{-5}$	20
Theft	0.0274	0.18	0.1	0.64	$2.02\times {10}^{-5}$	$4.54\times {10}^{-5}$	22
Burglary	0.0458	0.15	0.37	–1.32	$1.53\times {10}^{-5}$	$2.45\times {10}^{-5}$	20
Robbery	0.0304	0.05	0.0	0.7	$7.4\times {10}^{-6}$	$1.62\times {10}^{-5}$	20
Drug	–0.014	0.71	0.23	0.7	$8.66\times {10}^{-6}$	$2.34\times {10}^{-5}$	14

To put the effect sizes in context, the ATE point-estimate in number of assault events is 69.9 events relative to approximately 1760 weekly assault events; or an estimated 3.9% relative percentage increase. For Theft, the ATE point-estimate is 230.7 events relative to approximately 2667 weekly events, or 8.7% relative increase. To convert our per-capita estimates to absolute events, we obtain this estimate by computing $0.0083/1000*\text{2020 NYC population}$. Further, to characterize the extremal boundaries of effect sizes that would result in failing to reject the null hypothesis, we report lower and upper bounds from the placebo test distribution in Table 5. We also report multiple testing adjustments via the Holm-Šidák correction in Table 6 which adjust for the probability of rejecting the null by chance over multiple comparisons.

Table 5 Lower and upper confidence bounds on effect sizes (rejected under the null).

	Lower bound (5th percentile)	Upper bound (95th percentile)
Assault	$-4.95\times {10}^{-5}$	$2.01\times {10}^{-5}$
Theft	$-5.95\times {10}^{-5}$	$5.83\times {10}^{-5}$
Burglary	$-1.54\times {10}^{-4}$	$8.34\times {10}^{-5}$
Robbery	$-5.47\times {10}^{-5}$	$3.08\times {10}^{-5}$
Drug	$-7.95\times {10}^{-5}$	$4.26\times {10}^{-5}$

Table 6 Multiple testing adjustment, ridge (for ATE, RMSE test statistics separately).

	Robbery	Theft	Burglary	Assault	Drug
Adjusted p (ATE)	0.23	0.48	0.48	0.58	0.71
Conclusion	Fail to reject	Fail to reject	Fail to reject	Fail to reject	Fail to reject
Adjusted p (RMSE)	0.0	0.33	0.54	0.6	0.6
Conclusion	Reject	Fail to reject	Fail to reject	Fail to reject	Fail to reject

For property theft, robbery, and drug, there is some sensitivity to conclusions depending on the test statistic (although with a multiple testing-adjustment via Holm-Šidák correction reported in Table 6, at a $\alpha =0.1$ level we only reject the null for the violent robbery crime type). For burglary, we have the most consensus between the ATE and RMSE test statistics of non-significance of the increased effect.

For interpretation, in Table 7, we display the weights chosen by the synthetic control. Note that we allow for a degree of extrapolation by allowing for negative weights.

Table 7 Synthetic control tabular results, ridge, weekly aggregation.

City	Assault	Theft	Burglary	Robbery	Drug
Austin	0.04	0.05	0.07	0.07	0.27
Baltimore	0.01	0.05	0.04	–0.0	–
Boston	0.07	0.05	0.07	0.06	–0.03
Buffalo	0.03	0.02	–0.02	0.03	–
Chicago	0.01	0.05	0.05	0.04	0.06
Cincinnati	0.01	0.06	0.0	0.0	–
Dallas	0.07	0.07	0.06	0.05	0.14
Denver	0.07	0.02	0.06	0.05	–0.11
Detroit	–0.01	0.04	0.03	0.05	–0.07
Kansas City	0.01	0.01	–0.01	–0.0	–0.01
Little Rock	0.03	0.0	0.01	0.05	–
Los Angeles	0.06	0.08	0.08	0.06	0.3
Louisville	0.03	0.02	0.05	0.06	0.01
Milwaukee	0.04	0.04	0.01	0.03	–
Nashville	0.04	0.06	0.01	0.01	0.11
Philadelphia	0.05	0.05	0.06	0.06	0.04
Phoenix	0.09	0.07	0.07	0.06	0.15
Portland	0.06	0.05	0.06	0.06	0.18
Raleigh	0.07	0.06	0.06	0.06	–0.05
Seattle	0.06	0.05	0.06	0.06	0.02
Virginia Beach	0.06	0.05	0.08	0.07	0.02
Washington DC	0.09	0.03	0.07	0.05	–
Intercept	–0.0	0.0	0.0	0.0	–0.0

To summarize the findings from Tables 4 and 6: qualitatively, we would conclude that significance testing at $\alpha =0.1$ would result in failure to reject the null hypothesis of no increase in crime, except for violent robbery. That is, for most crimes (except violent robbery), we do not find statistically significant evidence of an increase in aggregate crime.

In Figure 2, we graphically display the placebo distribution and the NYC residuals in the context of the generated leave-one-out placebo distribution. For each city acting as the treated unit, we plot the residual series: actual observations minus the predicted values. This provides a qualitative sense of the validity of the placebo distribution. While the placebo fits tend to be noisier than the original NYC fit, the average RMSE is of the same order of magnitude. Comparing Violent Assault and Property Theft suggests that post-intervention deviations for the unit-level placebo checks are overall of similar magnitudes to the NYC-treated analysis.

Fig. 2 Unit-level placebo checks (ridge; Loess-smoothed for display).

We conduct a placebo check by assigning each other city as the treated unit, then recomputing a synthetic control from the other cities. We follow the suggestion of Abadie, Diamond, and Hainmueller (2011) to compute the distribution of placebo effects, restricting attention to those cities where good pretreatment fit is possible, relative to the original synthetic control. We consider two-sided tests of the weak null.

One test statistic that we assess via the placebo test is the average treatment effect estimate (ATE) defined in (2). We then consider the p-value of the ATE measured for NYC with respect to the distribution of ATEs from the placebo checks. Namely, we set the p-value to $1-F(\widehat{\tau })$, where F is the cumulative distribution function of the placebo ATE estimates and $\widehat{\tau }$ is the ATE estimate measured for NYC.

Another test statistic is the RMSE (root mean-squared error) statistic used in Abadie, Diamond, and Hainmueller (2011). The statistic r is given by the ratio of post-treatment RMSE and pretreatment RMSE: (3) $$r=\frac{\frac{1}{T-T_0}\sqrt{\sum _{t=T_0+1}^T{(Y_{1t}-{\widehat{Y}}_{1t})}^2}}{\frac{1}{T_0}\sqrt{\sum _{t=0}^{T_0}{(Y_{1t}-{\widehat{Y}}_{1t})}^2}}.$$(3)

We can similarly compute placebo RMSE statistics by permuting the identity of the treated unit and the p-value as the cumulative distribution function of the primary RMSE statistic among these placebo values. Relative to the more interpretable ATE test statistic, the RMSE test statistic normalizes by the pretreatment fit under a certain placebo. If a placebo unit, when considered as the treated unit, achieves a poor pretreatment fit, then any extreme differences in post-treatment contrasts, which may rightly be attributed to the poor pretreatment fit, are made less extremal in the placebo distribution by the normalization.

In Appendix D, we include additional robustness and placebos in time.

Overall the results of the placebo check suggest that under alternative specifications where we substantively know there is no treatment effect, we recover similar order of magnitude ATE estimates as under our main specification. Robbery does have some extremal p-values under the placebo check: it may suggest that the data series is quite noisy such that known null effects may nonetheless be extremal. Otherwise however, no statistical significance would be detected under the placebo specifications. The same conclusion holds under the early-roll-in of intervention start placebo check. Notably, despite the documented increase in ROR rates as we moved the intervention date earlier in time, we do not detect significant or extremal effects.

In conclusion, using synthetic control methodology that is more approprirate in this setting shows overall null effects of bail reform on aggregate crime. Synthetic control can account for unobserved confounders and anticipations of policy changes.

6 Limitations

Our analysis of the NYC bail reform, conducted via aggregating police department-level incident reporting data, with a short post-intervention timespan before the onset of COVID, is subject to important limitations. Below, we describe these with relevant descriptive statistics and conclude, when possible, how these limitations may impact our estimates. We describe limitations due to studying general versus specific deterrence, the short study period, a one-time retroactive release of defendants that occurred, and possible anticipation effects.

The proportion of cases directly impacted by bail reform is small

The primary way bail reform could impact crime is through the release of defendants who would’ve otherwise been detained pretrial (i.e., bail not paid/remanded). Between January 1, 2020 and March 15, 2020, there were a total of 18,485 cases with defendants who were (a) continued at arraignment, (b) ineligible for bail (flagging eligibility based on charge), and (c) not bailed/remanded at arraignment.

Though these cases were ineligible for bail, we need to further estimate the percentage that would have been bailed/remanded had the new bail law not been in effect. During the same period in 2019, 16.3% of defendants were bailed/remanded in cases that met the conditions above (excluding condition 3, not bailed/remanded). 55% of these defendants were eventually released back into the community during the pretrial period. This matters if we’re thinking about the marginal difference in terms of time at risk in the community, since after a release there would be little difference between these cases and cases mandatorily released post-reform. Applying this percentage to 2020 yields: 3013 cases that would otherwise have been bailed/remanded. This small proportion of cases impacted could lead to a downward bias of our estimate since NYC was already releasing a large percentage of defendants, so the number of defendants on the margin who were affected by the reform was low.

Short study period

The post-intervention period in our study runs from January 1, 2020 to March 15, 2020—only a 75-day period.

Among time to disposition of new incidents, and the amount of time on release in the community, the window with potential for rearrest in our post-intervention period is small.

In NYC, speedy trial rules require that misdemeanor cases be resolved within 60–90 days and felony cases within 180 days. However, we know the typical time to disposition is much longer (e.g., in 2019, the average time to disposition in indicted felony cases was 10 months (Weill et al. 2021)). On the other hand, during the 75-day study period, the number of days a case could have been on release in the community ranged from 1 to 75, with an average of about 30 days (assuming an equal distribution of cases for each day during the 75-day period).

This means that for the roughly 3000 defendants that would have otherwise been bailed/remanded, the study only captures a small portion of their overall time at risk (3000 * 30 days = 90,000 person-days of risk). In 2020, the pretrial rearrest rate for cases tracked through March 2021 was about 20% according to Office of Court Administration, (New York State Unified Court System). Given the short time at risk in the study (30 days on average), we need to assume a much lower rate of rearrest relative to statistics derived from longer time periods. For the sake of argument, suppose the pretrial rearrest rate from our shorter time period was 10%, so 300 of the 3000 defendants impacted by the reform were rearrested by March 15, 2020. On the other hand, the clearance rate during the study period was about 1 in 3 (NYPD 2020). A worst-case analysis, assuming these individuals were responsible for 3x the number of rearrests, suggests 900 rearrests in total. Overall, the impact of a shorter time frame for the study suggests that our estimates could be biased downwards from the estimate with a longer time-period, because of the short time at risk for re-offense.

Seasonality

The study occurs entirely during the winter months when crime is at its lowest ebb. While the models account for temperature and month-week fixed effects, we may not have enough historical data to identify the true seasonal patterns. This could bias estimates downwards because the post-intervention period is entirely within a seasonal lull in crime.

One-time retroactive release of defendants

It was estimated that about 900 defendants were retroactively released under the new law (Rosenberg and Golding 2019). Our estimates could therefore overstate the effect of the policy, since during the study period there were many more impacted defendants (released) than there would normally have been. Our early intervention placebo checks verify that the null results we observe are not due to a cancelation in positive treatment effect. If there were a positive treatment effect, due to this large retroactive release in November, a concern is that using our later intervention date wrongly suggests a null. But in our early intervention roll out placebo checks, we verify that these would have remained null findings, addressing this counterargument.

Anticipation effects

Many of the retroactive releases (described above) occurred ahead of the policy start date. Also, there is some evidence that judges had already started to change their behavior in anticipation of the new law in November–December, 2019 (less likely to bail/remand defendants who would soon be released anyway). Although this could lead to a downward bias in estimates, this is addressed in our early intervention roll-in sensitivity analysis, which also assesses null effects. The earlier sensitivity analysis suggests that this is not a serious limitation.

7 Conclusion

This article studies the impact of bail reform on aggregate crime, using the case study of New York’s 2020 reforms. Subject to the limitations discussed above, we find evidence that bail reform did not increase aggregate crime rates in NYC.

We consider a subset of crime outcomes, Assault, Theft, Drug, Burglary, and Robbery, which are the most frequent time series. Our main analysis uses the synthetic control method, which allows for adjusting for cross-unit variation by comparing NYC’s trend to a synthetic control, constructed by weighting the time series of other cities. In order to provide some sense of uncertainty quantification, we conduct a series of placebo checks as conventional in the synthetic control literature, and report p-values from the placebo distribution of ATEs that arise from fixing another unit as the treated outcome, to assess the variation in null effects arising from synthetic control. The results from the synthetic control overall fail to reject a null hypothesis of no effect for most outcomes (before and after adjusting for multiple comparisons); with the exception of rejecting the null (finding a statistically significant increase) for the robbery outcome. Our analysis focused on assessing the impacts of bail reform on aggregate crime rate, hence, informing general deterrence considerations of the broader policy context. Overall, we find evidence that bail reform did not increase aggregate crime rates in New York City.

Supplementary Materials

Supplementary materials include appendices with further discussions and description of the dataset and its construction and an archive of code and processed data. Raw data is available for download at https://www.openicpsr.org/openicpsr/project/152641/version/V1/view.

Disclosure Statement

The authors report there are no competing interests to declare.

Additional information

Funding

Nathan Kallus and Angela Zhou were supported under NSF #939704.

Notes

1 https://www.bjs.gov/recidivism/

2 https://ucr.fbi.gov/crime-in-the-u.s/2019/crime-in-the-u.s.-2019/topic-pages/offense-definitions

3 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3674032