Observational Study of the Effect of the Juvenile Stay-At-Home Order on SARS-CoV-2 Infection Spread in Saline County, Arkansas

Abstract

We assess the treatment effect of juvenile stay-at-home orders (JSAHO) on reducing the rate of SARS-CoV-2 infection spread in Saline County (“Saline”), Arkansas, by examining the difference between Saline’s and control Arkansas counties’ changes in daily and mean log infection rates of pretreatment (March 28–April 5, 2020) and treatment periods (April 6–May 6, 2020). A synthetic control county is constructed based on the parallel-trends assumption, least-squares fitting on pretreatment and socio-demographic covariates, and elastic-net-based methods, from which the counterfactual outcome is predicted and the treatment effect is estimated using the difference-in-differences, the synthetic control, and the changes-in-changes methodologies. Both the daily and average treatment effects of JSAHO are shown to be significant. Despite its narrow scope and lack of enforcement for compliance, JSAHO reduced the rate of the infection spread in Saline. Supplementary materials for this article are available online.

KEYWORDS:

• Changes-in-changes
• Conformal inference
• Counterfactual prediction
• COVID-19
• Difference-in-differences
• Elastic-net
• SARS-CoV-2
• Stay-at-home order
• Synthetic control

1 Introduction

In response to rising numbers of Covid-19 cases, state and local governments in the United States have used their jurisdictional authorities to implement a wide class of public health and social measures (PHSMs) aimed at slowing the spread of the SARS-CoV-2 (“Covid-19”) infections. At times, local governments have stepped up and issued orders when the policies they deemed necessary were not implemented at the state level. Much of these policy responses has focused on enforcing social-distancing through measures ranging from temporary closures of public-facing businesses, shelter-in-place orders (SIPO), and stay-at-home orders (SAHO) to mandatory mask-wearing orders (Courtemanche et al. 2020; Friedson et al. 2020; Hsiang et al. 2020; Dave et al. 2021; Abouk and Heydari 2021; Chernozhukov, Kasahara, and Schrimpf 2021a). There have been a number of studies to: quantify association between Covid-19 infection spread and implementation of SIPO and SAHO (Gao et al. 2020; Le et al. 2020; Lurie et al. 2020); evaluate causal effects of SAHO on restraining social interactions (e.g., see Abouk and Heydari 2021) and mobility (Chen et al. 2020); understand the effectiveness of SIPO in reducing the number of cases and fatalities (Friedson et al. 2020; Courtemanche et al. 2020); and demonstrate the importance of time (relative to the pandemic) and place of the implementation of SIPO on its impact.

SIPO is the most restrictive form of social-distancing measures with its compliance assurance coming from law enforcement and punitive fines (Caswell 2020; Friedson et al. 2020; Napoleon 2020; Dave et al. 2021) as well as social pressures (Ronayne and Thompson 2020; Dave et al. 2021). Some governments have considered more lenient measures, including issuing advisories (instead of orders) for all residents to restrict their movements out of home only to essential ones and narrowing the scope of orders to a certain segment of the residents. However, strict adherence to such measures is often not enforced; instead, its effectiveness to a large extent relies on the willingness of residents to voluntarily modify their behaviors to comply in light of the pandemic (Arey 2020). A natural question that arises is whether these measures are effective and, if so, to what extent.

The juvenile stay-at-home order (JSAHO) was put into effect in certain parts of Saline County of Arkansas (“Saline”) (Arey 2020; Simpson 2020) from April 6 through May 6 as a response to the pandemic. Its scope was limited in two respects. First, while JSAHO was declared by Saline County, it only applied to unincorporated areas of the county. Of the incorporated cities and townships in Saline, Benton City was the only one to implement JSAHO over the same period. The unincorporated areas and Benton had a combined population of 45,231, or approximately 42% of the county population according to the 2010 U.S. Census. Second, JSAHO was restricted to residents under the age of 18 and allowed them to leave home only if accompanied by adults. To the best of our knowledge, Saline was unique among the counties in Arkansas in having direct jurisdiction to impose JSAHO on certain parts of the county without having to work through local townships.

Among the few states in the United States that did not impose either SIPO or SAHO, Arkansas is unique in that a narrower-scope SAHO was nonetheless implemented in one of its counties. This provides an invaluable setting for a natural experiment to assess the effectiveness of the policy action adopted at the county level while controlling the effects of other policies that were concurrently implemented at the state-level.

Our main objective in this article is to study the effectiveness of policy measures on reducing Covid-19 infection spread when the scope is limited, and compliance is largely voluntary. The details of the methods used in the article are covered in Section 2. The results on the treatment effect of JSAHO based on the log infection rates is covered in Section 3. Section 4 contains a discussion on the results of the analysis and the effect of JSAHO on public health. The supplementary material contains an alternative analysis based on raw (rather than log) infection counts, along with a comprehensive list of tables and figures referenced throughout this article. Our implementation code in R is also available as supplementary material.

2 Methods

Drawing causal inference often entails choosing a comparison group to construct a “synthetic unit” and using it to estimate counterfactual outcomes, and computing and checking the significance of implied treatment effects. In this section, we discuss each of these steps in sequence. To start, we first describe the notation we use in this article.

2.1 Notation

A county i is classified as either a treatment group G_i = 1 or a control group G_i = 0, and is observed in time points $t\in \{0,\dots ,T_0,T_0+1,\dots ,T\}$ with the first T₀ points denoting the pretreatment period and $\{T_0+1,\dots ,T\}$ denoting the treatment period. Our outcome of interest is the log of infection rates, defined as the number of infections per 100,000 residents in a county and denoted Y_it. As is standard in the literature, we use the notation introduced in (Rubin 1974, 1978) and denote by $Y_{1t}^N$ the potential outcome for the treated county without treatment and $Y_{1t}^I$ with treatment. Observed outcome for county i at time t is denoted Y_it. Hence, the treatment effect is denoted ${\theta }_t=Y_{1t}^I-Y_{1t}^N$. In our study, Saline County is the sole treatment group and “control county” refers to either one of the other counties in Arkansas that meet the “parallel-trends” criteria or a synthetic county, as discussed in Section 2.2 below. {$Y_{0t,i}{\}}_{i\in [N_C]}$ denotes the set of counties that comprise the synthetic control county and N_C refers to the number of counties in that set. Following the terminology in the matching literature, we refer to the pool of potential counties from which a comparison group is chosen as the “donor pool.”

2.2 Selection of Control Counties

The composition of synthetic control depends on the choice of a similarity measure to be optimized relative to the treatment unit. A common approach to selecting a control group is to apply the parallel-trends assumption, in which those whose fitted lines based on pretreatment outcomes are most parallel to the treatment unit’s comprise the comparison group.¹ Other approaches select control groups that minimize the difference in pretreatment observations and covariate values deemed relevant for the situation.² Depending on the specific form in which individual units are combined to produce the counterfactual outcome, the set that optimizes the given similarity measure is selected. Different methodologies make different assumptions on the parameters of this counterfactual estimation model. Popular methodologies in the Covid-19 policy evaluation literature include difference-in-differences (DID) (Ashenfelter and Card 1984; Card 1990; Meyer, Viscusi, and Durbin 1995; Bertrand, Duflo, and Mullainathan 2004; Abadie 2005; Doudchenko and Imbens 2016; Abouk and Heydari 2021), synthetic control (SC) (Abadie and Gardeazabal 2003; Abadie 2005; Abadie, Diamond, and Hainmueller 2010, 2015; Hainmueller 2012; Doudchenko and Imbens 2016; Friedson et al. 2020), and elastic-net-based methods (Doudchenko and Imbens 2016; Chernozhukov, Wüthrich, and Zhu 2021b). Other methods discussed in the causal inference literature include propensity scores matching (Courtemanche et al. 2020), event-studies (Courtemanche et al. 2020; Dave et al. 2021; Abouk and Heydari 2021), matrix completion, interactive fixed effects, and changes-in-changes (CIC) (Athey and Imbens 2006).

An assumption that is of particular importance in the literature is that counterfactual outcomes can be represented as a linear combination of control unit outcomes, that is, $Y_{1t}^I=a+{\sum }_{i=1}^{N_C}{b}_i·Y_{0t,i}$. Doudchenko and Imbens (2016) presented a simple canonical context for considering the parameteric constraints underlying many popular linear methods, namely:

1. a = 0 (systematic additive difference between the control and the treated);

2. $b_i\ge 0,i\in [N_C]$ (positive coefficients);

3. ${\sum }_{i=1}^{N_C}{b}_i=1$ (coefficients adding up to 1);

4. $b_i=b_j,\forall i\ne j,b_i,b_j\ne 0$ (identical coefficients).

As we will see shortly, constraints (1)–(3) characterize SC and several of its variants, and (2)–(4) comprise what is known as the “parallel trends” assumption that underlie DID, while certain elastic-net based methods relax these constraints (Doudchenko and Imbens 2016). On the other hand, CIC allows the underlying counterfactual representational form to be nonlinear (Athey and Imbens 2006). Below, we discuss each of these methodologies in the context of control county selection.

2.2.1 Control Selection Based on the Parallel Trends Assumption

DID relies on the “parallel trends assumption,” that is, that the time trends for the treatment and control counties are similar, and assigns equal positive weighting to each control unit that adds up to 1. To select counties that best satisfy this assumption, we plotted the gap in logs of pretreatment infection rates for each county in Arkansas that had reported pretreatment infections:(1) $$\begin{array}{c}\hspace{0.17em}\text{log}\hspace{0.17em}(\text{infection}\mathrm{}\text{rate}\mathrm{}\text{in}\mathrm{}\text{Saline})\\ \hspace{1em}-\hspace{0.17em}\text{log}\hspace{0.17em}(\text{infection}\mathrm{}\text{rate}\mathrm{}\text{in}\mathrm{}\text{Control}\mathrm{}\text{County})\end{array}$$(1) and chose as control counties those for which the trends of this difference in logs are relatively flatter over the latter part of the pretreatment period, that is, days near the JSAHO start date of April 6. We use SC-DID to denote the synthetic county comprised of counties selected based on the parallel trends assumption.

2.2.2 Synthetic Control Construction

Constructing the comparison group for SC is a bit more involved. This is due in part to the question of whether considering socio-demographic covariates on top of pretreatment outcomes leads to a more accurate synthetic treated unit (see Doudchenko and Imbens (2016) for the negative argument and Abadie and Gardeazabal (2003) and Abadie, Diamond, and Hainmueller (2015) for the affirmative). We construct the synthetic control county in two ways. In one approach, we only use pretreatment outcomes to construct the comparison group. In the other approach, we consider both the pretreatment outcomes and socio-demographic covariates deemed relevant in the literature. We discuss each of these approaches in more detail below.

SC Based Only on Pretreatment Data

In this approach, we follow the methodology outlined in Doudchenko and Imbens (2016) and consider different choices for numbers of counties in the synthetic county, ranging from 3 to 8. These values are obtained by varying the size of the donor pool of counties in SC. One rationale for this approach is that by conditioning on the pretreatment trends, the counterfactual estimation models may effectively account for effects of covariates that contribute to the pretreatment trends as long as the covariate effects do not interact with the policy. For example, compared to control counties, Saline has the lowest ratio of smokers and the heart disease mortality rate (see Tables 4, 5, and 6 in the supplementary materials). While these covariates may impact infection rates, there is no obvious reason to suspect that the effects of such covariates on COVID susceptibility should dramatically change before and after the implementation of JSAHO. We use SC to denote the synthetic county comprised of counties selected using this methodology.

SC Based on Both Pretreatment and Covariate Data

We proceed as in Abadie and Gardeazabal (2003) and Abadie, Diamond, and Hainmueller (2015) and incorporate a total of 82 socio-demographic covariates discussed in the literature in addition to pretreatment log infections. Tables 4, 5 and 6 in the supplementary materials list all of the variables we used in our synthetic control county construction and show how Saline compares vis-a-vis control counties along each covariate. We denote the synthetic county constructed under this approach by ${\text{SC}}_{\mathrm{C}}$.

In Tables 4, 5, and 6 in the supplementary materials, one can observe that Saline ranks close to the median on most covariates with a few exceptions. To mention a few, note that while Saline has much lower poverty rate and the social vulnerability index (SVI) than the control counties, the association between these covariates and the infection rate is mixed with some studies reporting that higher income levels and lower disability rates are correlated with higher infection rates (Masetti et al. 2020) while others observe the opposite (Brown and Ravallion 2020; Siddique et al. 2020). Further, it was observed that counties with very low poverty levels reported the highest number of cases (Jung, Manley, and Shrestha 2021). Saline also has lower smoker percentage and heart disease mortality rate, but these covariates have been found to be associated with mortality rates of the disease rather than infections (Abedi et al. 2020; Masetti et al. 2020; Brown and Ravallion 2020). For a more complete discussion of the covariates considered in our analyze, see the supplementary materials.

As noted in Doudchenko and Imbens (2016), incorporating covariates other than pretreatment data into counterfactual estimation is not feasible in model specifications that allow either nonzero systematic additive difference between the treated and control (e.g., DID), or control weights to sum to a value other than one (e.g., elastic-net-based methods). Hence, we consider pretreatment outcomes only for SC.

2.3 Counterfactual Estimation

Broadly, there are two ways to estimate counterfactual outcomes. One approach (“linear methods”) assumes that the counterfactual outcome is a linear combination of the observed outcomes of the control group, while the other (“nonlinear methods”) relaxes this assumption. The former can be further classified depending on particular modeling constraints. For instance, in SC-DID, each control unit is given the same weight that adds up to one and a permanent additive difference exists between the treated and control groups. SC differs from SC-DID in that weights are allowed to differ from each other as long as they remain positive and there is no systematic additive difference between the treated and control. For the classic SC method that makes explicit the role of covariates, see Abadie and Gardeazabal (2003), Abadie, Diamond, and Hainmueller (2010), and Abadie, Diamond, and Hainmueller (2015). Lastly, Doudchenko and Imbens (2016) proposed a method in which the constraints in SC, namely, the positive weights and the zero additive difference, are relaxed in exchange for the elastic-net penalty term to narrow down the optimization search space. We refer to this method as “SC-EN” which can be written as follows:(2) $$\begin{array}{c}(\widehat{\mu },\widehat{\omega })=\underset{\mu ,\omega }{\text{arg}\text{min}}\sum _{i=1}^{T_0}(Y_{1t}-\mu -\sum _{i\ne 1}{\omega }_i·Y_{it})\\ \hspace{1em}+\lambda \left(\frac{1-\alpha }{2}\right|\left|\omega \right|{|}_2^2+\alpha \left|\right|\omega \left|{|}_1\right).\end{array}$$(2)

Nonlinear methods relax the aforementioned constraints. CIC relaxes a key assumption in DID that outcomes are linear in time and group membership. Instead, it assumes that a general, nonparameteric monotone function maps unobserved, time-invariant but unit-dependent, characteristic to an outcome of interest.

In this article, we consider the following methods to estimate the counterfactual outcomes of Saline and assess the implied treatment effects:

1. SC-DID, SC, and SC-EN as laid out in Doudchenko and Imbens (2016), with SC estimated based only on pretreatment outcomes;

2. ${\text{SC}}_{\mathrm{C}}$ estimated with both pretreatment outcomes and 82 covariates as presented in Abadie and Gardeazabal (2003) and Abadie, Diamond, and Hainmueller (2010, 2015) and,

3. CIC with the comparison group constructed with SC-EN.

The empirical results corresponding to these methods are presented in Sections 3.2, 3.3, and 3.4, respectively.

2.4 Treatment Effect Estimation

In this article, we demonstrate evidence of the causal effect of JSAHO in two ways. First, we show that pointwise (i.e., daily) confidence intervals of the treatment effect with the synthetic comparison county constructed based on the parallel-trends assumption and subset selection using the squared differences are significant using the conformal inference method in Chernozhukov, Wüthrich, and Zhu (2021b). Second, we show that the average treatment effect under more general assumptions estimated with CIC (Athey and Imbens 2006) is significant. In the supplementary materials, we also present alternative analyses based on raw (versus log) infection rates and show that the average treatment effect is significant.

2.4.1 Pointwise Treatment Effect with Linear Estimation

The counterfactual model in (Chernozhukov, Wüthrich, and Zhu 2021b) expresses potential outcomes as $Y_{1t}^N=P_{1t}^N+u_t$ and $Y_{1t}^I=P_{1t}^N+{\theta }_t$ where $\left\{P_{1t}^N\right\}$ are mean-unbiased proxies for $\left\{Y_{1t}^N\right\}$ and $\left\{u_t\right\}$ are intervention-invariant stochastic shock process with $E(u_t)=0$. Estimates for the proxies are obtained using one of the linear methods based on hypothesized values of $\left\{{\theta }_t\right\}$ under a sharp null and data from pretreatment and treatment periods. Then, residuals $\left\{\widehat{u_t}\right\}$ are computed, from which the following test statistic is obtained:(3) $$S(\widehat{u})=S_q(\widehat{u})={\left(\frac{1}{\sqrt{T_{*}}}\sum _{t=T_0+1}^T\right|\widehat{u_t}{|}^q)}^{1/q}$$(3)

Chernozhukov, Wüthrich, and Zhu (2021b) use moving block permutations over time to compute $\widehat{p}$ and pointwise confidence intervals for $\left\{{\theta }_t\right\}$ based on their observation that with stable estimators under mild regularity conditions, $\widehat{p}$ has a uniform distribution under the null hypothesis (see Chernozhukov, Wüthrich, and Zhu (2021b) for a complete discussion). For computations and implementation of this method, we used the open source software in Chernozhukov, Wüthrich, and Zhu (2021b).

2.4.2 Average Treatment Effect with Nonlinear Estimation

Athey and Imbens (2006) generalize DID by postulating that $Y_{gt}^N=h(U_g,t)$ and $Y_{gt}^I=h^I(U_g,t)$ for the treated group, where h(u, t) is a monotonically increasing function in u and u represents unobservable characteristics of the group. A key assumption in this model is that while the distribution of U_g can vary across groups, it is time-invariant within groups. In our case, this implies that populations in each county did not change during the time periods in question. Hence, all differences in infection rates of a pair of counties can be attributed to the differences in the distribution of U_g in those counties. Under these assumptions, the cumulative distribution of the counterfactual, denoted $F_{Y_{11}^N}$, is identified in terms of the other observed distributions as follows:(4) $$F_{Y_{11}^N}(y)=F_{Y_{10}}(F_{Y_{00}}^{-1}(F_{Y_{01}}(y))),$$(4) where the subscript gt denotes group $g\in \{0,1\}$ with $1=$ treated and time $t\in \{0,1\}$ with 1 indicating the treatment period. The distribution of the counterfactual is empirically estimated as(5) $${\widehat{F}}_{Y_{gy}}(y)=\frac{1}{N_{gt}}\sum _{i=1}^{N_{gt}}I\{Y_{gt,i}\le y\},$$(5) where $Y_{gt,i}$ denotes the observed outcome of county i belong to group g at time t. The average treatment effect is written as(6) $${\tau }^{\text{CIC}}:=E\left[Y_{11}^I\right]-E[F_{Y_{01}}^{-1}(F_{Y_{00}}(Y_{10}))].$$(6)

This is empirically estimated as(7) $${\widehat{\tau }}^{\text{CIC}}=\frac{1}{N_{11}}\sum _{i=1}^{N_{11}}{Y}_{11,i}-\frac{1}{N_{10}}\sum _{i=1}^{N_{10}}{\widehat{F}}_{Y_{01}}^{-1}({\widehat{F}}_{Y_{00}}(Y_{10,i}))$$(7)

It is then shown that ${\widehat{\tau }}^{\text{CIC}}$ is consistent and that $\sqrt{N}({\widehat{\tau }}^{\text{CIC}}-{\tau }^{\text{CIC}})$ is asymptoticall normally distributed. See Athey and Imbens (2006) for details.

We apply SC-EN from Doudchenko and Imbens (2016) to select control counties and compute their weights. We have directly implemented the main algorithm and accompanying procedures needed for CIC in the R programming language, and provide the full code as a reference in the supplementary materials.

3 Empirical Results

Daily case counts for the counties in Arkansas were accessed on September 5, 2020, at The New York Times Covid-19 data repository (Github 2020). The dates covered pretreatment period preceding the JSAHO start date of April 6, the policy effective dates from April 6 through May 6, and a month of posttreatment period. In our visualizations, we include 31 extra days after the end of the treatment period keeping in mind the incubation period of the virus (Dave et al. 2021; Guan et al. 2020; Lauer et al. 2020) and the delays in policy effectiveness (Abouk and Heydari 2021).

3.1 Selection of Control Counties

Confirming the parallel-trends assumption, Figure 1 displays plots of the 18 counties selected from the donor pool of 74 with relatively flatter trends in the differences of logs of infection rates for the last few days in the pretreatment period. Visual inspection of it confirms that the differences are approximately flat leading up to April 6. For robustness over control selection, we also consider comparison groups comprised of different numbers of counties. In the supplementary materials we include as a reference such plots for all counties. To construct the synthetic county, we first consider only pretreatment outcomes (i.e., the SC methodology discussed in Section 2.2.2). Using the 18 counties from SC-DID as the donor pool results in three counties that comprise the synthetic control while considering a larger pool of 36 with more variable pretreatment trends yield eight counties in the comparison group. Table 1 shows counties and weights for synthetic control computed using best-fitting least squares of pretreatment outcomes of varying numbers of counties.

Fig. 1 Y-axis: $\text{log}\hspace{0.17em}($infections in Saline per $100,000)$—$\text{log}\hspace{0.17em}($infections in Control County per $100,000)$, X-axis: pretreatment period. Dotted line denotes JSAHO announcement date (April 2) and solid line denotes the start date (April 6).

Table 1 Synthetic control counties (SC) and weights using only pretreatment outcomes.

Size^1	Counties and weights
8	H.Spring	Arkansas	Craighead	Desha	Miller	Lafayette	Pike	Polk
	0.23	0.22	0.16	0.13	0.11	0.08	0.07	0.01
7	H.Spring	Arkansas	Craighead	Desha	Miller	Lafayette	Pike
	0.30	0.22	0.15	0.12	0.09	0.08	0.05
6	H.Spring	Arkansas	Craighead	Desha	Miller	Lafayette
	0.32	0.21	0.18	0.13	0.10	0.06
5	H.Spring	Craighead	Miller	Desha	Lonoke
	0.44	0.36	0.14	0.05	0.003
4	H.Spring	Craighead	Miller	Desha
	0.45	0.36	0.14	0.05
3	Craighead	Desha	Lonoke
	0.81	0.18	0.01

NOTE: (1) Size denotes the number of counties comprising the synthetic control unit.

Next, we consider both the pretreatment outcomes and socio-demographic covariates using the ${\text{SC}}_{\mathrm{C}}$ methodology following Abadie and Gardeazabal (2003) and Abadie, Diamond, and Hainmueller (2015) discussed in Section 2.2.2. Unlike in the previous SC approach based on the DID specification, the ${\text{SC}}_{\mathrm{C}}$ here allows the length of the pretreatment period to vary. Figure 2 shows the counties that comprise various synthetic control groups, each defined by the length of the pretreament period in days preceding the treatment period.

Fig. 2 Log infection rate gap between Saline and synthetic counties constructed using the SC-DID (left) and SC (right) methodologies. The labels denote the synthetic control sizes, for example, “SC-8” denotes a synthetic control composed of eight counties shown in Table 1. Dotted line denotes JSAHO announcement date (April 2), solid blue line denotes the start date (April 6), and solid black line denotes the end date of the JSAHO.

From Table 2, we can readily see that county selections and weights are heavily dependent on the size of the pretreatment window, although it is much smaller than the covariates.

Table 2 Synthetic counties (${\text{SC}}_{\mathrm{C}}$) and weights using both pretreatment and covariate data.

	Weights^1				Weights^1
County	${\text{SC}}_{\mathrm{C}5}$	${\text{SC}}_{\mathrm{C}10}$	${\text{SC}}_{\mathrm{C}15}$	County	${\text{SC}}_{\mathrm{C}5}$	${\text{SC}}_{\mathrm{C}10}$	${\text{SC}}_{\mathrm{C}15}$
Benton	0.001	0.001	0.424
Boone			0.512	Grant	0.369		0.003
Carroll	0.091			Greene			0.001
Cleburne	0.001		0.010	Hot Spring		0.302
Crawford		0.139		Lonoke	0.406	0.508
Desha		0.001		Pulaski	0.132	0.047
Garland			0.049	Sevier		0.001

NOTE: Only those counties with positive weights are shown. (1) Numbers in the column headers represent the numbers of pretreatment days preceding the JSAHO starting date (April 6) included in synthetic control matching algorithm.

3.2 Treatment Effect Estimation with SC-DID and SC

With synthetic counties constructed using SC-DID and SC, we examine the estimated impact of JSAHO on log infection rates in Saline. Similar to the Proposition 99 study in Abadie, Diamond, and Hainmueller (2010), our estimate for this treatment effect is the gap between log infection rates in Saline and those in the control unit, as expressed in Equation (1)(1) $$\begin{array}{c}\hspace{0.17em}\text{log}\hspace{0.17em}(\text{infection}\mathrm{}\text{rate}\mathrm{}\text{in}\mathrm{}\text{Saline})\\ \hspace{1em}-\hspace{0.17em}\text{log}\hspace{0.17em}(\text{infection}\mathrm{}\text{rate}\mathrm{}\text{in}\mathrm{}\text{Control}\mathrm{}\text{County})\end{array}$$(1) . As shown in Figure 2 for SC-DID, soon after JSAHO was implemented, the gap between Saline and the mean of the 18 control counties widens drastically. A similar pattern emerges in the SC case, although the effect is more pronounced for the smaller control group sizes described in Table 1.

To assess the robustness of our results, we perform a series of placebo analyses using the methodology discussed in Abadie, Diamond, and Hainmueller (2010). We compare the treatment effect in Saline to placebo treated units by iteratively applying SC-DID and SC to each of the other counties in the donor pool. These placebo studies generate a distribution of estimated treatment effects when no JSAHO was implemented at the county-level.

Figure 3 shows our robustness test results. The gray dots represent the gaps defined in Equation (1)(1) $$\begin{array}{c}\hspace{0.17em}\text{log}\hspace{0.17em}(\text{infection}\mathrm{}\text{rate}\mathrm{}\text{in}\mathrm{}\text{Saline})\\ \hspace{1em}-\hspace{0.17em}\text{log}\hspace{0.17em}(\text{infection}\mathrm{}\text{rate}\mathrm{}\text{in}\mathrm{}\text{Control}\mathrm{}\text{County})\end{array}$$(1) for various placebo treated counties while the black dots represent the gaps for Saline. From the plots for SC-DID and SC₃, one can readily see that the estimated gaps for Saline are wider than those for placebo counties, while it is less apparent in SC₈. It is also apparent that the 18-county control group provides a confirmation of the parallel-trends assumption, while SC methods provide a good fit for Saline’s pretreatment trends. As discussed in Section 2.2, this is due to the fact that SC-DID allows for a permanent additive difference between the treatment and control groups, while SC forces this difference to be zero.

Fig. 3 Log infection rate gaps in Saline and placebo gaps in other counties. Subscripts denote the sizes of control counties in the case of SC. Dotted line denotes JSAHO announcement date (April 2), solid blue line denotes the start date (April 6), and solid black line denotes the end date of the JSAHO.

Next, as discussed in Section 2.2, we estimate the SC counterfactual using both the pretreatment data and covariates (Abadie and Gardeazabal 2003; Abadie, Diamond, and Hainmueller 2010), and study the magnitude of Saline’s treatment effect. The upper-left plot in Figure 4 displays log infection rate gap between Saline and various comparison groups constructed using different windows of pretreatment period. One can observe similar patterns as in synthetic controls fitted with pretreatment data only in Figure 2, reaffirming similar observations made in Abadie, Diamond, and Hainmueller (2010) and explanation in Doudchenko and Imbens (2016). For instance, one can see that immediately after JSAHO was put into effect, the gap widened in Saline relative to controls. Synthetic controls constructed with shorter pretreatment windows seem to exhibit stronger treatment effects. This can be explained by the higher relevance of more recent pretreatment outcomes closer to the policy intervention date. The other three remaining scatterplots in Figure 4 show the log infection rate gaps in Saline and placebo treatment counties for the synthetic counties constructed with each respective pretreatment windows. For SC${}_{C5}$, it is clear that the gaps in Saline are larger than those of placebo units.

Fig. 4 Log infection rate gap between Saline and “${\text{SC}}_{\mathrm{C}}$” Counties. The labels denote the length of pretreatment windows preceding the treatment period, for example, “${\text{SC}}_{\mathrm{C}5}$” denotes a synthetic control fit with 5 pretreatment days and covariates. Dotted line denotes JSAHO announcement date (April 2), solid blue line denotes the start date (April 6), and solid black line denotes the end date of the JSAHO.

3.3 Pointwise Treatment Effect

Figure 5 shows the differences in the log of daily reported infection rates over pretreatment and treatment periods in Saline and the 18 control counties. One can readily see that in all 18 counties, the differences are predominantly negative in the treatment period.

Fig. 5 Y-axis: $\text{log}\hspace{0.17em}($infections in Saline per $100,000)$—$\text{log}\hspace{0.17em}($infections in Control County per $100,000)$, X-axis: pretreatment (first reported date ∼ April 5) and treatment (April 6–May 6) periods, plus additional month (May 7–May 31). Dotted line denotes JSAHO announcement date (April 2), solid blue line denotes the start date (April 6), and solid black line denotes the end date of the JSAHO.

Figure 6 shows pointwise confidence intervals for the JSAHO treatment effect at the 10% significance level based on the test inversion method discussed in Chernozhukov, Wüthrich, and Zhu (2021b). Except for the first few days and SC with 8 control counties (denoted SC-8), the treatment effects under SC-DID and SC all appear to be significant at the 10% level.

Fig. 6 Pointwise confidence intervals at 10% significance level. Synthetic counties constructed using only pretreatment outcomes. Software credits: Chernozhukov, Wüthrich, and Zhu (2021b).

Table 3 summarizes the results of placebo tests of $H_0:{\theta }_{T_0-\tau +1}=\cdots ={\theta }_{T_0}=0$ where $T_0=$ April 5, 2021 and $\tau \in \{1,2,3\}$ days. One can easily confirm that we are not able to reject the null hypothesis at 10% level that treatment effect in the pretreatment period is zero, rendering support to the appropriateness of the underlying assumptions of the conformal model in Chernozhukov, Wüthrich, and Zhu (2021b) for our data setting.

Table 3 Placebo specification tests based on moving block permutations (Chernozhukov, Wüthrich, and Zhu 2021b).

τ	DID	$\mathbf{S}{\mathbf{C}}_3$	$\mathbf{S}{\mathbf{C}}_4$	$\mathbf{S}{\mathbf{C}}_5$	$\mathbf{S}{\mathbf{C}}_6$	$\mathbf{S}{\mathbf{C}}_8$
1	1.00	0.32	0.18	0.18	0.18	0.32
2	0.36	0.09	0.09	0.09	0.09	0.14
3	0.36	0.14	0.14	0.14	0.14	0.09

NOTE: It displays p-values from testing $H_0:{\theta }_{T_0-\tau +1}=\cdots ={\theta }_{T_0}=0,T_0=$ April 5, 2021 and $\tau \in \{1,2,3\}$ days. The subscripts in the column headers denote the control group sizes. Software credits: Chernozhukov, Wüthrich, and Zhu (2021b).

To demonstrate that our results do not depend on any one particular dominant county in the synthetic controls, we perform a leave-one-out robustness test, following the methodology in Chernozhukov, Wüthrich, and Zhu (2021b). Figure 7 displays the p-values obtained from testing $H_0:{\theta }_{T_0+1}=\cdots ={\theta }_{T_1}=0$ where $T_0=$ April 5, 2020, and $T_1=$ May 6, 2020, with each county in the synthetic control iteratively left out. Overall, it provides strong support for the claim that the results are not overly dependent on one specific county in comprising the comparison group. The small sizes of the synthetic controls in ${\text{SC}}_3$ and ${\text{SC}}_{\mathrm{C}5}$ explain the proportionally higher values of p-values.

Fig. 7 p-Values obtained with block permutations with one control county left out at a time. The x-axis labels denote the synthetic counties constructed with the given number of constituent counties, for example, “SC-3” denotes a synthetic control composed of three counties as shown in Table 1.

3.4 Average Treatment Effect

We first applied the 5-fold cross-validation to find the optimal values for the hyper-parameters α and λ in Equation (2)(2) $$\begin{array}{c}(\widehat{\mu },\widehat{\omega })=\underset{\mu ,\omega }{\text{arg}\text{min}}\sum _{i=1}^{T_0}(Y_{1t}-\mu -\sum _{i\ne 1}{\omega }_i·Y_{it})\\ \hspace{1em}+\lambda \left(\frac{1-\alpha }{2}\right|\left|\omega \right|{|}_2^2+\alpha \left|\right|\omega \left|{|}_1\right).\end{array}$$(2) . Solving Model 2 yielded the control group composed of 15 counties, which we denote “SC-EN.” Figure 8 displays the log difference of infection rates of Saline and synthetic county constructed using SC-EN.

Fig. 8 Differences in log infection rates of saline and EN-synthetic county.

With the empirical distribution computed in (5) and the point estimate for the average treatment effect in (7), we obtained that ${\widehat{\tau }}^{\mathit{CIC}}=-13.8$ with the p-value of 0.0454. See accompanying code for the full implementation details.

4 Conclusion

There has been active multi-disciplinary research on Covid-19. However, to date little has been said about the causal impact of SAHO with limited scopes, such as JSAHO in Saline County. This article presents evidence of the effectiveness of county-level JSAHO on reducing the growth rate of infection spreads. The methods used here can be applied to assess situations in other counties or local jurisdictions, and in the process strengthen the external validity of the findings by addressing the issues of limited duration and geographic specificity of the present study. There are other states that had not adopted statewide SIPO or SAHO while some of their local governments went ahead with their own orders at some point in the past, such as Utah, Wyoming, and Oklahoma (Mervosh, Lu, and Swales 2020). In addition, the analyses conducted in this article can be applied to study the effectiveness of other policy measures.

While it is possible that one or more of socio-demographic covariates, including those we did not consider here, for which Saline is an outlier could have been confounders, our results lend unequivocable credence to the association of JSAHO to lower increase in infection rate of Covid-19 in Saline.

The effect of JSAHO should be interpreted broadly: for example, it is not possible to tell whether the reduced rate of increase in cases is due to the direct effect of increased social distancing, or it is mediated by the effect of changes in residents’ behaviors, such as better hand-washing.

Supplementary Materials

The supplementary materials contain two sections: (1) Estimation of treatment effect using raw infection count data, and (2) Supplemental Tables.

Funding

The authors gratefully acknowledge support by NSF DMS 1812148 and PSC-CUNY grant 62781-00 50 grant.

Additional information

Funding

The authors gratefully acknowledge support by NSF DMS 1812148 and PSC-CUNY grant 62781-00 50 grant.

Notes

1 This assumption underlies the difference-in-differences methodology of estimating the counterfactual outcomes.

2 These are used in various forms of the counterfactual estimation methodology known as the synthetic control.