Testing for Equivalence of Pre-Trends in Difference-in-Differences Estimation

Abstract

The plausibility of the “parallel trends assumption” in Difference-in-Differences estimation is usually assessed by a test of the null hypothesis that the difference between the average outcomes of both groups is constant over time before the treatment. However, failure to reject the null hypothesis does not imply the absence of differences in time trends between both groups. We provide equivalence tests that allow researchers to find evidence in favor of the parallel trends assumption and thus increase the credibility of their treatment effect estimates. While we motivate our tests in the standard two-way fixed effects model, we discuss simple extensions to settings in which treatment adoption is staggered over time.

Keywords:

• Bootstrap/resampling
• Difference-in-Differences
• Mathematical statistics
• Regression
• Simulation

1 Introduction

In the classic case, the Difference-in-Differences (DiD) framework consists of two groups observed over two periods of time, where the “treatment group” is untreated in the initial period and has received a treatment in the second period whereas the “control group” is untreated in both periods. The key condition under which the DiD estimator yields sensible point estimates of the average treatment effect on the treated is known as the “parallel paths” or “parallel trends assumption,” henceforth referred to as PTA, which states that in the absence of treatment both groups would have experienced the same temporal trends in the outcome variable on average. If pre-treatment observations are available for both groups, the plausibility of this assumption is typically assessed by plots accompanied by a formal testing procedure showing that there is no evidence of differences in trends over time between the treatment and the control group. However, traditional pre-tests can suffer from low power to detect violations of the PTA (Kahn-Lang and Lang 2020). Thus, finding no evidence of differences in trends in finite samples does not imply that there are no differences in trends in the population. More concerningly, Roth (2022) points out that if differences in trends exist, conditional on not detecting violations of parallel trends at the pre-testing stage, the bias of DiD-estimators may be greatly amplified.

Given the severe consequences of falsely accepting the PTA, we propose that instead of testing the null hypothesis of “no differences in trends” between the treatment and the control group in the pre-treatments periods, one should apply a test for statistical equivalence. We provide three distinct types of equivalence that impose bounds on the maximum, the average and the root mean square change over time in the group mean difference between treatment and control in the pre-treatment periods. Given a threshold below which deviations from the PTA can be considered negligible, these tests allow the researcher to provide statistical evidence in favor of the PTA, thus, increasing its credibility. If no sensible equivalence threshold can be determined before analyzing the data, we propose to report the smallest equivalence threshold for which the null hypothesis of “non-negligible trend differences” can still be rejected at a given level of significance. Conceptually, this idea is similar to the “equivalence confidence interval” in Hartman and Hidalgo (2018) applied to a DiD setting. Our procedure reverses the burden of proof since the data has to provide evidence in favor of similar trends in the treatment and the control group, which is arguably more appropriate for an assumption as crucial to the DiD-framework as the PTA. Furthermore, the power to reject the null hypothesis of “non-negligible differences” is increasing with the sample size (also see Hartman and Hidalgo 2018). This improves upon the current practice of testing the null hypothesis of “no difference,” since large samples increase the chances of rejecting this null hypothesis (and thus seemingly making the DiD framework inapplicable), even if the true difference between treatment and control may be negligible in the given context. Finally, our equivalence test statistics can easily be implemented in practice. While we motivate our tests in the standard two-way fixed effects model with panel data, we discuss how our tests can be applied in situations where treatment timing differs across groups (i.e., “staggered treatment assignment”) or where average treatment effects depend on some observable characteristics.

As we use equivalence tests, our article is closely related to Bilinski and Hatfield (2020), who provide a discussion on the benefits of using equivalence (or “non-inferiority”) tests when testing for violations of modeling assumptions. Their “one-step-up” approach is based on a non-inferiority test of treatment effect estimates obtained from a standard DiD model and from a model augmented with a particular violation of the parallel trends assumption (e.g., a linear trend). While both approaches stress the potential benefits of equivalence testing in DiD setups, a distinctive feature is that we do not necessarily focus on a particular violation of the PTA. As pointed out in Kahn-Lang and Lang (2020), including for instance group-specific linear time trends can lead to a loss in degrees of freedom and thus to a substantial loss in power. In contrast, our approach focuses on testing for any non-negligible differences between treatment and control in the pre-treatment periods. Our article is also related to other approaches that allow for certain deviations from exactly parallel trends. Rambachan and Roth (2023) partially identify the ATT under restrictions that impose that the post-treatment violation of parallel trends is not too large relative to the pre-treatment violation. By contrast, we propose tests for the null hypothesis that the pre-treatment violation of parallel trends is large. If the null is rejected, so that the pre-treatment violation is determined to be small, then the researcher may decide that violations of parallel trends can safely be ignored with minimal bias. Alternatively, the researcher could use the upper bound on the pre-treatment violation given by our approach as a way of determining reasonable bounds on the post-treatment violations of parallel trends, which could then be used as input to the partial identification frameworks of Rambachan and Roth (2023) or Manski and Pepper (2018).

The rest of the article is organized as follows. Section 2 introduces the main two-way fixed effects model and discusses the widely used practice of testing for violations of the PTA. Equivalence tests and our hypotheses are discussed in Section 3. Section 4 introduces our assumptions and presents the test statistics for our hypotheses. In Section 5, we present extensions of the main model that allow for heterogeneous treatment effects due to differences in treatment timing or observable characteristics. Simulation evidence on the performance of our test procedures is provided in Section 6, while Section 7 contains an empirical illustration of our approach. Section 8 concludes. Finally, mathematical details and tables are collected in the Appendix.

2 Pre-Testing in the Canonical DiD Model

We initially contrast our test procedures with the usual pre-test in the canonical DiD case with only two groups and common treatment timing. The researcher observes a balanced panel of n individuals recorded over T + 2 periods of time. We assume that in the first T + 1 periods none of the individuals is treated whereas in period T + 2 a subset has received treatment. Individual i is a member of the treatment group if G_i = 1 and a member of the control group if G_i = 0. Following Kahn-Lang and Lang (2020) we refer to period T + 1 as the “base period” as treatment effects are usually assessed by comparing post-treatment outcomes with the outcomes in the last pre-treatment period. The potential outcomes of unit i in period t when treated and in the absence of treatment are denoted as $Y_{i,t}(1)$ and $Y_{i,t}(0)$, respectively. The object of interest is the average treatment effect on the treated (ATT), given as $${\pi }_{\mathit{\text{ATT}}}:=\mathbb{E}[Y_{i,T+2}(1)-Y_{i,T+2}(0)|G_i=1].$$

For identification of the ATT, we need to make several assumptions. First, we require that $\mathrm{\text{Pr}}(G_i=1)=p\in (0,1)$, which is an “overlap” condition that ensures that both treatment and control are non-empty (Sant’Anna and Zhao 2020). Next, we assume “no-anticipation,” which rules out treatment effects before the actual treatment date. Adapting Borusyak, Jaravel, and Spiess (2023), we assume (2.1) $$\mathbb{E}\left[Y_{i,t}\right|G_i]=\mathbb{E}[Y_{i,t}(0)\left|G_i\right]+{\pi }_{\mathit{\text{ATT}}}{G}_i{D}_{T+2}(t),t=1,\dots ,T+2,$$(2.1) with $D_l(t)=\mathbb{1}\{l=t\}$. This implies that the expected observed outcome coincides with the expected potential outcome corresponding to the actual treatment status both for the control units and the eventually treated units. Next, let (2.2) $$\mathbb{E}[Y_{i,t}(0)|G_i]={\alpha }_i+{\lambda }_t+G_i{\gamma }_t,$$(2.2) where ${\alpha }_i,{\lambda }_t$ and γ_t are some non-stochastic constants. Combining (2.1) with (2.2), we can write (2.3) $$Y_{i,t}={\alpha }_i+{\lambda }_t+G_i{\gamma }_t+{\pi }_{\mathit{\text{ATT}}}{G}_i{D}_{T+2}(t)+u_{i,t},$$(2.3) with $u_{i,t}=Y_{i,t}-\mathbb{E}[Y_{i,t}|G_i]$. It is clear from (2.3) that π_ATT is not identified without further restrictions on γ_t . The fundamental assumption that leads to the DiD estimator is the (augmented) PTA (2.4) $$\begin{array}{c}\mathbb{E}[Y_{i,t}(0)-Y_{i,T+1}(0)|G_i=1]-\mathbb{E}[Y_{i,t}(0)-Y_{i,T+1}(0)|G_i=0]\\ =0,t=1,\dots ,T+2,\end{array}$$(2.4) which implies that ${\gamma }_t-{\gamma }_{T+1}=0$ for all $t=1,\dots ,T+2$. Notice that this assumption over-identifies π_ATT , as identification only requires parallel trends between the post-treatment and the base period, that is ${\gamma }_{T+2}-{\gamma }_{T+1}=0$. However, as it is often difficult to imagine circumstances under which the latter condition is satisfied whereas (2.4) is not, the augmented PTA can be useful for a pre-testing procedure that allows researchers to assess the plausibility of the PTA post-treatment. To do so, one uses the data from periods $1,\dots ,T+1$ to estimate the two-way fixed effects (TWFE) model (2.5) $$Y_{i,t}={\alpha }_i+{\lambda }_t+\sum _{l=1}^T{\beta }_l{G}_i{D}_l(t)+u_{i,t},$$(2.5) where ${\beta }_l={\gamma }_l-{\gamma }_{T+1}$. Here, $\beta =({\beta }_1,\dots ,{\beta }_T)\prime$ collects all “placebo” treatment effects so that under (2.1) and (2.2), β = 0 if and only if the augmented PTA holds. As shown, for instance, in Baltagi (2021) or Wooldridge (2021), β can be estimated by pooled OLS on “double-demeaned” data. Let $W_{i,t,l}=G_i{D}_l(t)$ and $W_{i,t}=(W_{i,t,1},\dots ,W_{i,t,T})\prime$. Double-demeaning (2.5) then yields (2.6) $${\ddot{Y}}_{i,t}={\ddot{W}}_{i,t}^{\prime }\beta +{\ddot{u}}_{i,t},$$(2.6) for $i=1,\dots ,n$ and $t=1,\dots ,T+1$, where ${\ddot{Y}}_{i,t}=Y_{i,t}-{\overline{Y}}_{i,·}-{\overline{Y}}_{·,t}+{\overline{Y}}_{··}$, $$\begin{array}{c}{\overline{Y}}_{i,·}=\frac{1}{T+1}\sum _{t=1}^{T+1}{Y}_{i,t},{\overline{Y}}_{·,t}=\frac{1}{n}\sum _{i=1}^n{Y}_{i,t},\\ {\overline{Y}}_{··}=\frac{1}{n(T+1)}\sum _{i=1}^n\sum _{t=1}^{T+1}{Y}_{i,t}\end{array}$$ and ${\ddot{W}}_{i,t}$ and ${\ddot{u}}_{i,t}$ are defined analogously. Since $\mathbb{E}[u_{i,t}|G_i]=0$ by construction, consistency and asymptotic normality of the pooled OLS estimator in (2.6), denoted as $\widehat{\beta }$, follow under mild conditions. To find evidence against the plausibility of parallel trends, it is therefore common in applied economic research to test for individual significance (see Roth 2022), that is for every $l\in \{1,\dots ,T\}$ we test (2.7) $${\mathrm{H}}_0:{\beta }_l=0\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}\text{versus}\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}{\mathrm{H}}_1:{\beta }_l\ne 0.$$(2.7)

If the null hypothesis is rejected in a pre-treatment period, the PTA is deemed unreasonable, and consequently the DiD framework is often regarded as unsuitable in the corresponding context. This procedure has several shortcomings. For instance, a problematic common practice is to treat failure to reject the null hypothesis in (2.7) as evidence in favor of ${\mathrm{H}}_0$, that is one proceeds as if the null hypothesis was true and as if the PTA held. From a statistical point of view, this practice is incorrect as it neglects the error of type II. In some cases, there may be differences in trends between both groups in the population that cannot be detected with traditional test of (2.7) due to a lack of statistical power. Roth (2022) points out that ignoring these differences can amplify the bias and thus raise concerns of a “publication bias,” since articles using a DiD identification argument are more likely to be deemed publishable when a test of (2.7) could not detect evidence against the PTA. Moreover, the DiD framework is sometimes used even when ${\mathrm{H}}_0$ in (2.7) is rejected, as some statistically significant differences are deemed negligible in a given context. However, a potential threshold $\mathcal{U}>0$ that quantifies what constitutes a negligible difference is often not adequately discussed. On the other hand, if the DiD framework is not applied when ${\mathrm{H}}_0$ in (2.7) is rejected in at least one pre-treatment period, useful information may be lost if $\mathcal{U}$ can be interpreted as a plausible “upper bound” for trend differences. For these reasons, the plausibility of the PTA as the fundamental modeling assumption of the DiD framework can be more convincingly assessed using statistical equivalence tests as these tests address all of the above shortcomings of the current standard testing procedure. For instance, to rewrite (2.7) in terms of statistical equivalence, for some $l\in \{1,\dots ,T\}$ one would define the equivalence threshold $\mathcal{U}>0$ and test (2.8) $${\mathrm{H}}_0:\left|{\beta }_l\right|\ge \mathcal{U}\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}\text{versus}\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}{\mathrm{H}}_1:\left|{\beta }_l\right|<\mathcal{U}$$(2.8) so that rejecting ${\mathrm{H}}_0$ yields evidence in favor of negligible trend differences between periods l and T + 1. While the hypothesis (2.8) can easily be tested with the “two one-sided tests” procedure of Schuirmann (1987), we do not recommend this approach as it would lead to an accumulation of Type I error due to multiple testing. In the following, we elaborate on the benefits of equivalence tests and provide ways of summarizing the statistical evidence in favor of the PTA in the pre-treatment periods by formulating joint hypotheses.

3 Testing for Equivalence

Equivalence testing is well known in biostatistics (see Berger and Hsu 1996 or Wellek 2010). While it has recently been considered in the statistical literature for the analysis of structural breaks (e.g., Dette and Wied 2014; Dette and Wu 2019; Dette, Kokot, and Aue 2020 or Dette, Kokot, and Volgushev 2020), it is less frequently used in econometrics. Instead of assuming that treatment and control are perfectly comparable unless there is strong evidence against this assumption, we suggest several testing procedures that explicitly require finding evidence in favor of the comparability of both groups. Each of the tests is based on an upper bound $\mathcal{U}>0$ for changes in the group mean differences in the pre-treatment periods relative to the base period. There are two ways in which one can make use of the upper bound $\mathcal{U}$. First, as in the “classic” use of equivalence tests, one can specify a threshold $\mathcal{U}$ below which changes in the group mean differences over time are deemed negligible. Rejecting the null hypothesis that trend differences are larger than $\mathcal{U}$ at level of significance α then implies that deviations from parallel trends in the pre-treatment periods are negligible at confidence level $1-\alpha$. The researcher may then interpret this as support for negligible differences in trends post-treatment and hence assume that the PTA holds. This procedure improves upon the current pre-test as it requires an explicit rationalization of the threshold $\mathcal{U}$ and sufficient data to support the assumption of negligible violations of the PTA. The choice of the threshold $\mathcal{U}$ should thus reflect the specific scientific background of the application. In bio-statistics, the popularity of equivalence tests has led to a consensus on sensible choices for $\mathcal{U}$, and regulators frequently specify the equivalence thresholds that should be employed (see Wellek 2010 for a recent review). We expect that with a more frequent adoption of equivalence testing in applied economics a similar consensus will be reached. However, in some applications, it may still be difficult to objectively argue that a certain extent of violations of the PTA can be ignored in practice. It may then be sensible to report ${\mathcal{U}}^{*}$ as the smallest value at which ${\mathrm{H}}_0$ can be rejected at a given level of significance (i.e., for which “equivalence of pre-trends” can be concluded). Small values of ${\mathcal{U}}^{*}$ relative to the estimated treatment effect may then be regarded as reassuring as it is unlikely that the treatment effect is merely an artifact of differences in trends. On the contrary, if ${\mathcal{U}}^{*}$ is relatively large, the credibility of the estimated effect is in serious doubt. A similar idea has been proposed in Hartman and Hidalgo (2018). It can further be related to the “breakdown frontier” proposed by Masten and Poirier (2020), as treatment effects can be considered non-robust to violations of the PTA when ${\mathcal{U}}^{*}$ exceeds the estimated treatment effect. Finally, in cases where the choice of the threshold is difficult, the methodology presented here can also be used to provide (asymptotic) confidence intervals for violations of the PTA.

3.1 Formulating Equivalence Hypotheses

We assume that there exists a vector $\beta \in {\mathbb{R}}^T$ that can be used to assess the plausibility of the augmented PTA. For instance, in (2.5), each β_l corresponds to a “placebo” treatment effect in period $l\in \{1,\dots ,T\}$. To keep the notation tractable, the dimension of β corresponds to the number of available pre-treatment periods T. In practice, it is of course possible that the dimension of β is smaller than T (e.g., when only a subset of all pre-treatment period is used for a pre-test) or exceeds T (e.g., when a conditional PTA is tested; see Section 4). Our tests can then be applied with minor adjustments.

Overall, we consider three distinct hypotheses to test for equivalence of pre-trends in treatment and control. We start with a discussion of the maximum placebo treatment effect. For $\beta \in {\mathbb{R}}^T$, level of significance α and the equivalence threshold $\delta >0$, we test (3.1) $${\mathrm{H}}_0:\left|\right|\beta |{|}_{\infty }\ge \delta \mathrm{\hspace{1em}}\mathrm{\hspace{1em}}\text{versus}\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}{\mathrm{H}}_1:|\left|\beta \right|{|}_{\infty }<\delta ,$$(3.1) where $\left|\right|\beta |{|}_{\infty }:={\mathrm{\text{max}}}_{l\in \{1,\dots ,T\}}|{\beta }_l|$. Since we are now controlling the Type I error, this implies that with confidence level of at least $1-\alpha$, δ is an upper bound for the maximum placebo treatment effect.

In many applications, pre- and post-treatment periods are pooled, for instance to increase statistical power. Similarly, it may be sensible in some applications to consider a pooled or average measure of the pre-treatment deviations from parallel trends. Thus, defining $\overline{\beta }:=\frac{1}{T}{\sum }_{l=1}^T{\beta }_l$, one can find bounds on the average placebo effect by testing (3.2) $${\mathrm{H}}_0:\left|\overline{\beta }\right|\ge \tau \mathrm{\hspace{1em}}\mathrm{\hspace{1em}}\text{versus}\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}{\mathrm{H}}_1:\left|\overline{\beta }\right|<\tau .$$(3.2)

One disadvantage of (3.2) is that there may be cancelation effects in situations where the components of β are large in absolute terms but have opposing signs. Therefore, (3.2) should be used when differences in pre-trends can safely assumed to be of the same sign. As pointed out in Rambachan and Roth (2023), monotone violations of the PTA are frequently discussed in the applied literature. For instance, treatment effect estimates are often considered robust if potential violations of the PTA are of the opposing sign and can thus be ruled out as an explanation for the estimated effects. As an alternative to (3.2) that does not suffer from potential cancelation effects, we further consider the root mean square (RMS) of β, that is ${\beta }_{\mathit{\text{RMS}}}:=\left|\right|\beta \left|\right|/\sqrt{T}=\sqrt{\frac{1}{T}{\sum }_{l=1}^T{\beta }_l^2}$, where $\left|\right|·\left|\right|$ denotes the euclidean norm on ${\mathbb{R}}^T$. The RMS of β can thus be interpreted as the euclidean distance between treatment and control in the pre-treatment periods relative to the distance in the base period scaled by the number of pre-treatment periods. The scaling is induced to ensure that this distance between treatment and control does not increase with the number of pre-treatment periods available. The hypothesis is then formulated as (3.3) $${\mathrm{H}}_0:{\beta }_{\mathit{\text{RMS}}}\ge \zeta \mathrm{\hspace{1em}}\mathrm{\hspace{1em}}\text{versus}\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}{\mathrm{H}}_1:{\beta }_{\mathit{\text{RMS}}}<\zeta ,$$(3.3) which can equivalently be written as (3.4) $${\mathrm{H}}_0:{\beta }_{\mathit{\text{RMS}}}^2\ge {\zeta }^2\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}\text{versus}\mathrm{\hspace{1em}}\mathrm{\hspace{1em}}{\mathrm{H}}_1:{\beta }_{\mathit{\text{RMS}}}^2<{\zeta }^2.$$(3.4)

In Section 4.2 we develop a test statistic for (3.4) and recover ζ as $\sqrt{{\zeta }^2}$.

4 Theory and Implementation

We begin this section by introducing the assumptions underlying our tests. We then proceed by discussing the implementation and the statistical properties of our tests.

4.1 Assumptions

For our statistical theory, we mainly need that a “sequential” version of asymptotic normality holds:

Assumption 4.1.

Fix $\epsilon >0$. For $\lambda \in [\epsilon ,1]$, let $\widehat{\beta }(\lambda )$ an estimator of β based on $\lfloor n\lambda \rfloor$ individuals. Then (4.1) $${\{\sqrt{n}(\widehat{\beta }(\lambda )-\beta )\}}_{\lambda \in [\epsilon ,1]}⇝{\{\frac{1}{\lambda }D\overrightarrow{\mathbb{B}}(\lambda )\}}_{\lambda \in [\epsilon ,1]},$$(4.1) where $\overrightarrow{\mathbb{B}}$ is a T-dimensional vector of independent Brownian motions, D is a T × T-dimensional matrix of full rank, and “$⇝$” denotes weak convergence in the space ${({\mathcal{l}}^{\infty }[\epsilon ,1])}^T$ of all T-dimensional bounded functions on the interval $[\epsilon ,1]$.

Notice that (4.1) comprises “conventional” asymptotic normality of $\widehat{\beta }$ when λ = 1, that is (4.2) $$\sqrt{n}(\widehat{\beta }-\beta )\stackrel{d}{\to }\mathrm{N}(0,\Sigma )\text{as}n\to \infty ,$$(4.2) where Σ denotes a T × T-dimensional covariance matrix. A further assumption required in some of our results is that this matrix can be consistently estimated:

Assumption 4.2.

An estimator $\widehat{\Sigma }$ of Σ is given such that $\widehat{\Sigma }\stackrel{p}{\to }\Sigma \text{as}n\to \infty$.

The proofs of the validity of the first test for the hypothesis (3.1) and of the test for the hypothesis (3.2) do not require Assumption 4.1 but only the weaker condition (4.2) together with Assumption 4.2 (see the first part of Sections 4.2.1 and 4.2.2 for the exact definition of these two tests). A second more powerful test for the hypothesis (3.1) is introduced in the second part of Section 4.2.1, and its validity will be established in the TWFE model (2.5), under conditions that imply (4.2). In contrast, our test for the hypothesis (3.3) requires (4.1), but Assumption 4.2 is not needed, as this test is based on “self-normalization.” We emphasize that without any additional assumptions the asymptotic normality in (4.2) does not imply the process convergence in (4.1). In fact, this a very delicate probabilistic question, which has, to our best knowledge, only been solved for sums of random variables. For example, we refer to Kuelbs (1973) for the independent case and to Samur (1987) for the dependent case (note that these authors consider Banach space valued random variables, which contains the case of finite dimensional vectors).

Sequential asymptotic normality as in Assumption 4.1 can be shown to hold for stationary processes under various forms of dependence such as mixing or physical dependence (see Merlevède, Peligrad, and Utev 2006 and the references therein). Moreover, our tests can be flexibly adapted to different types of data (e.g., panel data or repeated cross-sections) and other features of the design such as clustering or staggered treatment assignment. As an illustration, in Appendix A, we verify that Assumption 4.1 holds under mild conditions in the canonical DiD model of Section 2.

Notice that our approach is a test of the PTA conditional on additional parametric restrictions (for instance, (2.1) and (2.2), or parametric restriction on the effect of observed covariates). While arguably the vast majority of empirical studies are willing to impose similar assumptions, one might want to test the PTA without these restrictions. In this case, as an alternative to our method, one could consider tests based on conditional moment inequalities (see, among others, Andrews and Soares 2010, Romano, Shaikh, and Wolf 2014 or Canay and Shaikh 2017, sec. 4).

4.2 Implementing Equivalence Tests

Based on Assumptions 4.1 and 4.2, we now derive test statistics for our equivalence hypotheses. We further analyze the statistical properties of the resulting test procedures.

4.2.1 Tests for the Hypothesis (3.1)

To describe the first test for the hypothesis (3.1) we initially consider the case T = 1 so that our objective is to test whether a single parameter β₁ exceeds a certain threshold. As ${\widehat{\beta }}_1$ is approximately distributed as $\mathrm{N}({\beta }_1,{\Sigma }_{11}/n)$, the test statistic $\left|{\widehat{\beta }}_1\right|$ approximately follows a folded normal distribution. We therefore propose to reject the null hypothesis in (3.1) whenever (4.3) $$\left|{\widehat{\beta }}_1\right|<{\mathcal{Q}}_{{\mathrm{N}}_F(\delta ,{\widehat{\Sigma }}_{11}/n)}(\alpha ),$$(4.3) where ${\mathcal{Q}}_{{\mathrm{N}}_F(\delta ,{\sigma }^2)}(\alpha )$ denotes the α quantile of the folded normal distribution with mean δ and variance ${\sigma }^2$. It is shown in Appendix B that this test is consistent, has asymptotic level α and is (asymptotically) uniformly most powerful for testing the hypothesis (3.1) in the case T = 1. In particular this test is more powerful than the two-sided t-test (TOST), which could be developed following the arguments in Hartman and Hidalgo (2018). For T > 1, we apply the idea of intersection-union (IU) tests outlined in Berger and Hsu (1996) and reject the null hypothesis in (3.1), whenever (4.4) $$\left|{\widehat{\beta }}_t\right|<{\mathcal{Q}}_{{\mathrm{N}}_F(\delta ,{\widehat{\Sigma }}_{\mathit{\text{tt}}}/n)}(\alpha )\forall t\in \{1,\dots ,T\}.$$(4.4)

While this test is computationally attractive, a well-known disadvantage of testing procedures based on the IU principle is that they tend to be rather conservative (see Berger and Hsu 1996, among others), which is confirmed by our simulation study (see Table C.1).

Specifically for the TWFE model (2.5), it is possible to obtain a more powerful test for the hypothesis (3.1) as follows: In the first step, estimate β in (2.5) to obtain the unconstrained TWFE estimator ${\widehat{\beta }}_u$. In the second step, re-estimate (2.5) by minimizing the sum of squared residuals under the constraint $\left|\right|\beta |{|}_{\infty }={\mathrm{\text{max}}}_{l=1,\dots ,T}|{\beta }_l|=\delta$ to obtain a constrained estimator, say ${\widehat{\beta }}_c$. We then define new estimators of the parameters as (4.5) $${\widehat{\widehat{\beta }}}_c=\{\begin{array}{ccc}{\widehat{\beta }}_u& \text{if}& \left|\right|\widehat{\beta }|{|}_{\infty }\ge \delta \\ {\widehat{\beta }}_c& \text{if}& \left|\right|\widehat{\beta }|{|}_{\infty }<\delta \end{array}\hspace{1em}$$(4.5) and (4.6) $${\widehat{\widehat{\sigma }}}_c=\frac{1}{(n-1)T}\sum _{i=1}^n\sum _{t=1}^{T+1}{({\ddot{Y}}_{i,t}-{\ddot{W}}_{i,t}^{\prime }{\widehat{\widehat{\beta }}}_c)}^2.$$(4.6)

Note that the vector ${\widehat{\widehat{\beta }}}_c$ satisfies the null hypothesis in (3.1). In the third step, for $b=1,\dots ,B\in \mathbb{N}$, we generate bootstrap samples with ${\ddot{u}}_{i,t}^{(b)}\stackrel{\text{iid}}{\sim }\mathrm{N}(0,{\widehat{\widehat{\sigma }}}_c)$ and ${\ddot{Y}}_{i,t}^{(b)}={\ddot{W}}_{i,t}^{\prime }{\widehat{\widehat{\beta }}}_c+{\ddot{u}}_{i,t}^{(b)}$ for $i=1,\dots ,n$ and $t=1,\dots ,T+1$. For each bootstrap sample, estimate ${\widehat{\beta }}^{(b)}$ and compute ${\mathcal{Q}}_{\alpha }^{*}$ as the empirical α-quantile of the bootstrap sample $\left\{{\mathrm{\text{max}}}_{l=1,\dots ,T}\right|{\widehat{\beta }}_l^{(b)}|:b=1,\dots ,B\}$. Finally, reject the null hypothesis ${\mathrm{H}}_0$ in (3.1) if (4.7) $$\left|\right|\widehat{\beta }|{|}_{\infty }<{\mathcal{Q}}_{\alpha }^{*}.$$(4.7)

Notice that the condition $\left|\right|\beta |{|}_{\infty }=\delta$ in the calculation of the constrained estimator ${\widehat{\beta }}_c$ is made for technical reasons in the proof of this statement. Numerical results show that the test has a very similar behavior if ${\widehat{\beta }}_c$ is calculated under the condition that $\left|\right|\beta |{|}_{\infty }\ge \delta$. The following result shows that this test is consistent and has asymptotic level α.

Theorem 4.1.

If the errors $u_i=(u_{i,1},\dots ,u_{i,T+1})\prime$ in model (2.5) satisfy $\mathbb{E}[u_i{u}_i^{\prime }|G_i]={\sigma }_u^2{\mathrm{I}}_{(T+1)\times (T+1)}$, then the test defined by (4.7) is consistent and has asymptotic level α for the hypothesis (3.1). More precisely,

1. if the null hypothesis in (3.1) is satisfied, then we have for any $\alpha \in (0,0.5)$

   $\underset{n\to \infty }{\mathrm{\text{limsup}}}{\mathbb{P}}_{\beta }(|\left|\widehat{\beta }\right|{|}_{\infty }<{\mathcal{Q}}_{\alpha }^{*})\le \alpha .$ (4.8)

2. if the null hypothesis in (3.1) is satisfied and the set

   $\mathcal{E}=\{\mathcal{l}=1,\dots ,T:|{\beta }_{\mathcal{l}}|=|\left|\beta \right|{|}_{\infty }\}$ (4.9)

   consists of one point, then we have for any $\alpha \in (0,0.5)$

   $\underset{n\to \infty }{\mathrm{\text{lim}}}{\mathbb{P}}_{\beta }(|\left|\widehat{\beta }\right|{|}_{\infty }<{\mathcal{Q}}_{\alpha }^{*})=\{\begin{array}{ccc}0& \text{if}& \left|\right|\beta |{|}_{\infty }>\delta \\ \alpha & \text{if}& \left|\right|\beta |{|}_{\infty }=\delta .\end{array}$

3. if the alternative in (3.1) is satisfied, then we have for any $\alpha \in (0,0.5)$

   $\underset{n\to \infty }{\mathrm{\text{lim}}}{\mathbb{P}}_{\beta }(|\left|\widehat{\beta }\right|{|}_{\infty }<{\mathcal{Q}}_{\alpha }^{*})=1.$

Remark 4.1.

1. It is well known that the bootstrap fails when the target parameter is non-differentiable, and so alternative approaches are needed (see, for example, Fang and Santos 2019; Hong and Li 2020). To explain why these observations are consistent with the statements in Theorem 4.1, we briefly explain the main arguments for its proof here. To be precise, note that Theorem 4.1 distinguishes several cases under the null hypothesis. First we consider the case (2), where the set $\mathcal{E}=\{\mathcal{l}=1,\dots ,T:|{\beta }_{\mathcal{l}}|=|\left|\beta \right|{|}_{\infty }\}$ consists of one point. In this case, as $\left|\right|\beta |{|}_{\infty }\ge \delta >0$, the mapping $\beta \to \left|\right|\beta |{|}_{\infty }$ is in fact Hadamard-differentiable, and one could alternatively use Theorem 3.1 in Fang and Santos (2019) to prove that

   $\underset{n\to \infty }{\mathrm{\text{lim}}}{\mathbb{P}}_{\beta }(|\left|\widehat{\beta }\right|{|}_{\infty }<{\mathcal{Q}}_{\alpha }^{*})=\alpha ,$ (4.10)

   if $\left|\right|\beta |{|}_{\infty }=\delta >0$, and that the limit vanishes if $\left|\right|\beta |{|}_{\infty }>\delta$. Next, we consider the case (1) of Theorem 4.1 and the situation where the set $\mathcal{E}$ in (4.9) contains more than 1 element. In this case the mapping $\beta \to \left|\right|\beta |{|}_{\infty }$ is not Hadamard-differentiable at this point. Consequently, by Theorem 3.1 in Fang and Santos (2019) a statement of the from (4.10) cannot hold in general. However, in (4.8) we neither claim the existence of the limit nor equality. In fact we can show by similar arguments as given for the derivation of eq. (A.32) in Dette et al. (2018) that

   $\sqrt{n}(|\left|{\widehat{\beta }}^{*}\right|{|}_{\infty }-\left|\right|\widehat{\beta }|{|}_{\infty })\le {\tilde{Z}}_n^{*}+o_{\mathbb{P}}(1),$

   where the statistic ${\tilde{Z}}_n^{*}$ has, conditionally on the data, the same asymptotic distribution as the original statistic $\sqrt{n}(|\left|\widehat{\beta }\right|{|}_{\infty }-\left|\right|\beta |{|}_{\infty })$. It turns out that this argument is sufficient to prove (4.8), and the same argument can also be used to show assertion (3) of Theorem 4.1.

2. We emphasize that Theorem 4.1 provides a pointwise result. We expect that uniform results can be obtained as well. However, as indicated in part (a), even proving pointwise results requires some nonstandard techniques. This is even more true if one would aim for uniform results, and we leave this problem for future research.

3. The assumption of spherical model errors, that is $\mathbb{E}[u_i{u}_i^{\prime }|G_i]={\sigma }_u^2{\mathrm{I}}_{(T+1)\times (T+1)}$, is crucial for the validity of Theorem 4.1. In fact, our simulation results in Section 6 suggest that the test may not keep its nominal level under high levels of serial dependence in the error terms. As a further alternative, we therefore suggest replacing the bootstrap procedure described before (4.7) with the wild cluster bootstrap (Cameron and Miller 2015). To implement this, first compute the residuals ${\widehat{\ddot{u}}}_{i,c}={\ddot{Y}}_i-{\ddot{W}}_i^{\prime }{\widehat{\widehat{\beta }}}_c$, where ${\ddot{Y}}_i=({\ddot{Y}}_{i,1},\dots ,{\ddot{Y}}_{i,T+1})\prime$ and ${\ddot{W}}_i=({\ddot{W}}_{i,1},\dots ,{\ddot{W}}_{i,T+1})$. Next, let R_i denote a Rademacher random variable that takes the values–1 and 1 with probability 0.5 each. For $b\in \{1,\dots ,B\}$, generate bootstrap samples ${\ddot{Y}}_i^{(b)}={\ddot{W}}_i^{\prime }{\widehat{\widehat{\beta }}}_c+{\widehat{\ddot{u}}}_i^{*}$, where ${\widehat{\ddot{u}}}_i^{*}=R_i\times {\widehat{\ddot{u}}}_{i,c}$. As before, compute ${\widehat{\beta }}^{(b)}$ in each bootstrap sample, compute ${\mathcal{Q}}_{\alpha }^{*}$ and reject ${\mathrm{H}}_0$ in (3.1) if (4.7) holds. We demonstrate by means of a simulation study in Section 6 that this bootstrap test yields reasonable results for nonspherical errors, and we conjecture that the wild bootstrap provides a valid solution in general. A formal proof of this statement is beyond the scope of the present article.

Remark 4.2.

Theorem 4.1 shows the consistency of a specific estimator in model (2.5). As pointed out by a referee, it is of interest to investigate if a similar approach is applicable for other estimators and models. We conjecture that this is possible, but proving the consistency of such an approach would be challenging, as the result would depend sensitively on the given model and the estimator used.

To be precise, let $\widehat{\beta }$ denote an estimator of a parameter β, such that $\sqrt{n}(\widehat{\beta }-\beta )\stackrel{d}{\to }\mathrm{N}(0,\Sigma )\text{as}n\to \infty$, where Σ denotes the covariance matrix, and define the constrained estimate ${\widehat{\widehat{\beta }}}_c$ via (4.5). Further let $\widehat{\Sigma }$ denote a consistent estimate of the matrix Σ. We propose to generate ${\widehat{\beta }}^{(b)}$ from a $\mathrm{N}({\widehat{\widehat{\beta }}}_c,\widehat{\Sigma }/n)$ distribution and compute ${\mathcal{Q}}_{\alpha }^{**}$ as the empirical α-quantile of the bootstrap sample $\left\{{\mathrm{\text{max}}}_{l=1,\dots ,T}\right|{\widehat{\beta }}_l^{(b)}|:b=1,\dots ,B\}$. The null hypothesis ${\mathrm{H}}_0$ in (3.1) is rejected, if $\left|\right|\widehat{\beta }|{|}_{\infty }<{\mathcal{Q}}_{\alpha }^{**}$. Note that implementation of this procedure neither requires a specific model nor a specific estimator. In particular, we have implemented this alternative approach in model (2.5) with $\widehat{\Sigma }={\widehat{\widehat{\sigma }}}_c^2{(\ddot{W}\prime \ddot{W})}^{-1}$ for spherical errors and with $\widehat{\Sigma }=\frac{n}{n-T}{(\ddot{W}\prime \ddot{W})}^{-1}({\sum }_{i=1}^n{\ddot{W}}_i{\widehat{\ddot{u}}}_{i,c}{\widehat{\ddot{u}}}_{i,c}^{\prime }{\ddot{W}}_i^{\prime }){(\ddot{W}\prime \ddot{W})}^{-1}$ for errors that are clustered on the individual level, where $\ddot{W}=({\ddot{W}}_1,\dots ,{\ddot{W}}_n)\prime$ and ${\widehat{\widehat{\sigma }}}_c^2$ and ${\widehat{\ddot{u}}}_{i,c}$ are defined in (4.6) and Remark 4.1, respectively. In a simulation study, we obtained very similar results as for the bootstrap tests considered in Theorem 4.1 for the spherical case and Remark 4.1 (c) for the clustered case (these results are not displayed in Section 6 for the sake of brevity).

4.2.2 A Test for the Hypothesis (3.2)

For some fixed $\tau >0$, a test can be constructed by first computing the statistic $$\overline{\widehat{\beta }}:=\frac{1}{T}\sum _{t=1}^T{\widehat{\beta }}_t=1\prime \widehat{\beta }/T,$$

where $1=(1,\dots ,1)\prime \in {\mathbb{R}}^T$. Note that it follows from Assumptions 4.1 and 4.2 that $$\sqrt{n}1\prime (\widehat{\beta }-\beta )\to \mathrm{N}(0,1\prime \Sigma 1)).$$

Consequently, based on the discussion in the first part of Section 4.2.1, we propose to reject the null hypothesis in (3.2), whenever (4.11) $$\left|\overline{\widehat{\beta }}\right|<{\mathcal{Q}}_{{\mathrm{N}}_F(\tau ,{\widehat{\sigma }}^2)}(\alpha ),$$(4.11) where ${\widehat{\sigma }}^2=1\prime \widehat{\Sigma }1$/n, and $\widehat{\Sigma }$ is a consistent estimator of Σ which is known by Assumption 4.2.

4.2.3 A Pivotal Test for the Hypothesis (3.4)

For $\lambda \in [\epsilon ,1]$, let ${\widehat{\beta }}_{\mathit{\text{RMS}}}(\lambda )$ denote the RMS of $\widehat{\beta }$ based on a random subsample of size $\lfloor \lambda n\rfloor$ of the original sample of n individuals. In order to construct a pivot test for the hypothesis (3.4), define (4.12) $${\widehat{M}}_n:=\frac{{\widehat{\beta }}_{\mathit{\text{RMS}}}^2(1)-{\beta }_{\mathit{\text{RMS}}}^2}{{\widehat{V}}_n},$$(4.12) where (4.13) $${\widehat{V}}_n=({\int }_{\epsilon }^1({\widehat{\beta }}_{\mathit{\text{RMS}}}^2(\lambda )-{\widehat{\beta }}_{\mathit{\text{RMS}}}^2(1){)}^2\nu (d \lambda ){)}^{1/2}$$(4.13) and ν denotes a measure on the interval $[\epsilon ,1]$. The following result is proved in the Appendix.

Theorem 4.2.

If Assumption 4.1 is satisfied and $\beta \cancel{=}0$, then the statistic ${\widehat{M}}_n$ defined in (4.12) converges weakly with a nondegenerate limit distribution, that is (4.14) $${\widehat{M}}_n\stackrel{d}{\to }\mathbb{W}:=\frac{\mathbb{B}(1)}{({\int }_{\epsilon }^1(\mathbb{B}(\lambda )/\lambda -\mathbb{B}(1){)}^2\nu (d\lambda ){)}^{1/2}},$$(4.14)

where ${\{\mathbb{B}(\lambda )\}}_{\lambda \in [\epsilon ,1]}$ is a Brownian motion on the interval $[\epsilon ,1]$. Moreover, if β = 0, then (4.15) $${\widehat{M}}_n\stackrel{d}{\to }\frac{{\mathbb{Z}}^2(1)}{({\int }_{\epsilon }^1({\mathbb{Z}}^2(\lambda )-{\mathbb{Z}}^2(1){)}^2\nu (d\lambda ){)}^{1/2}},$$(4.15) where ${\mathbb{Z}}^2(\lambda )=\frac{1}{{\lambda }^2}\overrightarrow{\mathbb{B}}\prime (\lambda )D\prime D\overrightarrow{\mathbb{B}}(\lambda ),\overrightarrow{\mathbb{B}}$ is a T-dimensional vector of independent Brownian motions and D is the matrix in (4.1).

It follows from the proof of Theorem 4.2 that the statistic ${\widehat{\beta }}_{\mathit{\text{RMS}}}^2$ is a consistent estimator of ${\beta }_{\mathit{\text{RMS}}}^2$. Therefore, we propose to reject the null hypothesis ${\mathrm{H}}_0$ in (3.4) (and consequently ${\mathrm{H}}_0$ in (3.3)), whenever (4.16) $${\widehat{\beta }}_{\mathit{\text{RMS}}}^2<{\zeta }^2+{\mathcal{Q}}_{\mathbb{W}}(\alpha ){\widehat{V}}_n,$$(4.16) where ${\mathcal{Q}}_{\mathbb{W}}(\alpha )$ is the α-quantile of the limiting distribution of the random variable $\mathbb{W}$ on the right-hand side of (4.14). Note that these quantiles can be easily obtained by simulation because the distribution of $\mathbb{W}$ is completely known. For instance, ${\mathcal{Q}}_{\mathbb{W}}(0.05)\approx -2.1$. The following result shows that this decision rule defines a valid test for the hypothesis (3.4).

Theorem 4.3.

If Assumption 4.1 is satisfied, then the test defined by (4.16) is a consistent asymptotic level α-test for the hypothesis (3.4), that is $$\underset{n\to \infty }{\mathrm{\text{lim}}}{\mathbb{P}}_{\beta }({\widehat{\beta }}_{\mathit{\text{RMS}}}^2<{\zeta }^2+{\mathcal{Q}}_{\mathbb{W}}(\alpha ){\widehat{V}}_n)=\{\begin{array}{ccc}0,& \text{if}& {\beta }_{\mathit{\text{RMS}}}^2>{\zeta }^2\\ \alpha ,& \text{if}& {\beta }_{\mathit{\text{RMS}}}^2={\zeta }^2\\ 1,& \text{if}& {\beta }_{\mathit{\text{RMS}}}^2<{\zeta }^2\end{array}$$

Remark 4.3.

1. Notice that in practice one chooses ν as a discrete distribution which makes the evaluation of the integrals in (4.13) and in the denominator of the random variable $\mathbb{W}$ very easy. For example, if ν denotes the uniform distribution on $\{\frac{1}{5},\frac{2}{5},\frac{3}{5},\frac{4}{5}\}$, then the statistics ${\widehat{V}}_n$ in (4.13) simplifies to

   ${\left(\frac{1}{4}{\sum }_{k=1}^4{({\widehat{\beta }}_{\mathit{\text{RMS}}}^2(\frac{k}{5})-{\widehat{\beta }}_{\mathit{\text{RMS}}}^2(1))}^2\right)}^{1/2}.$

   This measure is also used in the simulation study in Section 6, where we analyze the finite sample properties of the different procedures. In practice, it is thus not necessary to explicitly choose ε.

2. It follows from the proof of Theorem 4.2 that an asymptotic $(1-\alpha )$-confidence interval for the parameter ${\beta }_{\mathit{\text{RMS}}}^2>0$ is given by

   $[{\widehat{\beta }}_{\mathit{\text{RMS}}}^2+{\mathcal{Q}}_{\mathbb{W}}(\alpha /2){\widehat{V}}_n,{\widehat{\beta }}_{\mathit{\text{RMS}}}^2+{\mathcal{Q}}_{\mathbb{W}}(1-\alpha /2){\widehat{V}}_n].$

3. Theorem 4.3 can be extended to get uniform results. More precisely, define for a small positive constant c the sets

   $\begin{array}{c}\mathcal{H}=\left\{\beta \right|\Delta (\beta )>c,{\widehat{\beta }}_{\mathit{\text{RMS}}}^2\ge {\zeta }^2\},\\ \mathcal{A}(x)=\left\{\beta \right|\Delta (\beta )>c,{\widehat{\beta }}_{\mathit{\text{RMS}}}^2<{\zeta }^2-x/\sqrt{n}\},\end{array}$

   corresponding to the null hypothesis and the alternative, respectively, where $\Delta (\beta )$ is defined in (B.6) in the Appendix. Then $$\underset{n\to \infty }{\mathrm{\text{limsup}}}\underset{\beta \in \mathcal{H}}{\mathrm{\text{sup}}}{\mathbb{P}}_{\beta }({\widehat{\beta }}_{\mathit{\text{RMS}}}^2<{\zeta }^2+{\mathcal{Q}}_{\mathbb{W}}(\alpha ){\widehat{V}}_n)=\alpha .$$

Furthermore there exists a nondecreasing function $f:{\mathbb{R}}_{>0}\to {\mathbb{R}}_{>0}$, with $f(x)>\alpha$ for all x > 0 and $\underset{x\to \infty }{\mathrm{\text{lim}}}f(x)=1$, such that $$\underset{n\to \infty }{\mathrm{\text{liminf}}}\underset{\beta \in \mathcal{A}(x)}{\mathrm{\text{inf}}}{\mathbb{P}}_{\beta }({\widehat{\beta }}_{\mathit{\text{RMS}}}^2<{\zeta }^2+{\mathcal{Q}}_{\mathbb{W}}(\alpha ){\widehat{V}}_n)=f(x).$$

The details are omitted for the sake of brevity.

5 Equivalence Testing with Heterogeneous Treatment Effects and Staggered Adoption

The use of the simple TWFE model has recently experienced substantial criticism in the presence of multiple groups, heterogeneous treatment effects and differences in treatment timing. In this situation, the TWFE estimator often does not correspond to a reasonable estimate of the ATT, and alternative estimators have been proposed by several authors (see, for instance, Goodman-Bacon 2021; Callaway and Sant’Anna 2021; Sun and Abraham 2021; Borusyak, Jaravel, and Spiess 2023 or de Chaisemartin and D’Haultfœuille 2020. Excellent reviews of this fast-growing literature are provided by Roth et al. 2023 and de Chaisemartin and D’Haultfœuille 2022.). Specifically, Wooldridge (2021) shows that the deficiency of the TWFE estimator can be regarded as a model misspecification problem. He then proposes model adjustments that allow for treatment effect heterogeneity due to differences in treatment timing and observed characteristics (which are assumed to be unaffected by the treatment). While we conjecture that our equivalence tests can be used with most (if not all) of the mentioned estimators, we focus on the regression-based approach of Wooldridge (2021) in the following, as it is straightforward to show that our assumptions hold with minor adjustments to the arguments in Appendix A. We now consider the case of the staggered adoption (i.e., the initial treatment period varies across groups) of an absorbing treatment (i.e., the treatment status does not change after the initial treatment) in the presence of a never-treated group. Following Wooldridge (2021, Sec. 6), we assume that the time since the initial treatment adoption produces different levels of exposure to the treatment, resulting in treatment effect heterogeneity across time. As in Section 2, we consider a balanced panel of n individuals that are observed in T + 1 pre-treatment periods. In periods $T+2,\dots ,\overline{T}$, a subset of individuals adopts treatment, leading to “treatment cohorts.” As before, period T + 1 is used as the base period. To define a treatment cohort dummy, let $G_i^r=1$ if individual i has first adopted treatment in period $r\in \mathcal{R}:=\{T+2,\dots ,\overline{T},\infty \}$ and zero otherwise, where $G_i^{\infty }$ is a dummy indicating that individual i is a member of the never treated group. The potential outcome of unit i in treatment cohort $r\in \mathcal{R}$ observed in time period $t\in \{1,\dots ,\overline{T}\}$ is denoted by $Y_{i,t}(r)$, where the “baseline” potential outcome in period t if unit i is untreated is given by $Y_{i,t}(\infty )$. The objects of interest are the cohort and time-specific treatment effects that may depend on a vector of observed and time-invariant covariates X_i , that is (5.1) $${\pi }_{r,t}(X_i):=\mathbb{E}[Y_{i,t}(r)-Y_{i,t}(\infty )|G_i^r=1,X_i].$$(5.1)

In many applications, it is assumed that the treatment effect is a linear function of the observed covariates (which may contain polynomials of X_i ), so that $${\pi }_{r,t}(X_i)={\pi }_{r,t}+{\rho }_{r,t}^{\prime }\stackrel{.}{X_i},$$ where ${\dot{X}}_i=X_i-\mathbb{E}[X_i|G_i^r=1]$ is the covariate centered around the cohort mean so that ${\pi }_{r,t}$ is the treatment effect averaged across the distribution of X_i conditional on $G_i^r=1$. Adapting (2.1), we assume $$\mathbb{E}\left[Y_{i,t}\right|{\mathbb{G}}_i,X_i]=\mathbb{E}[Y_{i,t}(\infty )|{\mathbb{G}}_i,X_i]+\sum _{r=T+2}^{\overline{T}}\sum _{s=r}^{\overline{T}}{\pi }_{r,s}(X_i)G_i^r{D}_s(t),$$ where ${\mathbb{G}}_i=(G_i^{T+2},\dots ,G_i^{\overline{T}})$, so that deviation from the designated treatment path (e.g., through anticipation) are ruled out. Further extending (2.2), $$\begin{array}{c}\mathbb{E}[Y_{i,t}(\infty )|{\mathbb{G}}_i,X_i]={\alpha }_i+{\lambda }_t+\kappa \prime X_i+\sum _{r=T+2}^{\overline{T}}{\zeta }_r\prime X_i{G}_i^r\\ +\sum _{\stackrel{s=1}{s\ne T+1}}^{\overline{T}}{\iota }_s\prime X_i{D}_s(t)+\sum _{\stackrel{r=1}{r\ne T+1}}^{\overline{T}}\sum _{\stackrel{s=1}{s\ne T+1}}^{\overline{T}}{\gamma }_{r,s}{G}_i^r{D}_s(t),\end{array}$$ where ${\gamma }_{r,s}$ is a non-stochastic constant that differs across cohorts and time. Notice that the covariates are again assumed to enter the baseline outcome linearly. As in Section 2, the cohort and time-specific ATTs cannot be identified without further restrictions. In order to ensure that ${\gamma }_{r,s}=0$, we adapt (2.4) to impose a conditional staggered parallel trends assumption (CSPTA), using the never-treated group as the control: $$\begin{array}{c}\mathbb{E}[Y_{i,t}(\infty )-Y_{i,T+1}(\infty )|G_i^r=1,X_i]\\ \hspace{1em}=\mathbb{E}[Y_{i,t}(\infty )-Y_{i,T+1}(\infty )|G_i^{\infty }=1,X_i],t,r=1,\dots ,\overline{T}.\end{array}$$

Combining the above assumptions, we obtain (5.2) $$\begin{array}{c}\mathbb{E}\left[Y_{i,t}\right|{\mathbb{G}}_i,X_i]={\alpha }_i+{\lambda }_t+\kappa \prime X_i\sum _{r=T+2}^{\overline{T}}{\zeta }_r\prime X_i{G}_i^r+\sum _{\stackrel{s=1}{s\ne T+1}}^{\overline{T}}{\iota }_s\prime X_i{D}_s(t)\\ \hspace{1em}+\sum _{r=T+2}^{\overline{T}}\sum _{s=r}^{\overline{T}}({\pi }_{r,s}{G}_i^r{D}_s(t)+{\rho }_{r,s}^{\prime }{\dot{X}}_i{G}_i^r{D}_s(t))\\ \hspace{1em}+\sum _{m=T+2}^{\overline{T}}\sum _{\stackrel{k=1}{k\ne T+1}}^{m-1}({\tilde{\pi }}_{m,k}{G}_i^m{D}_k(t)+{\tilde{\rho }}_{m,k}^{\prime }{\dot{X}}_i{G}_i^m{D}_k(t)).\end{array}$$(5.2)

Simple algebra sows that the placebo conditional cohort and time specific treatment effects ${\tilde{\pi }}_{m,k}+{\tilde{\rho }}_{m,k}^{\prime }{\dot{X}}_i$ are identified by taking differences-in-differences, that is by comparing the evolution in the average outcome between periods k and the base T + 1 between treatment cohort m and the never treated group. Given (5.1)–(2.2), the CSPTA implies that the placebo treatments are zero. We therefore avoid any “contamination” by treatment effects at time $m\prime >m$, which, as noted by Sun and Abraham (2021), can lead to a rejection of the CSPTA in the pre-treatment periods even in cases where it actually holds. Since the assumptions in Section 4.1 can easily be shown to hold under mild conditions by adapting the arguments in Section A, we can directly apply our equivalence tests to the placebo treatment effects. Moreover, the model in (5.2) can be flexibly adjusted to the problem at hand. For instance, one may be willing to exclude a subset of the placebo treatment effects from the model in order to allow for some pooling across cohorts and time. As noted by Wooldridge (2021), in this case, the pooled OLS estimator of ${\pi }_{r,s}$ is an averaged “rolling DiD” where, on top of the never-treated group and the base period, any cohort and period that corresponds to an omitted placebo treatment effect is used as a control. In this case, the CSPTA needs to be adjusted accordingly (e.g., as in Roth et al. 2023), as parallel trends need to be plausible between multiple groups and periods. Finally, notice that in practice $\mathbb{E}[X_i|G_i^r=1]$ needs to be replaced by the sample average of X_i in cohort r. As suggested in Wooldridge (2021), one should adjust the standard errors to account for the additional sampling variation.

As a further note of caution, practitioners should be aware that our tests only consider an implication of the CSPTA, as we are relying on assumptions that impose a particular form of treatment effect heterogeneity (e.g., linear dependence on observed covariates). In fact, we are thus testing a joint test of a parametric restriction on treatment effect heterogeneity and conditional parallel trends. While parametric restrictions are popular in the applied literature, one might still be concerned about their validity when testing for parallel trends. As an alternative, one may then consider tests based on conditional moment inequalities (see, for instance, Andrews and Shi 2013).

6 Simulations

In order to investigate the small sample properties of our tests, we conduct a simulation study in R. For that, we create a panel dataset with $T\in \{1,4,8,12\}$ and $n\in \{100,1000\}$. For $i=1,\dots ,n$ and $t=1,\dots ,T+2$, we generate the data from (6.1) $$Y_{i,t}={\alpha }_i+{\lambda }_t+\sum _{l=1}^T{\beta }_l{G}_i{D}_l(t)+{\pi }_{\mathit{\text{ATT}}}{G}_i{D}_{T+2}(t)+u_{i,t},$$(6.1) with α_i and λ_t standard normal, $\mathrm{\text{Pr}}(G_i=1)=\frac{1}{2}$ and ${\pi }_{\mathit{\text{ATT}}}=0$.

In an initial step, we investigate the level of the proposed tests. To do so, we set the level of significance to $\alpha =5\%$ and the equivalence threshold for all hypotheses to 1. We then choose the parameters β_l in the pre-treatment periods such that we are on the “boundary” of the hypotheses, that is ${\beta }_1=1$ and ${\beta }_l=0$ for l > 1 or ${\beta }_l=1$ for all $l=1,\dots ,T$. Moreover, we also investigate the power of the test procedures by choosing ${\beta }_l\in \{0.8,0.9\}$ for all $l=1,\dots ,T$. The bootstrap based tests for (3.1) are computed using 1000 bootstrap draws. Finally, the generated error terms are both serially correlated and heteroscedastic, as they are drawn from a stationary AR(3) process with autoregressive parameters $({\varphi }_1,{\varphi }_2,{\varphi }_3)=(0.5,0.3,0.1)$ and with standard deviation $1+G_i$. Consequently, the tests that require an estimate of Σ are based on standard errors that are clustered on the individual level. The results for all tests based on $20,000$ simulations are presented in Tables C.1–C.3.

Table 1 Smallest equivalence thresholds such that the null hypotheses in (3.1), (3.2), and (3.3) can be rejected for varying numbers of pre-treatment periods.

Estimates periods	June	May&June	April–June
${\delta }_{\mathit{\text{IU}}}^{*}$	0.147	0.147	0.147
${\delta }_{\mathit{\text{Boot}}}^{*}$	0.156	0.156	0.15
${\delta }_{c.\mathit{\text{Boot}}}^{*}$	0.152	0.141	0.137
${\tau }^{*}$	0.147	0.086	0.093
${\zeta }^{*}$	0.152	0.12	0.129

In the following scenarios, we choose the level of significance $\alpha =5\%$ and compute ${\delta }_{\mathit{\text{IU}}}^{*},{\delta }_{\mathit{\text{Boot}}}^{*}$ and ${\delta }_{c.\mathit{\text{Boot}}}^{*}$ as the smallest equivalence thresholds for the IU, the bootstrap and the cluster bootstrap tests such that the null hypothesis in (3.1) can still be rejected (i.e., for which equivalence of pre-trends can be concluded). Similarly, we compute the smallest equivalence thresholds ${\tau }^{*}$ and ${\zeta }^{*}$ for the corresponding null hypotheses in (3.2) and (3.3) using the tests in (4.11) and (4.16), respectively. The reported numbers correspond to the average over M = 2500 simulations and can be used to assess at what value of the equivalence threshold a particular test can be expected to reject the null hypothesis. Finally, we report the usual 95% confidence interval ${\text{CI}}^{{\widehat{\pi }}_{\mathit{\text{ATT}}}}$ and the number of simulations in which each β_l for $l=1,\dots ,T$ was found to be statistically insignificant. We then investigate how violations of the PTA affect the chance of falsely detecting a treatment effect and how these violations affect the smallest equivalence thresholds for which equivalence can be concluded. For simplicity, the model errors are drawn independently from a standard normal distribution.

Table C.5 shows the results under the PTA. We further simulate scenarios in which the PTA is violated due to the presence of unobserved effects that affect the treatment group but not the control group. First, we simulate a pre-program shock, also known as “Ashenfelter’s dip” by replacing $u_{i,t}$ in (6.1) by ${\tilde{u}}_{i,t}=u_{i,t}+G_i{D}_{T+1}(l)V_i$, where $V_i\sim \mathrm{N}(\mu ,1)$ with mean $\mu \in \{\frac{1}{4},\frac{1}{2}\}$. This implies that $\mathbb{E}[{\widehat{\pi }}_{\mathit{\text{ATT}}}]=\mathbb{E}[{\widehat{\beta }}_l]=-\mu$ for $l=1,\dots ,T$. The results are presented in Tables C.6 and C.7. In our second scenario, ${\tilde{u}}_{i,t}=\psi \times t\times G_i$ with $\psi =0.025$, modeling an unobserved linear difference in time trends, starting in t = 1. Consequently, $\mathbb{E}[{\widehat{\beta }}_l]=\psi (l-T-1)$ and $\mathbb{E}[{\widehat{\pi }}_{\mathit{\text{ATT}}}]=\psi$. The results are presented in Table C.8.

6.1 Simulation Results—Discussion

Table C.1 shows that the test in (4.11) approximately keeps the desired level for every T even in small samples. The test in (4.16) appears to be slightly over-rejecting when n = 100 but keeps its nominal level in larger samples. Notice that in Table C.2 the tests in (4.11) and (4.16) rightfully reject the null hypothesis in an increasing number of cases as n and T increase as $\overline{\beta }$ and ${\beta }_{\mathit{\text{RMS}}}$ are further away from the boundary of the null, resulting in an increase in statistical power. Regarding the tests for (3.1) Tables C.3 and C.4 illustrate that the IU and cluster bootstrap based tests maintain their nominal level for T = 2. When only one parameter is at the boundary of the null hypothesis, both tests also perform well in the sense that the empirical rejection frequency is close to the nominal level for sufficiently large n. In comparison, the test in (4.7) over-rejects even for large n, showing that it is not robust to high levels of serial correlation. If ${\beta }_l=1$ for all pre-treatment periods, all three tests become conservative for larger values of T. This phenomenon appears to be much more pronounced for the test based on the IU principle, for which it is well-documented (Berger and Hsu 1996). For instance, the empirical level of the IU based test is more than seven times smaller than the corresponding level of the bootstrap based tests for T = 12 (see Table C.1). As shown in Table C.3, this has important consequences for the power of both tests as the cluster bootstrap based test procedure outperforms the IU based test for T > 2. On the other hand, the IU based test may still be attractive for practical applications as it is numerically much less demanding. As compared to the tests for (3.1), the power of our test in (4.16) is substantially larger, only surpassed by the power of the test in (4.11). All tests have in common that the power decreases with T. This is true even for the test in (4.11), which is the (asymptotically) uniformly most powerful test for (3.2) for any T. Thus, concluding equivalence of pre-trends becomes more demanding with an increase in the number of pre-treatment periods. This makes intuitive sense in the DiD setup, where equivalence of pre-trends in a larger number of periods is often regarded as stronger evidence for the plausibility of the PTA.

For T = 1, we find that ${\zeta }^{*}>{\delta }_{\mathit{\text{Boot}}}^{*}\approx {\delta }_{c.\mathit{\text{Boot}}}^{*}>{\delta }_{\mathit{\text{IU}}}^{*}={\tau }^{*}$. This is not surprising, since the IU based test is asymptotically uniformly most powerful for the hypothesis (3.1), where (3.1) coincides with (3.2) and (3.3) when a single pre-treatment parameter is tested. For T > 1, we roughly observe that ${\tau }^{*}<{\zeta }^{*}\le {\delta }_{\mathit{\text{Boot}}}^{*}\le {\delta }_{c.\mathit{\text{Boot}}}^{*}<{\delta }_{\mathit{\text{IU}}}^{*}$. This relationship between the smallest equivalence thresholds of the maximum, average and RMS tests is expected, as $\left|\overline{\beta }\right|\le {\beta }_{\mathit{\text{RMS}}}\le \left|\right|\beta |{|}_{\infty }$. The better performance of the bootstrap based tests for (3.1) as compared to the IU based test may attributed to their higher power as evidenced in Tables C.3 and C.4. We also observe that ${\delta }_{\mathit{\text{Boot}}}^{*}$ performs only slightly better than ${\delta }_{c.\mathit{\text{Boot}}}^{*}$ under the PTA when errors are spherical. Under violations of the PTA, ${\delta }_{\mathit{\text{Boot}}}^{*}<{\delta }_{c.\mathit{\text{Boot}}}^{*}$ for T > 1. However, as ${\tilde{u}}_{i,t}$ becomes nonspherical, only the cluster bootstrap based test maintains its nominal level (see Table C.2). We therefore recommend to use the cluster-robust version of the bootstrap based test in practice. Further notice that even when the PTA holds, the practice of rejecting the DiD framework when ${\widehat{\beta }}_l$ is statistically insignificant for at least one $l\in \{1,\dots ,T\}$ is clearly inefficient as is shown by the first row of Table C.5, as an increase in available pre-treatment periods increases the chance of incorrectly rejecting the DiD framework under the PTA. A similar observation can be made in the presence of a linear time trend as shown in Table C.8. Here, even when the empirical coverage rate of the usual confidence interval is only slightly lower than the nominal level, the DiD framework is rejected in a large number of cases.

When the PTA is violated due to a small temporary shock as in Table C.6, the usual practice of adopting the PTA when no significant differences in pre-trends could be found can lead to a false discovery of a nonzero treatment effect in a substantial number of cases, in particular when the sample size is small. If the temporal shock is larger as in Table C.7, a non-existing treatment effect will be found to be significantly different from zero in almost all cases. All our test procedures require an unrealistically large equivalence threshold in order to be able to conclude equivalence of pre-trends. In particular, any equivalence threshold for which equivalence could be concluded would have to be larger than the estimated treatment effect, therefore, casting serious doubt on the validity of the latter. Similarly, when the PTA is violated due to a linear difference in trends (see Table C.8), the equivalence thresholds would have to be chosen larger than the estimated treatment effect, thus, suggesting that the estimated ATT may contain bias due to insufficient support for the PTA. Moreover, our methodology can be useful in identifying the presence of a linear time trend, as ${\tau }^{*}$ and ${\zeta }^{*}$ tend do remain stable in T under the PTA or when the violation of the PTA is only temporary, whereas under the presence of a linear trend, they increase with T.

7 Empirical Illustration

We illustrate our approach by re-considering the influential Difference-in-Differences analysis in Di Tella and Schargrodsky (2004). They use a shock to the allocation of police forces as a consequence of a terrorist attack on a Jewish institution as a natural experiment to study the effect of police on crime. The original authors conduct the usual pre-test in (2.7) and find no evidence for violations of the PTA. However, Donohue, Ho, and Leahy (2013) point out several shortcomings of the original paper (e.g., spillover effects from the treated to the untreated group). In particular, they find that the PTA is not plausible if the pre-treatment data is inspected on a more granular level, thus, casting doubt on the validity of the estimated treatment effects. While the traditional test failed to detect evidence against the PTA, we will apply our test procedures to analyze how much evidence in favor of the PTA can be extracted from the original specification in Di Tella and Schargrodsky (2004).

The data consists of monthly averages of the number of car thefts between April and December 1994 in each out of 876 Buenos Aires city blocks out of which 37 blocks received additional protection after the attack. The main specification in Di Tella and Schargrodsky (2004) is given by $Y_{\mathit{\text{it}}}={\alpha }_i+{\lambda }_t+\beta D_{\mathit{\text{it}}}$, where Y_it denotes the number of car thefts in block i and month t and D_it is a dummy variable taking the value 1 if block i is treated in period t. Finally, α_i and λ_t are block- and time-specific fixed effects. By using this specification, the pre- and post-treatment periods are pooled together so that the estimated treatment effect compares the post-treatment difference in car thefts between treated and non-treated blocks to the corresponding pre-treatment difference. To analyze group mean differences in the pre-treatment periods, we fit (2.5) to subsets of the data that include one, two or three pre-treatment periods, corresponding to June, May and June and April–June. As in the original paper, we include block- and time-specific effects and cluster on the block level. We find that using heteroscedasticity-robust standard errors instead of clustering has no substantial effect. Finally, we compute ${\delta }_{\mathit{\text{IU}}}^{*},{\delta }_{\mathit{\text{Boot}}}^{*},{\delta }_{c.\mathit{\text{Boot}}}^{*},{\tau }^{*}$ and ${\zeta }^{*}$. The results are summarized in Table 1 below. Notice that for all tests the smallest equivalence threshold that still allows us to conclude equivalence of pre-trends are the largest when only the pre-treatment period June is used. This hints toward a temporary shock to treatment or control in June which may bias the pooled estimates in Table 3 of Di Tella and Schargrodsky (2004). The latter are significant and range between–0.058 and–0.081. One important outcome of our equivalence test based analysis is that, even without the granular data inspection of Donohue, Ho, and Leahy (2013), the equivalence thresholds have to be chosen unrealistically large in order to conclude equivalence of pre-trends. In fact, the smallest equivalence thresholds for which the null hypotheses can be rejected are larger than the estimated effect size of police on crime. Therefore, it is questionable whether there is any effect at all, since the estimated effect may be an artifact of the violated PTA only.

8 Conclusion

We have derived several tests that allow researchers to provide statistical evidence in support of the parallel trends assumption in difference-in-differences estimation by testing for negligible differences in pre-trends in treatment and control. The tests can easily be implemented in popular models such as the two-way fixed effects model, and they can be flexibly adjusted to accommodate more complex settings. Our simulation analysis yields support for our theoretical results as our tests maintain their nominal level and exhibit high statistical power in sufficiently large samples. Finally, we apply our methodology to the data provided by Di Tella and Schargrodsky (2004). Even without a granular inspection of the data as in Donohue, Ho, and Leahy (2013), our methodology casts doubt on the estimated effects, as they may simply be the result of trend differences that could not be detected using traditional pre-tests.

Acknowledgments

We thank the editor Ivan Canay, an associate editor, and two anonymous referees for comments that greatly improved this article. We further thank participants of the International Panel Data Conference 2023, the European Summer Meeting of the Econometric Society 2023 and the European Winter Meeting of the Econometric Society 2023.

Supplementary Materials

The supplementary material consists of two R files that replicate Table 1 in Section 7 and the simulation results in Appendix C.

Disclosure Statement

The authors report there are no competing interests to declare.

Appendix A:

Assumption 4.1 in the Canonical DiD Model of Section 2

For $\lambda \in [\epsilon ,1]$, compute the double-demeaned variables based on a random subsample ${\mathcal{S}}_{n,\lambda }$ of $\lfloor n\lambda \rfloor$ individuals, each observed in periods $1,\dots ,T+1$. For instance, we define the double-demeaned data as ${\ddot{Y}}_{i,t}(\lambda )=Y_{i,t}-{\overline{Y}}_{i,·}-{\overline{Y}}_{·,t}(\lambda )+{\overline{Y}}_{··}(\lambda )$, where $${\overline{Y}}_{·,t}(\lambda )=\frac{1}{\lfloor n\lambda \rfloor }\sum _{j\in {\mathcal{S}}_{n,\lambda }}{Y}_{j,t} \text{and} {\overline{Y}}_{··}(\lambda )=\frac{1}{\lfloor n\lambda \rfloor (T+1)}\sum _{j\in {\mathcal{S}}_{n,\lambda }}\sum _{s=1}^{T+1}{Y}_{j,s},$$ and other variables such as ${\ddot{W}}_{i,t}(\lambda )$ and ${\ddot{u}}_{i,t}(\lambda )$ are defined analogously. Collecting the individual time series, (A.1) $${\ddot{Y}}_i(\lambda )={\ddot{W}}_i(\lambda )\prime \beta +{\ddot{u}}_i(\lambda ),$$(A.1) where ${\ddot{Y}}_i(\lambda )\in {\mathbb{R}}^{T+1},{\ddot{W}}_i(\lambda )\in {\mathbb{R}}^{T\times (T+1)}$ and ${\ddot{u}}_i(\lambda )\in {\mathbb{R}}^{T+1}$. Then, let $\widehat{\beta }(\lambda )$ denote the OLS estimator for β in model (A.1) from the subsample of $\lfloor n\lambda \rfloor$ individuals, that is $$\begin{array}{c}\widehat{\beta }(\lambda )={(\widehat{\Gamma }(\lambda ))}^{-1}\frac{1}{\lfloor n\lambda \rfloor }\sum _{i\in {\mathcal{S}}_{n,\lambda }}{\ddot{W}}_i(\lambda ){\ddot{Y}}_i(\lambda )\\ =\beta +{(\widehat{\Gamma }(\lambda ))}^{-1}\frac{1}{\lfloor n\lambda \rfloor }\sum _{i\in {\mathcal{S}}_{n,\lambda }}{\ddot{W}}_i(\lambda ){\ddot{u}}_i(\lambda ),\end{array}$$ where the matrix $\Gamma (\lambda )$ is defined by $\widehat{\Gamma }(\lambda )=\frac{1}{\lfloor n\lambda \rfloor }\sum _{i\in {\mathcal{S}}_{n,\lambda }}{\ddot{W}}_i(\lambda ){\ddot{W}}_i(\lambda )\prime$. Now, assume that $\Gamma ={\text{plim}}_{n\to \infty }\frac{1}{n}{\sum }_{i=1}^n{\ddot{W}}_i{\ddot{W}}_i^{\prime }$ is nonsingular, where ${\ddot{W}}_i={\ddot{W}}_i(1)$ and other variables are defined analogously. Under random sampling across individuals (see, e.g., Callaway and Sant’Anna 2021) and assuming that sufficient moments exist, it then follows that $$\underset{\lambda \in [\epsilon ,1]}{\mathrm{\text{sup}}}\left|\right|\widehat{\Gamma }(\lambda )-\Gamma \left|\right|=o_{\mathbb{P}}(1)\text{as}n\to \infty ,$$

and $$\sqrt{n}(\widehat{\beta }(\lambda )-\beta )={\Gamma }^{-1}\frac{\sqrt{n}}{\lfloor n\lambda \rfloor }\sum _{i\in {\mathcal{S}}_{n,\lambda }}{\ddot{W}}_i(\lambda ){\ddot{u}}_i(\lambda )+o_{\mathbb{P}}(1)$$ uniformly with respect to $\lambda \in [\epsilon ,1]$. Consequently, we obtain from the Cramer-Wold device and Theorem 2.12.1 in van der Vaart and Wellner (1996) that $${\{\sqrt{n}(\widehat{\beta }(\lambda )-\beta )\}}_{\lambda \in [\epsilon ,1]}⇝{\{\frac{V^{1/2}}{\lambda }\overrightarrow{\mathbb{B}}(\lambda )\}}_{\lambda \in [\epsilon ,1]}$$

where $\overrightarrow{\mathbb{B}}$ is a T-dimensional vector of independent Brownian motions, $V={\Gamma }^{-1}W{\Gamma }^{-1}$ with $$W={\text{plim}}_{n\to \infty }\frac{1}{n}\sum _{i=1}^n{\ddot{W}}_i{\ddot{u}}_i{\ddot{u}}_i^{\prime }{\ddot{W}}_i^{\prime },$$ and the symbol $⇝$ means weak convergence in the space ${({\mathcal{l}}^{\infty }[\epsilon ,1])}^T$ of all T-dimensional bounded functions on the interval $[\epsilon ,1]$.

Appendix B:

Mathematical Proofs

B.1. Properties of the Test (4.3)

For sufficiently large sample sizes the quantile $f_{\alpha }:={\mathcal{Q}}_{{\mathrm{N}}_F(\delta ,{\widehat{\Sigma }}_{11}/n)}(\alpha )$ satisfies (B.1) $$\begin{array}{c}\alpha =\mathbb{P}(|\mathrm{N}(\delta ,{\widehat{\Sigma }}_{11}/n)|\le {\mathcal{Q}}_{{\mathrm{N}}_F(\delta ,{\widehat{\Sigma }}_{11}/n)}(\alpha ))\\ =\Phi \left(\frac{f_{\alpha }-\delta }{{\Sigma }_{11}}\right)-\Phi \left(\frac{-f_{\alpha }-\delta }{{\Sigma }_{11}}\right)+\mathrm{O}\left(\frac{1}{\sqrt{n}}\right)\end{array}$$(B.1)

where $\Phi$ is the cdf of the standard normal distribution. Consequently, we obtain for the probability of rejection (B.2) $${\mathbb{P}}_{{\beta }_1}(|{\widehat{\beta }}_1|\le f_{\alpha })\approx \Phi \left(\frac{f_{\alpha }-{\beta }_1}{{\Sigma }_{11}}\right)-\Phi \left(\frac{-f_{\alpha }-{\beta }_1}{{\Sigma }_{11}}\right).$$(B.2)

It is well known that the right-hand side of (B.2) (with the quantile $f_{\alpha }$ defined by (B.1)) is the power function of the uniformly most powerful unbiased test (see Example 1.1 in Romano 2005).

B.2. Proof of Theorem 4.1

The proof follows essentially by the same arguments as given in Dette et al. (2018), and, for the sake of brevity, we only explain why this is the case (also see the discussion below in Section B.3, where a sequential version of the result is derived). By Assumption 4.1, $\sqrt{n}(\widehat{\beta }-\beta )$ has an asymptotic T-dimensional centered normal distribution. We denote the corresponding asymptotic covariance matrix by $\Sigma ={({\sigma }_{\mathit{\text{ij}}})}_{i,j=1,\dots T}$. Now we interpret all vectors as stochastic processes on the set $\mathcal{X}=\{1,\dots ,T\}$ and rewrite the weak convergence of the vector $\widehat{\beta }=({\widehat{\beta }}_1,\dots ,{\widehat{\beta }}_T)\prime$ as (B.3) $${\{\sqrt{n}({\widehat{\beta }}_x-{\beta }_x)\}}_{x\in \mathcal{X}}⇝{\{\mathbb{G}(x)\}}_{x\in \mathcal{X}},$$(B.3) where ${\{\mathbb{G}(x)\}}_{x\in \mathcal{X}}$ is a centered Gaussian process on $\mathcal{X}=\{1,\dots ,T\}$ with covariance structure $\text{cov}(\mathbb{G}(x),\mathbb{G}(y))={\sigma }_{\mathit{\text{xy}}}$ ($x,y\in \mathcal{X}$). Note that (B.3) is the analog of eq. (A.7) in Dette et al. (2018), and it follows by exactly the same arguments as stated in this article that (B.4) $$\sqrt{n}(|\left|\widehat{\beta }\right|{|}_{\infty }-\left|\right|\beta |{|}_{\infty })\to \mathrm{\text{max}}\{\underset{x\in {\mathcal{E}}^{+}}{\mathrm{\text{max}}}\mathbb{G}(x),\underset{x\in {\mathcal{E}}^{-}}{\mathrm{\text{max}}}-\mathbb{G}(x)\},$$(B.4) provided that $\left|\right|\widehat{\beta }|{|}_{\infty }>0$, where the sets ${\mathcal{E}}^{+}$ and ${\mathcal{E}}^{-}$ are defined by $$\begin{array}{c}{\mathcal{E}}^{+}=\{\mathcal{l}=1,\dots ,T:{\beta }_{\mathcal{l}}=|\left|\beta \right|{|}_{\infty }\},\\ {\mathcal{E}}^{-}=\{\mathcal{l}=1,\dots ,T:{\beta }_{\mathcal{l}}=-|\left|\beta \right|{|}_{\infty }\},\end{array}$$ respectively. Note that ${\mathcal{E}}^{-}\cup {\mathcal{E}}^{+}=\mathcal{E}$, where $\mathcal{E}$ is defined in (4.9), and that (B.3) is the analog of Theorem 3 in Dette et al. (2018). Moreover, if ${\widehat{\beta }}^{*}=({\widehat{\beta }}_1^{*},\dots ,{\widehat{\beta }}_T^{*})\prime$ denotes the estimate from the bootstrap sample, we obtain an analog of the weak convergence in (B.3), that is (B.5) $${\{\sqrt{n}({\widehat{\beta }}_x^{*}-{\widehat{\widehat{\beta }}}_x)\}}_{x\in \mathcal{X}}⇝{\{\mathbb{G}(x)\}}_{x\in \mathcal{X}}$$(B.5) conditional on the sample $({\ddot{W}}_1,{\ddot{Y}}_1),\dots ,({\ddot{W}}_n,{\ddot{Y}}_n)$. Note that this statement corresponds to the statement (A.25) in Dette et al. (2018). Now the statements (A.7) and (A.25) and their Theorem 3 are the main ingredients for the proof of Theorem 5 in Dette et al. (2018). In the present context these statements can be replaced by (B.3), (B.5), and (B.4), respectively, and a careful inspection of the arguments given in Dette et al. (2018) shows that Theorem 4.1 holds (the arguments even simplify substantially as in our case the index set $\mathcal{X}$ of the processes is finite).

B.3. Proof of Theorem 4.2

Let Assumption 4.1 hold. In the case $\beta =0$ the result in Theorem 4.2 now follows directly from the continuous mapping theorem. On the other hand, if $\beta \cancel{=}0$, it follows that $$\begin{array}{c}{H}_n(\lambda )=\sqrt{n}(||\widehat{\beta }(\lambda )|{|}^2-\left|\right|\beta |{|}^2)\\ =\sqrt{n}\left\{\right||\widehat{\beta }(\lambda )-\beta |{|}^2+2(\widehat{\beta }(\lambda )-\beta )\prime \beta \\ =2\sqrt{n}(\widehat{\beta }(\lambda )-\beta )\prime \beta +o_{\mathbb{P}}(1)\end{array}$$ uniformly with respect to $\lambda \in [\epsilon ,1]$, and a further application of the continuous mapping theorem yields $${\{H_n(\lambda )\}}_{\lambda \in [\epsilon ,1]}⇝{\{2\beta \prime D\frac{\overrightarrow{\mathbb{B}}(\lambda )}{\lambda }\}}_{\lambda \in [\epsilon ,1]}$$ in ${\mathcal{l}}^{\infty }([\epsilon ,1])$. It is easy to see that for $\beta \ne 0$ the process on the right-hand side equals in distribution $${\{\Delta (\beta )\frac{{\mathbb{B}}_1(\lambda )}{\lambda }\}}_{\lambda \in [\epsilon ,1]}$$

where ${\mathbb{B}}_1$ is a one-dimensional Brownian motion and (B.6) $$\Delta (\beta )=4\beta \prime \mathit{\text{DD}}\prime \beta$$(B.6) is a positive constant. Recalling the definition of the statistic ${\widehat{M}}_n$ in (4.12) and a further application of the continuous mapping theorem shows that $$\begin{array}{c}{\widehat{M}}_n=\frac{{\widehat{\beta }}_{\mathit{\text{RMS}}}^2(1)-{\beta }_{\mathit{\text{RMS}}}^2}{{\widehat{V}}_n}\\ =\frac{\left|\right|\widehat{\beta }(1)|{|}^2-|\left|\beta \right|{|}^2}{({\int }_{\epsilon }^1(||\widehat{\beta }(\lambda )|{|}^2-||\widehat{\beta }(1)|{|}^2{)}^2\nu (d\lambda ){)}^{1/2}}\\ =\frac{H_n(1)}{({\int }_{\epsilon }^1(H_n(\lambda )-H_n(1){)}^2\nu (d\lambda ){)}^{1/2}}\\ \stackrel{d}{\to }\mathbb{W}=\frac{{\mathbb{B}}_1(1)}{({\int }_{\epsilon }^1({\mathbb{B}}_1(\lambda )/\lambda -{\mathbb{B}}_1(1){)}^2\nu (d\lambda ){)}^{1/2}},\end{array}$$ which proves the assertion.

B.4. Proof of Theorem 4.3

Observing the definition of ${\widehat{M}}_T$ in (4.12) we obtain $${\mathbb{P}}_{\beta }({\widehat{\beta }}_{\mathit{\text{RMS}}}^2<{\zeta }^2+{\mathcal{Q}}_{\mathbb{W}}(\alpha ){\widehat{V}}_n)={\mathbb{P}}_{\beta }({\widehat{M}}_n<\frac{{\zeta }^2-{\beta }_{\mathit{\text{RMS}}}^2}{{\widehat{V}}_n}+Q_{\mathbb{W}}(\alpha )).$$

It follows from the proof of Theorem 4.2 that ${\widehat{V}}_n=O_{\mathbb{P}}(1/\sqrt{n})$. Consequently, if ${\beta }_{\mathit{\text{RMS}}}^2>0$, the assertion follows by a simple calculation considering the three cases separately. On the other hand, if ${\beta }_{\mathit{\text{RMS}}}=0$, the proof of Theorem 4.2 also shows that $\left|\right|\widehat{\beta }(1)|{|}^2=O_{\mathbb{P}}(\frac{1}{n})$ and the assertion follows from the weak convergence (4.15) in Theorem 4.2.

Appendix C:

Simulation Results

Table C.1 Rejection frequencies for ${\beta }_t=1,t=1,\dots ,T$ with equivalence threshold 1 at nominal level of significance $\alpha =5\%$.

	n = 100				n = 1000
$\text{Test}$	T = 1	T = 4	T = 8	T = 12	T = 1	T = 4	T = 8	T = 12
(4.4)	0.0546	0.0043	0.0020	0.0004	0.0470	0.0046	0.0013	0.0009
(4.7)	0.0517	0.0134	0.0092	0.0073	0.0477	0.0137	0.0103	0.0069
(4.7) ${}_{\text{cluster}}$	0.0520	0.0114	0.0088	0.0079	0.0483	0.0116	0.0086	0.0065
(4.11)	0.0546	0.0528	0.0517	0.0537	0.0470	0.0493	0.0532	0.0491
(4.16)	0.0727	0.0663	0.0737	0.0801	0.0469	0.0450	0.0487	0.0500

Table C.2 Rejection frequencies for ${\beta }_1=1$ and ${\beta }_l=0,l=2,\dots ,T$ with equivalence threshold 1 at nominal level of significance $\alpha =5\%$.

	$n=100$				$n=1000$
$\text{Test}$	T = 1	T = 4	T = 8	T = 12	T = 1	T = 4	T = 8	T = 12
(4.4)	0.0554	0.0394	0.0089	0.0013	0.0507	0.0512	0.0498	0.0502
(4.7)	0.0526	0.0998	0.1283	0.1417	0.0516	0.0866	0.1038	0.1126
(4.7) ${}_{\text{cluster}}$	0.0544	0.0839	0.1188	0.1316	0.0520	0.0513	0.0505	0.0505
(4.11)	0.0554	0.9690	0.9709	0.9446	0.0507	1.0000	1.0000	1.0000
(4.16)	0.0761	0.7331	0.7590	0.6957	0.0482	0.9998	1.0000	1.0000

Table C.3 Rejection frequencies for n = 1000 with equivalence threshold 1 at nominal level of significance $\alpha =5\%$.

	${\beta }_l=0.8,l=1,\dots ,T$				${\beta }_l=0.9,l=1,\dots ,T$
$\text{Test}$	T = 1	T = 4	T = 8	T = 12	T = 1	T = 4	T = 8	T = 12
(4.4)	0.8807	0.5460	0.3132	0.2017	0.4042	0.1099	0.0456	0.0246
(4.7)	0.8805	0.6827	0.5489	0.4507	0.4058	0.2034	0.1401	0.1060
(4.7) ${}_{\text{cluster}}$	0.8809	0.6397	0.4838	0.3965	0.4068	0.1807	0.1238	0.0984
(4.11)	0.8807	0.8958	0.8197	0.7500	0.4042	0.4276	0.3542	0.3112
(4.16)	0.7000	0.7146	0.6268	0.5675	0.2935	0.3006	0.2575	0.2311

Table C.4 Rejection frequencies for n = 1000 with equivalence threshold 1 at nominal level of significance $\alpha =5\%$.

	${\beta }_1=0.8,{\beta }_l=0,l=2,\dots ,T$				${\beta }_1=0.9,{\beta }_l=0,l=2,\dots ,T$
$\text{Test}$	T = 1	T = 4	T = 8	T = 12	T = 1	T = 4	T = 8	T = 12
(4.4)	0.8776	0.6803	0.5274	0.4479	0.4103	0.2768	0.2161	0.1863
(4.7)	0.8763	0.7746	0.6768	0.6207	0.4108	0.3769	0.3472	0.3191
(4.7) ${}_{\text{cluster}}$	0.8771	0.6804	0.5299	0.4488	0.4103	0.2794	0.2172	0.1878
(4.11)	0.8776	1.0000	1.0000	1.0000	0.4103	1.0000	1.0000	1.0000
(4.16)	0.6920	1.0000	1.0000	1.0000	0.3003	1.0000	1.0000	1.0000

Table C.5 Treatment effect estimation (${\pi }_{\mathit{\text{ATT}}}=0$) and smallest equivalence thresholds under the PTA at nominal level of significance $\alpha =5\%$.

	$n=100$				$n=1000$
	T = 1	T = 4	T = 8	T = 12	T = 1	T = 4	T = 8	T = 12
$\#\mathit{\text{insig}}/M$	0.9512	0.8340	0.7204	0.6468	0.9516	0.8472	0.7392	0.6800
${\widehat{\pi }}_{\mathit{\text{ATT}}}$	0.0019	–0.0066	–0.0036	0.0031	–0.0001	0.0002	0.0001	–0.0008
${\text{CI}}^{{\widehat{\pi }}_{\mathit{\text{ATT}}}}$	0.9532	0.9512	0.9460	0.9496	0.9528	0.9496	0.9480	0.9564
${\delta }_{\mathit{\text{IU}}}^{*}$	0.6300	0.8491	0.9265	0.9626	0.2018	0.2673	0.2930	0.3045
${\delta }_{\mathit{\text{Boot}}}^{*}$	0.6426	0.6875	0.7020	0.7017	0.2029	0.2128	0.2194	0.2208
${\delta }_{c.\mathit{\text{Boot}}}^{*}$	0.6401	0.6915	0.7040	0.7049	0.2028	0.2128	0.2195	0.2210
${\tau }^{*}$	0.6295	0.5029	0.4808	0.4662	0.2017	0.1570	0.1527	0.1477
${\zeta }^{*}$	0.7132	0.6961	0.6884	0.6881	0.2247	0.2166	0.2142	0.2153

Table C.6 Treatment effect estimation (${\pi }_{\mathit{\text{ATT}}}=0$) and smallest equivalence thresholds under a violation of the PTA due to a temporary group-specific shock with mean 0.25 at nominal level of significance $\alpha =5\%$.

	$n=100$				$n=1000$
	T = 2	T = 4	T = 8	T = 12	T = 2	T = 4	T = 8	T = 12
$\#\mathit{\text{insig}}/M$	0.8256	0.6668	0.5532	0.4456	0.1560	0.1016	0.1816	0.0772
${\widehat{\pi }}_{\mathit{\text{ATT}}}$	–0.2501	–0.2528	–0.2574	–0.2580	–0.2495	–0.2495	–0.2450	–0.2499
${\text{CI}}^{{\widehat{\pi }}_{\mathit{\text{ATT}}}}$	0.8276	0.8060	0.6820	0.8916	0.1436	0.2044	0.3408	0.2868
${\delta }_{\mathit{\text{IU}}}^{*}$	0.7035	0.9763	1.3027	1.3185	0.3916	0.4613	0.5188	0.5528
${\delta }_{\mathit{\text{Boot}}}^{*}$	0.7173	0.8000	1.0382	0.9474	0.3937	0.4194	0.4438	0.4657
${\delta }_{c.\mathit{\text{Boot}}}^{*}$	0.7141	0.8622	1.1419	1.0820	0.3936	0.4456	0.4786	0.5031
${\tau }^{*}$	0.7032	0.7018	0.8940	0.8073	0.3916	0.3838	0.3979	0.4082
${\zeta }^{*}$	0.7619	0.8266	1.0326	0.9845	0.4025	0.4022	0.4258	0.4321

Table C.7 Treatment effect estimation (${\pi }_{\mathit{\text{ATT}}}=0$) and smallest equivalence thresholds under a violation of the PTA due to a temporary group-specific shock with mean 0.5 at nominal level of significance $\alpha =5\%$.

	$n=100$				$n=1000$
	T = 2	T = 4	T = 8	T = 12	T = 2	T = 4	T = 8	T = 12
$\#\mathit{\text{insig}}/M$	0.5240	0.3468	0.3080	0.2928	0.0000	0.0000	0.0076	0.0000
${\widehat{\pi }}_{\mathit{\text{ATT}}}$	–0.5020	–0.5027	–0.4971	–0.4942	–0.4973	–0.5008	–0.5005	–0.5017
${\text{CI}}^{{\widehat{\pi }}_{\mathit{\text{ATT}}}}$	0.5040	0.5384	0.5500	0.6760	0.0000	0.0004	0.0168	0.0000
${\delta }_{\mathit{\text{IU}}}^{*}$	0.9245	1.1822	1.4257	1.4784	0.6367	0.7121	0.7648	0.8013
${\delta }_{\mathit{\text{Boot}}}^{*}$	0.9377	1.0416	1.1826	1.1626	0.6371	0.6718	0.6980	0.7182
${\delta }_{c.\mathit{\text{Boot}}}^{*}$	0.9338	1.1149	1.2897	1.3010	0.6371	0.6981	0.7350	0.7566
${\tau }^{*}$	0.9244	0.9236	1.0280	0.9966	0.6367	0.6357	0.6471	0.6587
${\zeta }^{*}$	0.9527	1.0077	1.1419	1.1178	0.6661	0.6722	0.6820	0.6932

Table C.8 Treatment effect estimation (${\pi }_{\mathit{\text{ATT}}}=0$) and smallest equivalence thresholds under a violation of the PTA due to a time trend with slope 0.025 at nominal level of significance $\alpha =5\%$.

	$n=100$				$n=1000$
	T = 2	T = 4	T = 8	T = 12	T = 2	T = 4	T = 8	T = 12
$\#\mathit{\text{insig}}/M$	0.9280	0.8156	0.6568	0.4848	0.9456	0.6736	0.1876	0.0144
${\widehat{\pi }}_{\mathit{\text{ATT}}}$	0.0304	0.0240	0.0337	0.0208	0.0261	0.0230	0.0238	0.0253
${\text{CI}}^{{\widehat{\pi }}_{\mathit{\text{ATT}}}}$	0.9432	0.9456	0.9380	0.9516	0.9348	0.9456	0.9416	0.9452
${\delta }_{\mathit{\text{IU}}}^{*}$	0.6474	0.8596	0.9551	1.0392	0.2039	0.2992	0.3892	0.4859
${\delta }_{\mathit{\text{Boot}}}^{*}$	0.6604	0.7060	0.7454	0.8189	0.2052	0.2589	0.3538	0.4590
${\delta }_{c.\mathit{\text{Boot}}}^{*}$	0.6580	0.7101	0.7470	0.8221	0.2048	0.2586	0.3538	0.4587
${\tau }^{*}$	0.6470	0.5103	0.4978	0.5203	0.2038	0.1853	0.2229	0.2696
${\zeta }^{*}$	0.7396	0.7055	0.7048	0.7144	0.2282	0.2342	0.2643	0.3122