Inference on a New Class of Sample Average Treatment Effects

Abstract

We derive new variance formulas for inference on a general class of estimands of causal average treatment effects in a randomized control trial. We generalize the seminal work of Robins and show that when the researcher’s objective is inference on sample average treatment effect of the treated (SATT), a consistent variance estimator exists. Although this estimand is equal to the sample average treatment effect (SATE) in expectation, potentially large differences in both accuracy and coverage can occur by the change of estimand, even asymptotically. Inference on SATE, even using a conservative confidence interval, provides incorrect coverage of SATT. We demonstrate the applicability of the new theoretical results using an empirical application with hundreds of online experiments with an average sample size of approximately 100 million observations per experiment. An R package, estCI, that implements all the proposed estimation procedures is available. Supplementary materials for this article are available online.

Keywords:

• Average treatment effect
• Causality
• Neyman causal model

Supplementary Materials

The supplemental material includes replication code, proofs, and additional results that are mentioned in the text only briefly.

Acknowledgments

We thank Peter Aronow, Max Balandat, Eytan Bakshy, Avi Feller, Johann Gagnon-Bartsch, Peng Ding, Ben Hansen, Guido Imbens, James Robins, Sören Kunzel, Winston Lin, Juliana Londoño Vélez, Fredrik Sävje, and John Myles White for helpful comments and discussions. In addition, we thank Max Balandat, Eytan Bakshy, and John Myles White for help with the data application. We also thank the participants of the Berkeley Statistics Annual Research Symposium 2017 and the Atlantic Causal Inference Conference 2017.

Notes

1 This holds even though SATE and SATT are equal in expectation.

2 The terms SATE, SATT, PATE, and PATT have been first used, to our knowledge, by Imbens (2004).

3 A related literature discusses the idea of an optimal estimand in terms of covariate balance in observational studies (Crump et al. 2009; Li, Morgan, and Zaslavsky 2018). Crump et al. (2009) suggested a procedure for choosing the optimal estimand in observational studies where there is limited overlap in the covariates. The population overlap issue does not arise in randomized experiments.

4 For a review of the classic CLT results under the finite population model, see Li and Ding (2016). Note that, the randomization model implies that the number of treated units, m, is a fixed number and not a random variable. The only random component in the model is the treatment indicators, T .

5 The estimator in Equation (5) corresponds to Neyman’s second variance estimator when the estimand is SATE (Neyman 1923/1990).

6 These are not the weakest possible conditions.

7 Note that Lemma 3.3 does not cover other estimators for the variance of $\text{var}(t_{\text{diff}}-\text{SATE})$, such as that of Aronow, Green, and Lee (2014). CIs based on these other variance estimators may be shorter than our PIs even when ${\sigma }_1\ne {\sigma }_0$. To address these variance estimators, we derive the accuracy gains for a more general case in which ρ can be bounded by ${\rho }^{*}$, $\rho \le {\rho }^{*}$. As the variance of $\text{var}(t_{\text{diff}}-\text{SATE})$ is increasing w.r.t. ρ, it follows that substituting ${\rho }^{*}$ with ρ yields a conservative variance estimator that is smaller than Neyman’s variance estimator. The idea of substituting a bound of ρ instead of the true parameter value was proposed before in the literature (Reichardt and Gallob 1999; Aronow, Green, and Lee 2014). The percentage gain in terms of CI length is: $1-\frac{1}{\sqrt{p^2+{(1-p)}^2·{(\frac{{\sigma }_1}{{\sigma }_0})}^2+2p(1-p)·{\rho }^{*}·\frac{{\sigma }_1}{{\sigma }_0}}}$.

8 In Appendix Figure C.2, we show that the simulation results are not sensitive to our choice of using N = 1000 and would have been the same using N = 100 or N = 1300. This holds for both this data-generating process and the Tobit model that is discussed below.

Additional information

Funding

Sekhon wishes to acknowledge Office of Naval Research (ONR) grants N00014-15-1-2367 and N00014-17-1-2176.