Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions That Need to Be Powered to Detect Small Impacts

Abstract

Evaluators of education interventions are increasingly designing studies to detect impacts much smaller than the 0.20 standard deviations that Cohen characterized as “small.” While the need to detect smaller impacts is based on compelling arguments that such impacts are substantively meaningful, the drive to detect smaller impacts may create a new challenge for researchers: the need to guard against smaller biases. The purpose of this article is twofold. First, we examine the potential for small biases to increase the risk of making false inferences as studies are powered to detect smaller impacts, a phenomenon we refer to as asymdystopia. We examine this potential for two of the most rigorous designs commonly used in education research—randomized controlled trials and regression discontinuity designs. Second, we recommend strategies researchers can use to avoid or mitigate these biases.

Keywords:

• Bias
• attrition
• regression discontinuity designs
• randomized controlled trials

Acknowledgments

We thank Jessie Mazeika for excellent research assistance. We thank Luke Miratrix and two anonymous referees for helpful comments.

Open Scholarship

This article has earned the Center for Open Science badges for Open Data and Open Materials through Open Practices Disclosure. The data and materials are openly accessible at https://ies.ed.gov/ncee/wwc/StudyFindings, https://ies.ed.gov/ncee/projects/evaluation/data_files.asp, https://osf.io/3x2zu/?view_only=92275575ad2c4ce5ab5cac4bdc38787a and https://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=NCEE20184002.

Notes

1 We acknowledge the recent critiques of the NHST framework (Amrhein et al., 2019; Wasserstein & Lazar, 2016) and do not intend for this article to implicitly endorse its continued use. The issues raised in this article are equally applicable to any inferential method (for example, Bayesian posterior probabilities) that ignores small biases when assessing and reporting uncertainty in studies powered to detect small impacts. We therefore stick with the NHST framework in this article for simplicity and because it is likely to be familiar to the widest range of readers.

2 Some studies—particularly retrospective nonexperimental studies using administrative data—have the statistical power to detect effects that are too small to be substantively important. This article does not focus on “overpowered” studies. Instead, we focus on studies that are designed to have just enough statistical power to detect the smallest impact that is substantively important.

3 The term “assignment variable” is often used interchangeably with “forcing variable,” “running variable,” and “score.” Truly continuous assignment variables are atypical in practice, although methods do exist to account for discreteness (for example, Armstrong & Kolesár, 2018; Barreca et al., 2016; Kolesár & Rothe, 2018; Lee & Card, 2008).

4 See Bloom (2005); Bloom et al. (2007); Deke and Dragoset (2012); Hedges and Hedberg (2007); Murray (1998); and Schochet (2008a, 2008b) for more information about calculating statistical power in both RCTs and RDDs.

5 The What Works Clearinghouse, managed by the U.S. Department of Education’s Institute of Education Sciences, systematically reviews and synthesizes education research studies with the goal of providing a reliable source of scientific evidence for what works in education to improve student outcomes. For more information, see http://ies.ed.gov/ncee/wwc/.

6 Asymdystopia can also arise in other causal impact designs, such as quasi-experiments (QEDs). For example, omitted variable bias can often lead to upwardly biased QED impact estimates, which likely do not diminish with sample size. A formal consideration of asymdystopia in QEDs is beyond the scope of this article, however.

7 This summary draws heavily from the WWC’s technical methods paper entitled Assessing Attrition Bias (https://ies.ed.gov/ncee/wwc/Document/243), which includes complete details of the attrition model.

8 The U.S. Department of Health and Human Services has also used this model. See, for example, the Home Visiting Evidence of Effectiveness Review (http://homvee.acf.hhs.gov) and the Teen Pregnancy Prevention Evidence Review (http://tppevidencereview.aspe.hhs.gov).

9 For any observed overall and differential attrition rates, there are many values of ${\alpha }_t$ and ${\alpha }_c$ that would yield a given level of bias (see Equation (3)(3) $$B=E\left(y_t|{z_t>z}_t^{\mathrm{*}}\right)-E\left(y_c|z_c>z_c^{\mathrm{*}}\right)={\alpha }_{t}E\left(z_t|z_t>z_t^{\mathrm{*}}\right)-{\alpha }_{c}E\left(z_c|z_c>z_c^{\mathrm{*}}\right)=\frac{{\alpha }_t\times \varphi \left({\mathrm{\Phi }}^{-1}\left(1-P_t\right)\right)}{P_t}\mathrm{ }-\mathrm{ }\frac{{\alpha }_c\times \varphi \left({\mathrm{\Phi }}^{-1}\left(1-P_c\right)\right)}{P_c}.$$(3) ). To calculate a unique pair of model parameters for each given level of bias, we assume that ${\alpha }_t=r{\alpha }_c,$ where r is a constant equal to the ratio of ${\alpha }_t$ to ${\alpha }_c$ implicit in the WWC parameters (0.27/0.22). This approach allows us to uniquely characterize how optimistic the study parameters would need to be to contain bias.

10 Note that it does not matter which attrition rates and values of $\alpha$ correspond to the treatment or control groups—switching all treatment and control labels would yield the same conclusions.

11 We exclude quick reviews because the review protocol differs from other types of reviews. The database is available at https://ies.ed.gov/ncee/wwc/StudyFindings.

12 Typically the MDE is expressed as a function of the standard error. However, the WWC does not record standard errors, so we infer the standard error from the combination of the impact estimate, p-value, and analytical sample size. For each study, we calculated the MDE using the following formula: $\mathit{MDE}=\left[T^{-1}\left(N-1,1-\frac{\alpha }{2}\right)+T^{-1}\left(N-1,\beta \right)\right]\times \left|ES/T^{-1}\left(N-1,\frac{p}{2}\right)\right|,$ where $T^{-1}$ is the inverse $t$-distribution, $\alpha$ is the significance level (assumed to be 0.05), $\beta$ is the power (assumed to be 0.80), ES is the effect size, p is the p-value, $N$ is the analytical sample size for the unit of randomization, and the vertical bars indicate the absolute value.

13 A non-exhaustive list of relevant papers includes: Armstrong and Kolesár (2020); Bartalotti (2019); Bartalotti et al. (2017); Branson et al. (2019); Calonico et al. (2020); Cattaneo et al. (2015, 2017); He and Bartalotti (2020); Imbens and Wager (2019); Noack and Rothe (2020); Sales and Hansen (2019).

14 In some cases, these algorithms select bandwidths that are so narrow there are not enough data to calculate an impact and/or a standard error. In those cases, we automatically expand the bandwidth until we can calculate an impact and standard error. Of the seven DGPs (see Figures A3 and A4), three had cases where the bandwidth had to be expanded. For the DGPs represented in Figure A3 panes (b) and (c), the bandwidth had to be expanded in 78% of Monte Carlo replications when the CCT bandwidth selection algorithm was used with a sample size of 100,000. For the DGP represented in Figure A4, pane (d), the bandwidth had to be expanded in up to 1% of replications when the IK bandwidth was used (regardless of sample size). When the CCT algorithm was used, the bandwidth had to be expanded in 70% of replications with a sample size of 1,000 and over 99% of replications with a sample size of 10,000 or 100,000.

15 The WWC attrition model does not directly incorporate covariates. However, the benefits of adjusting for covariates can be reflected in the model by making more optimistic assumptions regarding the negative consequences of attrition.

16 We recognize that in the case of retrospective studies, researchers have much less control over the data they have and the analyses those data can support.

Additional information

Funding

This article is based on the following report released by the Institute of Education Sciences at the U.S. Department of Education: Deke, J., Wei, T., & Kautz, T. (2017). Asymdystopia: The threat of small biases in evaluations of education interventions that need to be powered to detect small impacts [NCEE 2018-4002]. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. This article has been funded in part by federal funds from the U.S. Department of Education under contract [number ED-IES-12-C-0083]. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Education nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government. This article has been funded in part by Mathematica, Inc. (©2019 Mathematica, Inc.).