Design and Analytic Features for Reducing Biases in Skill-Building Intervention Impact Forecasts

Abstract

Despite policy relevance, longer-term evaluations of educational interventions are relatively rare. A common approach to this problem has been to rely on longitudinal research to determine targets for intervention by looking at the correlation between children’s early skills (e.g., preschool numeracy) and medium-term outcomes (e.g., first-grade math achievement). However, this approach has sometimes over—or under—predicted the long-term effects (e.g., 5th-grade math achievement) of successfully improving early math skills. Using a within-study comparison design, we assess various approaches to forecasting medium-term impacts of early math skill-building interventions. The most accurate forecasts were obtained when including comprehensive baseline controls and using a combination of conceptually proximal and distal short-term outcomes (in the nonexperimental longitudinal data). Researchers can use our approach to establish a set of designs and analyses to predict the impacts of their interventions up to 2 years post-treatment. The approach can also be applied to power analyses, model checking, and theory revisions to understand mechanisms contributing to medium-term outcomes.

Keywords:

• Prediction
• forecasting
• non-experimental
• intervention
• evaluation

Notes

1 Intervention designers may view impacts measured after two years of end-of-treatment as long-term impacts since the interventions were optimized to improve students’ outcomes for up to one-year after end-of-treatment. On the other hand, many proposed benefits of early math instruction relate to children’s longer-term outcomes. We find merit in both of these arguments and do not attempt a thorough critique of either of them here but see Bailey et al. (2020) and commentary by Schneider and Bradford (2020) for discussion of both views.

2 We would like to thank an anonymous reviewer for their suggestions for how to word this section, which substantially improved its clarity.

3 We thank an anonymous reviewer for the idea to pursue this as a future direction.

Additional information

Funding

The Number Knowledge Tutoring research was supported by [2 R01 HD053714 and Core Grant [U54HD083211] from the Eunice Kennedy Shriver National Institute of Child Health and Human Development to Lynn S. Fuchs at Vanderbilt University. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Eunice Kennedy Shriver National Institute of Child Health and Human Development or the National Institutes of Health. The Pre-K Mathematics research was supported by the Institute of Education Sciences, U.S. Department of Education through Grant [R305K050004] to Alice Klein and Prentice Starkey at WestEd. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education. Drew Bailey is funded by a Jacobs Foundation Fellowship. Daniela Alvarez-Vargas is supported by the National Science Foundation Graduate Research Fellowship under Grant [No. DGE-1839285]. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.