Explaining Variation in Findings From Efficacy and Effectiveness Studies for English Reading Interventions for English Learners

Abstract

Beginning in 2002, researchers developed, implemented, and evaluated the efficacy of an English reading intervention for first-grade English learners using multiple randomized control trials (RCTs). As a result of this efficacy work, researchers successfully competed for an IES Goal 4 effectiveness study using the same intervention. Unlike the efficacy studies, the findings from the effectiveness study begun in 2011 did not yield significant differences in favor of the treatment. We investigated the identification and pre-intervention performance levels of students in efficacy and effectiveness RCT studies and provide data and compare samples from these studies as one means of explaining variation in findings. Findings indicate that pre-intervention performance levels are higher for students in effectiveness RCT studies. This article serves as an exemplar for other effectiveness studies that may not replicate findings from previous efficacy trials.

Keywords:

• program evaluation
• equating
• effectiveness study

Acknowledgments

This research was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305A110297 to the University of Houston. The opinions expressed are those of the authors and do not represent the views of the Institute or the U.S. Department of Education.

Notes

1 Effectiveness study power, based on median effect sizes from the efficacy study, was β = .87 under the most conservative assumptions.

2 W scores are converted raw scores on an equal interval scale.