A Systematic Review on Quality Indicators of Randomized Control Trial Reading Fluency Intervention Studies

Abstract

Although several studies have examined the effectiveness of reading fluency interventions, the methodological rigor of these studies remains unclear. The purpose of this study was to examine the quality of randomized controlled trial (RCT) reading fluency intervention studies using the Council for Exceptional Children (CEC) standards. Twenty-six studies met the inclusion criteria for the review. All included studies met the four quality indicators for context and setting, participants, intervention agent, and data analysis. In addition, 22 out of the 26 studies met the quality indicators for internal validity and implementation fidelity, and 18 met the quality indicator for outcome measure. Finally, 16 of the studies met all eight quality indicators, and the majority of them also reported significant effects on reading fluency outcomes. Findings and future directions are discussed considering the quality indicators for methodologically rigorous reading fluency intervention studies.

Notes

1 Gersten et al. (2005) listed ten essential and eight desirable QIs. Their essential QIs were further grouped into four categories: QIs for describing participants, QIs for implementation of the intervention and description of comparison conditions, QIs for outcome measures, and QIs for data analysis. Horner et al. (2005) identified seven QIs (description of participants and settings, dependent variable, independent variable, baseline, experimental control/internal validity, external validity, and social validity) with 21 subcomponents.

2 Notice that none of the RCT studies included in our review were examined by Chard et al. (2009) or O'Keeffe et al. (2012) because their RCT studies were published before 2000, which was used here as the beginning year of our search (see Methods section).

3 Book chapters and dissertations were excluded from our review as we could not check whether they went through a rigorous peer review process. For the same reason, we did not include any unpublished studies.

4 We selected the year 2000 as our starting point for two reasons: first, it gives us a reasonable time window (19 years) to identify enough studies for the purpose of this methodological analysis, and second, it seems to be the starting point of most recently published meta-analyses and systematic reviews.

5 We considered using the methodological design steps proposed by What Works Clearinghouse (WWC, 2020) to rate GDS. However, 10 of the 26 studies included in this review did not provide sufficient information regarding the reliability of the outcome measure or attrition and differential attrition levels to adequately receive one of the three rankings from WWC (i.e., meets standards without reservations, meets standards with reservations, and does not meets standards). To avoid the exclusion or the potentially inappropriate rating of these studies in our review, we decided to only use the more detailed list of 23 indicators for GDS proposed by the CEC (2014).

6 In the absence of explicit information, a QI can still be met if the missing information does not threaten the validity of the study (Cook et al., 2015). For example, some instructional procedures and materials are commonly understood and do not require an explicit description.

7 We excluded QI 7.4 (requires information on frequency and timing of outcome measures for single-case and alternating treatment designs) as it was not relevant for RCTs.

8 According to Cook et al. (2015), the QI for validity (QI 7.6) is met if reviewers consider that the outcome measure accurately represents the construct (i.e., content validity).

9 Most studies included in this review measured WRCM to assess reading fluency intervention gains.

10 Three studies met all CEC (2014) QIs but reported non-significant treatment effects on reading fluency: Patton et al. (2010), Vadasy and Sanders (2008a), and Wexler et al. (2010). Patton et al. attributed the results to the fact that most children in their sample were too young (i.e., Grade 1) to benefit from a fluency training program. In the study by Vadasy and Sanders, the fluency program had significant effects on other reading skills such as passage comprehension and vocabulary; however, no improvement was observed in WCPM. The authors concluded that deficient word reading skills might have prevented students from developing fluency. Finally, Wexler et al. argued that lack of treatment effects was due to the participants’ severe reading difficulties (i.e., below the 5th percentile).

Additional information

Funding

This work was supported by a grant from the Social Sciences and Humanities Research Council of Canada to the second author (RES0029061).