Assessing Fidelity of Implementation of an Unprescribed, Diagnostic Mathematics Intervention

Abstract

This article draws on previously employed methods for conducting fidelity studies and applies them to an evaluation of an unprescribed intervention. We document the process of assessing the fidelity of implementation of the Math Recovery first-grade tutoring program, an unprescribed, diagnostic intervention. We describe how we drew on recent conceptions and standards of fidelity assessment to evaluate the implementation of the program, detailing the process of developing and testing instruments for assessing implementation fidelity, including (a) identifying the intervention's program theory and core components; (b) creating operational definitions of the intervention's core components; (c) developing coding instruments; (d) selecting and training coders; (e) instituting a sampling frame sufficient for generalizing fidelity findings to the study population; and (f) determining the reliability and validity of the data. We also provide a brief overview of the results of the fidelity assessment. We conclude by discussing the aspects of our work that have implications for assessments of fidelity of implementation of unprescribed interventions more generally.

Keywords:

• Fidelity of implementation
• program evaluation
• unprescribed interventions

ACKNOWLEDGMENTS

The opinions expressed are those of the authors and do not necessarily represent the views of the U.S. Department of Education. The authors thank coders Angela Hacker, Cheryl Henderson, Michelle Mountes, Rena Malkofsky-Berger, and Marianne Reale; project manager Kelly Baird; and assistant James Nardella for their contributions to this work.

Notes

For a description of psychometric properties and other details related to the MR Proximal, we refer the reader to Smith et al. (2013).

The acts of direct demonstration and “behavior eliciting” are prohibited because they are viewed by program developers as counterproductive to supporting students in developing autonomy with respect to sense making and reasoning. “Do not demonstrate a method for solving a problem” and “Do not interrupt the student's thinking,” then, are examples of negatively phrased instructions that help define and rationalize, for tutors, the more positively phrased instructions that might accompany those prohibitions: “Instead, invite them to generate their own strategy” and “Instead, provide sufficient wait-time,” respectively.

To determine the extent to which coders differed systematically in applying the coding scheme, analyses of variance were conducted for each FOI variable by coder (limited to only students whose video data had not been double-coded for reliability purposes). Only one omnibus F test indicated a significant difference in fidelity data by coder—that of the variable concerning whether tutors committed major errors on the initial assessment. The results of Tukey's wholly significant difference post hoc comparison of means suggested that coding of this variable by both Coders 2 and 4 differed significantly from that of Coders 1 and 5. However, the Tukey–Kramer method for pairwise comparisons, which is preferred in cases such as this where group sizes differ, revealed no differences at the p = .05 level. Nonetheless, we controlled for possible differences in coders’ application of this particular code in analyses pertaining to the initial assessment.

Other phases of our analysis focused on FOI of the initial assessment, but, with the exception of profile percentage, we did not examine the relationship between those indicators and study outcomes, as there was no theoretical link between them.

To avoid overfitting, we tested the nine instructional session variables through a progression of models, beginning with structure- and moving to process-related variables (finishing with positive infidelity), keeping those with significant relationships to outcomes along the way.