The Mathematical Quality of Instruction (MQI) in Kindergarten: An Evaluation of the Stability of the MQI Using Generalizability Theory

ABSTRACT

Research Findings: We evaluated the score stability of the Mathematical Quality of Instruction (MQI), an observational measure of mathematics instruction. Three raters each scored, independently, 100 video-recorded lessons taught by 20 kindergarten teachers in the spring. Using generalizability theory analyses, we decomposed the MQI’s score stability into potential sources of variation (teachers, lessons, raters, and their interactions). The 13-item (3-domain) Ambitious Mathematics Instruction scale and the Whole Lesson scale each explained about one third of the variance attributed to differences in the main construct of interest (teachers’ instructional strategies). The MQI’s Errors and Imprecision scale was not relevant at the kindergarten level; there were virtually no errors and/or ambiguities observed across the 100 mathematics lessons. In a series of decision studies, we examined improvements in reliability with combinations of up to 6 raters and 8 lessons. Only the Richness of Mathematics domain scores and the Whole Lesson scores achieved acceptable reliabilities. Practice or Policy: The findings have important implications for the use of observation measures to document teachers’ mathematics practices in the early years of school.

Acknowledgments

We greatly appreciate the work of the three Mathematical Quality of Instruction raters as well as the involvement of the kindergarten teachers and their students.

Additional information

Funding

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant No. R305A140664 to Helen Patrick, Panayota Mantzicopoulos, and Brian F. French. The opinions expressed are our own and do not represent views of the Institute or the U.S. Department of Education.