The Mathematical Quality of Instruction (MQI) in Kindergarten: An Evaluation of the Stability of the MQI Using Generalizability Theory

References

• Agodini, R., & Harris, B. (2016). How teacher and classroom characteristics moderate the effects of four elementary math curricula. The Elementary School Journal, 117, 216–236. doi:10.1086/688927
   Web of Science ®Google Scholar
• American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
   Google Scholar
• Aunola, K., Leskinen, E., Lerkkanen, M., & Nurmi, J. (2004). Developmental dynamics of math performance from preschool to grade 2. Journal of Educational Psychology, 96, 699–713. doi:10.1037/0022-0663.96.4.699
   Web of Science ®Google Scholar
• Ball, D. L., Hill, H. C., & Bass, H. (2005). Knowing mathematics for teaching: Who knows mathematics well enough to teach third grade, and how can we decide? American Educator, 29(1), 20–22, 43–46
   Google Scholar
• Bell, C. A., Qi, Y., Croft, A. J., Leusner, D., McCaffrey, D. F., Gitomer, D. H., & Pianta, R. C. (2014). Improving observational score quality: Challenges in observer thinking. In T. J. Kane, K. A. Kerr, & R. C. Pianta (Eds.), Designing teacher evaluation systems (pp. 50–97). San Francisco, CA: Jossey-Bass.
   Google Scholar
• Berry, III, R. Q., Rimm-Kaufman, S. E., Ottmar, E. M., Walkowiak, T. A., & Merritt, E. (2010). The mathematics scan (M-Scan): A measure of mathematics instructional quality. Unpublished measure, University of Virginia, Charlottesville, VA.
   Google Scholar
• Bill and Melinda Gates Foundation. (2013). Measures of effective teaching. Retrieved from http://www.metproject.org/
   Google Scholar
• Blazar, D. (2015). Effective teaching in elementary mathematics: Identifying classroom practices that support student achievement. Economics of Education Review, 48, 16–29. doi:10.1016/j.econedurev.2015.05.005
   Web of Science ®Google Scholar
• Blazar, D., Braslow, D., Charalambous, C. Y., & Hill, H. C. (2017). Attending to general and content-specific dimensions of teaching: Exploring factors across two observation instruments. Educational Assessment, 22, 71–94. doi:10.1080/10627197.2017.1309274
   Web of Science ®Google Scholar
• Blazar, D., & Kraft, M. A. (2017). Teacher and teaching effects on students’ attitudes and behaviors. Educational Evaluation and Policy Analysis, 39, 146–170. doi:10.3102/0162373716670260
   PubMed Web of Science ®Google Scholar
• Bottia, M. C., Moller, S., Mickelson, R. A., & Stearns, E. (2014). Foundations of mathematics achievement: Instructional practices and diverse kindergarten students. The Elementary School Journal, 115, 124–150. doi:10.1086/676950
   Web of Science ®Google Scholar
• Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
   Google Scholar
• Brennan, R. L. (2011). Using generalizability theory to address reliability issues for PARCC assessments: A white paper. In Center for Advanced Studies in Measurement and Assessment (CASMA), 1–22. Iowa City, IA: University of Iowa.
   Google Scholar
• Center for Education Policy Research. (n.d.a). Mathematical Quality of Instruction (MQI). Retrieved from https://cepr.harvard.edu/mqi-access
   Google Scholar
• Center for Education Policy Research. (n.d.b). Whole lesson codes [Unpublished training module for the Mathematics Quality of Instruction]. Retrieved from https://cepr.harvard.edu/mqi-access
   Google Scholar
• Charalambous, C. Y., & Praetorious, A. (2018). Studying mathematics instruction through different lenses: Setting the ground for understanding instructional quality more comprehensively. ZDM: Mathematics Education. Advance online publication. doi:10.1007/s11858-018-0914-8
   Google Scholar
• Clements, D. H. (2004). Part one: Major themes and recommendations. In D. H. Clements, J. Sarama, & A. M. DiBiase (Eds.), Engaging young children in mathematics: Standards for early childhood mathematics education (pp. 1–72). Mahwah, NJ: Erlbaum.
   Google Scholar
• Clements, D. H., Sarama, J., Spitler, M. E., Lange, A. A., & Wolfe, C. B. (2011). Mathematics learned by young children in an intervention based on learning trajectories: A large-scale cluster randomized trial. Journal of Mathematics Education, 42, 127–166. doi:10.5951/jresematheduc.42.2.0127
   Web of Science ®Google Scholar
• Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Philadelphia, PA: Harcourt.
   Google Scholar
• Cronbach, L. J., Gleser, G., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York, NY: Wiley.
   Google Scholar
• Curby, T. W., Grimm, K. J., & Pianta, R. (2010). Stability and change in early childhood classroom interactions during the first two hours of a day. Early Childhood Research Quarterly, 25, 373–384. doi:10.1016/j.ecresq.2010.02.004
   Web of Science ®Google Scholar
• Curby, T. W., Stuhlman, M., Grimm, K., Mashburn, A., Chomat-Mooney, L., Downer, J., & Pianta, R. C. (2011). Within-day variability in the quality of classroom interactions during third and fifth grade: Implications for children’s experiences and conducting classroom observations. Elementary School Journal, 112, 16–37. doi:10.1086/660682
   Web of Science ®Google Scholar
• Danielson, C. (2013). The Framework for Teaching evaluation instrument. Retrieved from http://www.teachscape.com/frameworkforteaching/home
   Google Scholar
• Every Student Succeeds Act of 2015, Pub. L. No. 114-95 § 114 Stat. 1177 (2015–2016). Washington, DC: U.S. Government.
   Google Scholar
• Grimm, K. J., Curby, T. W., Pianta, R. C., Mashburn, A. J., Downer, J., Chomat-Mooney, L., & Hamre, B. (2008, March). Partitioning variance associated with classroom observations. Paper presented at the annual meeting of the American Educational Research Association, New York, NY.
   Google Scholar
• Guarino, C., Dieterle, S. G., Bargagliotti, A. D., & Mason, W. M. (2013). What can we learn about effective early mathematics teaching? A framework for estimating causal effects using longitudinal survey data. Journal of Research on Educational Effectiveness, 6, 164–198. doi:10.1080/19345747.2012.706695
   Web of Science ®Google Scholar
• Hatchey, A. C. (2013). The early childhood mathematics education revolution. Early Education & Development, 24, 419–430. doi:10.1080/10409289.2012.756223
   Web of Science ®Google Scholar
• Hill, H. (2011). Mathematical Quality of Instruction. Retrieved from https://cepr.harvard.edu/ncte-conference-2011
   Google Scholar
• Hill, H. (2014). Mathematical Quality of Instruction (MQI: 4-point version). Retrieved from http://isites.harvard.edu/icb/icb.do?keyword=mqi_training
   Google Scholar
• Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41, 56–64. doi:10.3102/0013189X12437203
   Web of Science ®Google Scholar
• Ho, A. D., & Kane, T. J. (2013). The reliability of classroom observations by school personnel. Retrieved from http://k12education.gatesfoundation.org/resource/the-reliability-of-classroom-observations-by-school-personnel/
   Google Scholar
• Horizon Research. (2003). Looking inside the classroom: A study of K-12 mathematics and science education in the United States. Retrieved from http://www.horizon-research.com/insidetheclassroom/reports/looking/complete.pdf
   Google Scholar
• Kane, T. J., McCaffrey, D. F., Miller, T., & Staiger, D. O. (2013). Have we identified effective teachers? Validating measures of effective teaching using random assignment. Seattle, WA: Bill and Melinda Gates Foundation.
   Google Scholar
• Kane, T. J., & Staiger, D. O. (2012). Gathering feedback for teaching. Retrieved from https://files.eric.ed.gov/fulltext/ED540960.pdf
   Google Scholar
• Kilday, C. R., & Kinzie, M. B. (2009). An analysis of instruments that measure the quality of mathematics teaching in early childhood. Early Childhood Education Journal, 36, 365–372. doi:10.1007/s10643-008-0286-8
   Google Scholar
• Lakes, D. K., & Hoyt, W. T. (2010). Applications of generalizability theory to clinical child and adolescent psychology research. Journal of Clinical & Adolescent Psychology, 38, 144–165. doi:10.1080/15374410802575461
   Web of Science ®Google Scholar
• Learning Mathematics for Teaching Project. (2011). Measuring the Mathematical Quality of Instruction. Journal of Mathematics Teacher Education, 14, 25–47. doi:10.1007/s10857-010-9140-1
   Google Scholar
• Mashburn, A. J., Meyer, J. P., Allen, J. P., & Pianta, R. C. (2014). The effect of observation length and presentation order on the reliability and validity of an observational measure of teaching quality. Educational and Psychological Measurement, 74, 400–422. doi:10.1177/0013164413515882
   Web of Science ®Google Scholar
• McGuire, P. R., Kinzie, M., Thunder, K., & Berry, R. (2016). Methods of analysis and overall mathematics teaching quality in at-risk prekindergarten classrooms. Early Education & Development, 27, 89–109. doi:10.1080/10409289.2015.968241
   Web of Science ®Google Scholar
• National Council on Teacher Quality. (2015). State of the states 2015: Evaluating teaching, leading, and learning. Retrieved from http://www.nctq.org/dmsStage/StateofStates2015
   Google Scholar
• National Mathematics Advisory Panel. (2008). Final report of the National Mathematics Advisory Panel. Washington, DC: U.S. Department of Education.
   Google Scholar
• National Research Council. (2001). Adding it up: Helping children learn mathematics. Washington, DC: National Academies Press.
   Google Scholar
• Nunnally, J. C., & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
   Google Scholar
• Patrick, H., & Mantzicopoulos, P. (2016). Is effective teaching stable? Journal of Experimental Education, 84, 23–47. doi:10.1080/00220973.2014.952398
   Web of Science ®Google Scholar
• Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008). Classroom Assessment Scoring System manual K-3. Baltimore, MA: Brookes Publishing.
   Google Scholar
• Piburn, M., Sawada, D., Turley, J., Falconer, K., Benford, R., Bloom, I., & Judson, E. (2000). Reformed Teaching Observation Protocol (RTOP): Reference manual (ACEPT Tech. Rep. No. IN003). Retrieved from http://files.eric.ed.gov/fulltext/ED447205.pdf
   Google Scholar
• Praetorius, A., Pauli, C., Reusser, K., Rakoczy, K., & Klieme, E. (2014). One lesson is all you need? Stability of instructional quality across lessons. Learning and Instruction, 31, 2–12. doi:10.1016/j.learninstruc.2013.12.002
   Web of Science ®Google Scholar
• Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. London, UK: Sage.
   Google Scholar
• Swiss Society for Research in Education Working Group. (2006). EduG user guide. Neuchatel, Switzerland: Institut de recherche et de documentation pédagogique (IRDP).
   Google Scholar
• U.S. Department of Education. (2011). Fact sheet: Bringing flexibility and focus to education law. Retrieved from http://www.whitehouse.gov/sites/default/files/fact_sheet_bringing_flexibility_and_focus_to_education_law_0.pdf
   Google Scholar
• Walkington, C., Arora, P., Ihorn, S., Gordon, J., Walker, M., Abraham, L., & Marder, M. (2012). Development of the UTeach Observation Protocol: A classroom observation instrument to evaluate mathematics and science teachers from the UTeach preparation program. Retrieved from http://cwalkington.com/UTOP_Paper_2011.pdf
   Google Scholar
• Watts, T. W., Duncan, G. J., Clements, D. H., & Sarama, J. (2018). What is the long-run impact of mathematics during preschool? Child Development, 89, 539–555.
   PubMed Web of Science ®Google Scholar
• Whitehurst, G. J., Chingos, M. M., & Lindquist, K. (2014). Evaluating teachers with classroom observations. Lessons learned in four districts. Retrieved from http://www.brookings.edu/~/media/research/files/reports/2014/05/13-teacher-evaluation/evaluating-teachers-with-classroom-observations.pdf
   Google Scholar