A Formative Assessment Observation Protocol to Measure Implementation: Evaluating the Scoring Inference

References

• Andrade, H. L., & Brookhart, S. M. (2016). The role of classroom assessment in supporting self-regulated learning. In D. Laveault & L. Allal (Eds.), Assessment for learning: Meeting the challenge of implementation (pp. 293–310). Switzerland: Springer International Publishing.
   Google Scholar
• Andrade, H. L., Du, Y., & Mycek, K. (2010). Rubric‐referenced self‐assessment and middle school students’ writing. Assessment in Education: Principles, Policy & Practice, 17(2), 199–214.
   Google Scholar
• Andrade, H. L., Du, Y., & Wang, X. (2008). Putting rubrics to the test: The effect of a model, criteria generation, and rubric‐referenced self‐assessment on elementary school students’ writing. Educational Measurement: Issues and Practice, 27(2), 3–13. doi:10.1111/j.1745-3992.2008.00118.x
   Google Scholar
• Andrade, H. L., & Valtcheva, A. (2009). Promoting learning and achievement through self-assessment. Theory Into Practice, 48(1), 12–19. doi:10.1080/00405840802577544
   Web of Science ®Google Scholar
• Association of Mathematics Teacher Educators (AMTE). (2017). Standards for Preparing Teachers of Mathematics. In: Available online at amte.net/standards
   Google Scholar
• Bailey, A., & Heritage, M. (2018). Self-regulation in learning and the role of language and formative assessment. Cambridge, MA: Harvard Education Press.
   Google Scholar
• Ball, D. L. (1993). With an eye on the mathematical horizon: Dilemmas of teaching elementary school mathematics. The Elementary School Journal, 93(4), 373–397. doi:10.2307/1002018
   Web of Science ®Google Scholar
• Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., & Qi, Y. (2012). An argument approach to observation protocol validity. Educational Assessment, 17(2–3), 62–87.
   Google Scholar
• Bennett, R. E. (2010). Cognitively based assessment of, for, and as learning (CBAL): A preliminary theory of action for summative and formative assessment. Measurement, 8(2–3), 70–91.
   Google Scholar
• Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education: Principles, Policy & Practice, 18(1), 5–25.
   Google Scholar
• Bennett, R. E. (2013). Formative assessment: Process and tool. Paper presented at the National Conference on Student Assessment, National Harbor, MD.
   Google Scholar
• Bennett, R. E. (2014). Preparing for the future: What educational assessment must do. Teachers College Record, 116(11).
   Web of Science ®Google Scholar
• Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability (Formerly: Journal of Personnel Evaluation in Education), 21(1), 5–31.
   Web of Science ®Google Scholar
• Blazar, D., Braslow, D., Charalambous, C. Y., & Hill, H. C. (2017). Attending to general and mathematics-specific dimensions of teaching: Exploring factors across two observation instruments. Educational Assessment, 22(2), 71–94.
   Web of Science ®Google Scholar
• Boud, D., & Molloy, E. (2013). Rethinking models of feedback for learning: The challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698–712.
   Web of Science ®Google Scholar
• Bourgeois, L. (2016). Supporting students’ learning: From teacher regulation to co-regulation. In D. L. L. Allal (Ed.), Assessment for learning: Meeting the challenge of implementation (pp. 345–363). Switzerland: Springer International Publishing.
   Google Scholar
• Brookhart, S. M., Andolina, M., Zuza, M., & Furman, R. (2004). Minute math: An action research study of student self-assessment. Educational Studies in Mathematics, 57(2), 213–227.
   Google Scholar
• Brookhart, S. M., & Moss, C. (2009). Advancing formative assessment in every classroom. Alexandria, VA: ASCD.
   Google Scholar
• Carpenter, T. P., Fennema, E., Petereson, P. L., Chiang, C.-P., & Loef, M. (1989). Using knowledge of children’s mathematics thinking in classroom teaching: An experimental study. American Educational Research Journal, 26(4), 499–531.
   Web of Science ®Google Scholar
• Cobb, J. A. (1972). Relationship of discrete classroom behaviors to fourth-grade academic achievement. Journal of Education Psychology, 68(1), 74–80.
   Google Scholar
• Coffey, J. E., Hammer, D., Levin, D. M., & Grant, T. (2011). The missing disciplinary substance of formative assessment. Journal of Research in Science Teaching, 48(10), 1109–1136.
   Web of Science ®Google Scholar
• Cohen, D., Raudenbush, S. W., & Ball, D. L. (2003). Resources, instruction, and research. Educational Evaluation and Policy Analysis, 25(2), 119–142.
   Web of Science ®Google Scholar
• Council of Chief State School Officers (CCSSO). (2018). Revising the Definition of Formative Assessment. Created by E. C. Wylie, ETS, for the Formative Assessment for Students and Teachers (FAST) collaborative. Washington, DC: CCSSO.
   Google Scholar
• Cronbach, L. J. (1988). Five perspectives on validity argument. In H. Wainer & H. I. Braun (Eds.), Test validity (pp. 3–17). Hillsdale, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Crooks, T. J. (1988). The impact of classroom evaluation practices on students. Review of Educational Research, 58(4), 438–481.
   Web of Science ®Google Scholar
• Danielson, C. (2007). Enhancing professional practice: A framework for teaching. Alexandria, VA: ASCD.
   Google Scholar
• Deane, P. (2011). Writing assessment and cognition (RR-11-14). Retrieved from Princeton, NJ: http://www.ets.org/Media/Research/pdf/RR-11-14.pdf
   Google Scholar
• Drost, B. (2017). Digital Formative Assessment Tools to Improve Motivation. Paper presented at the Presentation at the National Council on Measurement in Education (NCME) Special Conference on Classroom Assessment and Large-Scale Psychometrics, Lawrence, KS..
   Google Scholar
• Emerson, R. M., Fretz, R. I., & Shaw, L. L. (1995). Writing ethnographic fieldnotes. Chicago, IL: University of Chicago Press.
   Google Scholar
• Ertmer, P. A., & Ottenbreit-Leftwich, F. T. (2010). Teacher technology change: How knowledge, confidence, beliefs, and culture intersect. Journal of Research on Technology in Education, 3(42), 255–284.
   Google Scholar
• Everson, H. T., & Tobias, S. (1998). The ability to estimate knowledge and performance in college: A metacognitive analysis. Instructional Science, 26(1–2), 65–79.
   Web of Science ®Google Scholar
• Fernandes, M., & Fontana, D. (1996). Changes in control beliefs in Portuguese primary school pupils as a consequence of the employment of self‐assessment strategies. British Journal of Educational Psychology, 66(3), 301–313. doi:10.1111/j.2044-8279.1996.tb01199.x
   PubMedGoogle Scholar
• Finn, J. D., Pannozzo, G. M., & Voelkl, K. E. (1995). Disruptive and inattentive-withdrawn behavior and achievement among fourth graders. The Elementary School Journal, 95(5), 421–434.
   Web of Science ®Google Scholar
• Finn, J. D., & Rock, D. (1997). Academic success among students at risk for school failure. Journal of Applied Psychology, 82(2), 221–234.
   PubMed Web of Science ®Google Scholar
• Firestone, W. A. (1987). Meaning in method: The rhetoric of quantitative and qualitative research. Educational Researcher, 16(7), 16–21.
   Google Scholar
• Franke, M. L., & Kazemi, E. (2001). Learning to teach mathematics: Focus on student thinking. Theory Into Practice, 40(2), 102–109. doi:10.2307/1477271
   Web of Science ®Google Scholar
• Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415.
   PubMed Web of Science ®Google Scholar
• Fuchs, L. S., Hamlett, D. F. C. L., & Stecker, P. M. (1991). Effects of curriculum-based measurement and consultation on teacher planning and student achievement in mathematics operations. American Educational Research Journal, 28(3), 617–641.
   Web of Science ®Google Scholar
• Fuson, K. C., Kalchman, M., & Bransford, J. D. (2005). Mathematical understanding: An introduction. In M. S. Donovan & J. D. Bransford (Eds.), How Students Learn: History, Mathematics and Science in the Classroom (pp. 217–256). Washington, DC: Division of Behavioral and Social Sciences and Education, The National Academies Press.
   Google Scholar
• Graf, E. A. (2009). Defining mathematics competency in the service of cognitively based assessment for grades 6 through 8. (RR-09-42). Princeton, NJ: Educational Testing Service.
   Google Scholar
• Graham, M., Milanowski, A., & Miller, J. (2012). Measuring and Promoting Inter-Rater Agreement of Teacher and Principal Performance Ratings. Online Submission.
   Google Scholar
• Graham, S., McKeown, D., Kiuhara, S., & Harris, K. R. (2012). Meta-analysis of writing instruction for students in elementary grades. Journal of Educational Psychology, 104, 879–896.
   Web of Science ®Google Scholar
• Hartmann, D. P. (1977). Considerations in the choice of interobserver reliability estimates. Journal of Applied Behavior Analysis, 10(1), 103–116.
   PubMed Web of Science ®Google Scholar
• Hays, D. G., & Singh, A. A. (2012). Qualitative Inquiry in Clinical and Educational Settings. New York, NY: Guilford.
   Google Scholar
• Heritage, M., Kim, J., Vendlinski, T., & Herman, J. L. (2009). From evidence to action: A seamless process in formative assessment? Educational Measurement: Issues and Practice, 28(3), 24–31.
   Google Scholar
• Herman, J. L., & Choi, K. (2008). Formative assessment and the improvement of middle school science learning: The role of teacher accuracy (740). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing.
   Google Scholar
• Herold, B. (2013). Teacher-observation tools get easier for districts to purchase. Education Week. Retrieved from http://blogs.edweek.org/edweek/teacherbeat/2013/08/teacher_observation_tools_get_.html
   Google Scholar
• Hill, H. C., Charalambous, C. Y., Blazat, D., McGinn, D., Kraft, M., Beisiegel, M., … Lynch, K. (2012). Validating arguments for observational instruments: Attending to multiple sources of variation. Educational Assessment, 17, 88–106. doi:10.1080/10627197.2012.715019
   Google Scholar
• Hill, H. C., Charalambous, C. Y., & Kraft, M. A. (2012). When rater reliability is not enough: Teacher observation systems and a case for the generalizability study. Educational Researcher, 41(2), 56–64.
   Web of Science ®Google Scholar
• Hill, H. C., & Grossman, P. (2013). Learning from teacher observations: Challenges and opportunities posed by new teacher evaluation systems. Harvard Educational Review, 83(2), 371–384.
   Web of Science ®Google Scholar
• Horizon Research  Inc. (2000). Inside the classroom: Observation and analytic protocol Retrieved from Chapel Hill, NC: HRI.
   Google Scholar
• Jones, E. D., & Krouse, J. R. (1988). The effectiveness of data-based instruction by student teachers in classrooms for pupils with mild learning handicaps. Teacher Education and Special Education: The Journal of the Teacher Education Division of the Council for Exceptional Children, 11(1), 9–19.
   Google Scholar
• Jonsson, A., Lundahl, C., & Holmgren, A. (2015). Evaluating a large-scale implementation of Assessment for Learning in Sweden. Assessment in Education: Principles, Policy & Practice, 22(1), 104–121.
   Web of Science ®Google Scholar
• Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). Westport, CT: American Council on Education/Praeger.
   Google Scholar
• Kane, T. J., & Staiger, D. O. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Policy and Practice Brief. MET Project. Seattle, WA: Bill & Melinda Gates Foundation.
   Google Scholar
• Kanter, D. E., & Konstantopolous, S. (2010). The impact of a project-based science curriculum on minority student achievement, attitudes and careers: The effects of teacher content and pedagogical content knowledge and inquiry-based practices. Science Education, 94(5), 855–887. doi:10.1002/sce.20391
   Web of Science ®Google Scholar
• King, A. (1992). Facilitating elaborative learning through guided student-generated questioning. Educational Psychologist, 27(1), 111–126.
   Web of Science ®Google Scholar
• Kingston, N., & Nash, B. (2011). Formative assessment: A meta‐analysis and a call for research. Educational Measurement: Issues and Practice, 30(4), 28–37.
   Google Scholar
• Lahaderne, H. M. (1968). Attitudinal and intellectual correlates of attention: A study of four sixth-grade classrooms. Journal of Educational Psychology, 59(5), 320–324.
   PubMed Web of Science ®Google Scholar
• Leahy, S., Lyon, C. J., Thompson, M., & Wiliam, D. (2005). Classroom assessment: Minute by minute, day by day. Educational Assessment, 63(3), 19–24.
   Google Scholar
• Lin, X., & Lehman, J. D. (1999). Supporting learning of variable control in a computer-based biology environment: Effects of prompting college students to reflect on their own thinking. Journal of Research in Science Teaching, 36(7), 837–858.
   Web of Science ®Google Scholar
• Liu, L., Rogat, A., & Bertling, M. (2013). A CBAL science model of cognition: DA CBALTM Science Model of Cognition: Developing a Competency Model and Learning Progressions to Support Assessment Development (RR-13-29). Princeton, NJ: Educational Testing Service.
   Google Scholar
• Luxton-Reilly, A. (2009). A systematic review of tools that support peer assessment. Computer Science Education, 19(4), 209–232.
   Google Scholar
• Lyon, C. J., Brenneman, M. W., Nabors Oláh, L., & Brown, C. (2013). Developing a formative assessment protocol to measure implementation Paper presented at the Annual meeting of the American Educational Research Association, San Francisco, CA.
   Google Scholar
• Lyon, C. J., Leusner, D. M., & Mullholland, M. (2012). The evaluation of the implementation of CBAL formative assessment materials: Final report. Final report for CBAL Allocation Initiative. ETS. Princeton, NJ
   Google Scholar
• Lyon, C. J., Wylie, E. C., Brockway, D., & Mavronikolas, E. (2018). Formative assessment and the role of teachers’ content area. School Science and Mathematics, 118(5), 144–155. doi:10.1111/ssm.12277
   Web of Science ®Google Scholar
• McClellan, C. (2013). What it looks like master coding videos for observer training and assessment. Policy and Practice Brief. MET Project. Bill and Melinda Gates Foundation, Washington, DC.
   Google Scholar
• McDonald, B., & Boud, D. (2003). The impact of self-assessment on achievement: The effects of self-assessment training on performance in external examinations. Assessment in Education: Principles, Policy & Practice, 10(2), 209–220.
   Google Scholar
• McLuckie, J., & Topping, K. J. (2004). Transferable skills for online peer learning. Assessment & Evaluation in Higher Education, 29(5), 563–584.
   Google Scholar
• McMillan, J., & Hearn, J. (2008). Student self-assessment: The key to strong student motivation and higher achievement. Educational Horizons, 87(1), 40–49.
   Google Scholar
• Mercer, N., Dawes, L., Wegerif, R., & Sams, C. (2004). Reasoning as a scientist: Ways of helping children to use language to learn science. British Educational Research Journal, 30(3), 359–377.
   Web of Science ®Google Scholar
• Moss, C. M., & Brookhart, S. M. (2012). Learning targets: Helping students aim for understanding in today’s lesson. Alexandria, VA: ASCD.
   Google Scholar
• Munns, G., & Woodward, H. (2006). Student engagement and student self‐assessment: The REAL framework. Assessment in Education: Principles, Policy & Practice, 13(2), 193–213. doi:10.1080/09695940600703969
   Google Scholar
• Nabors Oláh, L., Lawrence, N. R., & Riggan, M. (2010). Learning to learn from benchmark assessment data: How teachers analyze results. Peabody Journal of Education, 85(2), 226–245.
   Google Scholar
• National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. Reston, VA: NCTM.
   Google Scholar
• National Research Council. (2001). Classroom assessment and the National Science Education Standards. Washington, DC: National Academy Press.
   Google Scholar
• O’Leary, M., Lysaght, Z., & Ludlow, L. (2013). A measurement instrument to evaluate teachers’ assessment for learning classroom practices. International Journal of Educational and Psychological Assessment, 14, 40–60.
   Google Scholar
• Penuel, W. R., & Shepard, L. A. (2016). Assessment and teaching. In D. Gitomer & C. Bell (Eds.), Handbook of research on teaching (pp. 787–850). Washington, DC: AERA.
   Google Scholar
• Pianta, R. C., LaParo, K. M., & Hamre, B. K. (2008). Classroom Assessment Scoring System manual K-3. Baltimore, MD: Paul H. Brookes.
   Google Scholar
• Popham, W. J. (2008). Transformative assessment. Alexandria, VA: Association for Supervision and Curriculum Development..
   Google Scholar
• Praetorius, A., Pauli, C., Reusser, K., Rakoczy, K., & Klieme, E. (2014). One lesson is all you need? Stability of instructional quality across lessons. Learning and Instruction, 31, 2–12.
   Web of Science ®Google Scholar
• Riggan, M., & Oláh, L. N. (2011). Locating interim assessments within teachers’ assessment practice. Educational Assessment, 16(1), 1–14. doi:10.1080/10627197.2011.551085
   Google Scholar
• Schell, J., Lukoff, B., & Mazur, E. (2013). Catalyzing learner engagement using cutting-edge classroom response systems in higher education. In Increasing student engagement and retention using classroom technologies: Classroom response systems and mediated discourse technologies (pp. 233–261). United Kingdom: Emerald Group Publishing Limited.
   Google Scholar
• Shavelson, R. J., Young, D. B., Ayala, C. C., Brandon, P. R., Furtak, E. M., Ruiz-Primo, M. A., … Yin, Y. (2008). On the impact of curriculum-embedded formative assessment on learning: A collaboration between curriculum and assessment developers. Applied Measurement in Education, 21(4), 295–314.
   Web of Science ®Google Scholar
• Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.
   Web of Science ®Google Scholar
• Skinner, E. A., Wellborn, J. G., & Connell, J. P. (1990). What it takes to do well in school and whether I’ve got it: A process model of perceived control and children’s engagement and achievement in school. Journal of Educational Psychology, 82(1), 22–32.
   Web of Science ®Google Scholar
• Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4), 66–78.
   Google Scholar
• Tell, C. A., Bodone, F. M., & Addie, K. L. (2000). A framework of teacher knowledge and skills necessary in a standards-based system: Lessons from high school and university faculty. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
   Google Scholar
• Thompson, M., & Wiliam, D. (2007). Tight but loose: A conceptual framework for scaling up school reforms. In E. C. Wylie (Ed.), Tight but loose: Scaling up teacher professional development in diverse contexts (pp. 13). Princeton, NJ: Educational Testing Service.
   Google Scholar
• Tobin, K. G., & Capie, W. (1982). Relationships between classroom process variables and middle-school science achievement. Journal of Educational Psychology, 74(3), 441.
   Web of Science ®Google Scholar
• Topping, K. J. (2010). Peers as a source of formative assessment. In H. L. Andrande & G. J. Cizek (Eds.), Handbook of formative assessment (pp. 73–86). New York, NY: Routledge.
   Google Scholar
• van de Pol, J., Volman, M., & Beishuizen, J. (2010). Scaffolding in teacher-student interaction: A decade of research. Educational Psychology Review, 22(3), 271–296.
   Web of Science ®Google Scholar
• van de Pol, J., Volman, M., Oort, F., & Beishuizen, J. (2015). The effects of scaffolding in the classroom: Support contingency and student independent working time in relation to student achievement, task effort and appreciation of support. Instructional Science, 43(5), 615–641.
   Web of Science ®Google Scholar
• Wolf, M. K., Guzman-Orth, D., Lopez, A., Castellano, K., Himelfarb, I., & Tsutagawa, F. S. (2016). Integrating scaffolding strategies into technology-enhanced assessments of English learners: Task types and measurement models. Educational Assessment, 21(3), 157–175.
   Web of Science ®Google Scholar
• Wylie, E. C. (this issue). Introduction to the Special Issue: Lessons Learned Through Validation. Submitted to Educational Assessment Journal, (n.d).
   Google Scholar
• Wylie, E. C., & Ciofalo, J. (2008). Supporting teachers’ use of individual diagnostic items. Teachers College Record, http://www.tcrecord.org/content.asp?contentid=15363.
   Google Scholar
• Wylie, E. C., & Dolan, R. P. (2013). The role of formalized tools in formative assessment. Paper presented at the American Educational Research Association, San Francisco, CA.
   Google Scholar
• Wylie, E. C., & Lyon, C. J. (2012). Quality instruction and quality formative assessment: The same or different? Paper presented at the annual meeting of the American Educational Research Association and the National Council on Measurement in Education, Vancouver, Canada.
   Google Scholar
• Wylie, E. C., & Lyon, C. J. (2015). The fidelity of formative assessment implementation: Issues of breadth and quality. Assessment in Education: Principles, Policy & Practice, 22(1), 140.
   Web of Science ®Google Scholar
• Wylie, E. C., & Lyon, C. J. (2019). The Role of Self- and Peer Assessment in Fromative Assessment and How Technology Can Support It. In S. M. Brookhart & J. H. McMillan (Eds.). In Classroom Assessment and Educational Measurement. New York, NY: Routledge, 170.
   Google Scholar
• Yin, R. K. (2014). Case study research: Design and methods (5 ed.). Thousands Oaks, CA: Sage Publications.
   Google Scholar
• Zhao, Y. (1998). The effects of anonymity on computer-mediated peer review. International Journal of Educational Telecommunications, 4(4), 311–345.
   Google Scholar
• Zimmerman, B. J. (2008). Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. American Educational Research Journal, 45(1), 166–183. doi:10.3102/0002831207312909
   Web of Science ®Google Scholar