Exploring the Effectiveness of a Measurement Error Tutorial in Helping Teachers Understand Score Report Results

Abstract

The goal of this study was to explore the effectiveness of a short web-based tutorial in helping teachers to better understand the portrayal of measurement error in test score reports. The short video tutorial included both verbal and graphical representations of measurement error. Results showed a significant difference in comprehension scores between each of two tutorial groups (basic and enhanced) and the control group (no tutorial) but not between the two tutorial groups. Results also provided evidence of teachers' misconceptions about the meaning of measurement error and confidence bands.

Acknowledgments

We acknowledge Janet Stumper, Thomas Florek, and Debbie Pisacreta, for their support developing the video tutorial and online delivery instruments; Melanie Schine for her work recruiting participants for the study; Steven Holtzman for contributing to the analysis of the data; and Peter van Rijn, Samuel Livingston, Randy Bennett, Teresa Egan, Mary Hegarty, Caroline Wylie, and Michael Zieky for their feedback on prior versions of the tutorial script. We acknowledge the support of the CBAL research initiative. Finally, we thank the teachers who participated in our study.

Notes

¹ Tutorial scripts and videos are available to interested researchers on request.

² For these six items, the item percent correct was very similar (within 0.09) for the two tutorial groups, with the exception of Item 16, which had a somewhat higher percent correct for the enhanced tutorial (0.13).