Self-Monitoring Assessments for Educational Accountability Systems

Abstract

Test-based accountability is now the cornerstone of U.S. education policy, and it is becoming more important in many other nations as well. Educators sometimes respond to test-based accountability in ways that produce score inflation. In the past, score inflation has usually been evaluated by comparing trends in scores on a high-stakes test to trends on a lower-stakes audit test. However, separate audit tests are often unavailable, and their use has several important drawbacks, such as potential bias from motivational differences. As an alternative, we propose self-monitoring assessments (SMAs) that incorporate audit components into operational high-stakes assessments. This paper provides a framework for designing SMAs. It describes 5 specific SMA designs that could be incorporated into the non-equivalent groups anchor test linking approaches used by most large-scale assessments and discusses analytical issues that would arise in their use.

This article is based on an invited paper presented at the annual meeting of the American Educational Research Association, Denver, CO, May 2010, as part of the symposium Innovations in Test-Based Educational Accountability (H. Braun, Chair).

Notes

¹The findings of Morley et al. (2004) suggest a third possible form for audit items. They found that “appearance variants,” items that shared item style but not content with comparison items, were in general more difficult than either matched items or isomorphs. The implications of this for present purposes are unclear. If the content of the appearance variants were sufficiently different from that of comparison items, the appearance variants would be content audit items. However, it is also possible that the weak performance on these items represents a focus on item style that distracts students from content. If so, this would provide an opportunity for a third type of audit, which would be harder rather than easier than comparison items in the presence of score inflation. This possibility is not addressed further in this paper.