An Investigation of the Reliability of Using Comparative Judgment to Score Creative Products

ABSTRACT

In this article we describe a three-year study that was conducted in three phases to evaluate the feasibility of assessing digitized portfolios of student creative work for high-stakes purposes. The first two phases suggested that creative work could be digitized with adequate fidelity, and that students could submit their own work from schools to an online portfolio system. These portfolios were assessed using online tools to facilitate both an analytical and comparative judgment method of scoring, with the latter appearing to be the more reliable. In the final phase this approach was evaluated to facilitate moderation and professional learning of standards with rural teachers. The study showed that comparative judgment is a viable and reliable method for assessing digitized creative products and coupled with online communication systems provides an effective and efficient approach to scoring, moderation and teacher professional development, particularly in rural locations.

Acknowledgments

The study discussed in this article was the work of a research team led by Paul Newhouse and included researchers Jeremy Pagram, Lisa Paris, Mark Hackling, Martin Cooper, Pina Tarricone, Alistair Campbell, Alun Price and many research assistants. The work of everyone in this team, particularly Martin Cooper and Pina Tarricone, and the teachers and students involved, contributed to the research outcomes presented in this article.