Using calibrated exemplars in the teacher-assessment of writing: an empirical study

Abstract

Background: Many in education argue for the importance of incorporating teacher judgements in the assessment and reporting of student performance. Advocates of such an approach are cognisant, though, that obtaining a satisfactory level of consistency in teacher judgements poses a challenge.

Purpose: This study investigates the extent to which the use of a two-stage method of assessment involving calibrated exemplars provides judgements from teachers that are consistent. Teachers were not given extensive training and moderation. We chose the assessment of early writing as a context to investigate the method as it is fundamental to students’ progress in schooling.

Sample: Stage 1: Eleven teachers of four- to seven-year olds (kindergarten to year 2) were invited to collect their students’ performances. Sixty performances that represented the range of ability were selected from approximately 300 performances. Fifteen teachers from 12 schools made pairwise comparisons of performances. Stage 2: Fourteen teachers representing six schools plus the co-ordinator of the study participated in this stage of the exercise. Convenience sampling of teachers was employed.

Design and method: Stage 1: The method of pairwise comparison was used to calibrate the performances of students by developing a performance scale. These performances were then used as exemplars, which are referred to here as calibrated exemplars. Stage 2: Teachers assessed student performances simply by judging which calibrated exemplar a performance was most alike. In a separate exercise, two experienced markers assessed another set of 118 writing performances using both (1) a criterion-based rubric and (2) the calibrated exemplars.

Results: The two-staged process showed a level of consistency in teacher judgement-making. In addition, judgements made by experienced markers with the calibrated exemplars correlated well with judgements made using the criterion-based rubric.

Conclusions: The findings suggest that using calibrated exemplars has potential as a method of teacher assessment in contexts where extensive training and moderation is not possible or desirable. Further research is needed to establish whether the findings generalise to the classroom context and whether consistency could be demonstrated on a large scale in this and other curriculum areas. Research is also needed to investigate whether the calibrated exemplars can be supported with qualitative information for use in formative assessment.

Keywords:

• scoring methods
• accountability
• reliability
• teacher assessment
• pairwise comparison
• calibrated exemplars

Acknowledgements

The authors wish to thank Stephen Breen, Jean Rice and Katrina Bonetti from the Western Australian Primary Principals Association; Wendy Gorman from the Association of Independent Schools of Western Australia; A/Prof Mary Rohl and Kellie Picker from the University of Western Australia; Melanie Osborne and the staff at Comet Bay PS; Laura O’Hara and the staff at Noranda PS; Perette Minciullo and the staff from Moerlina School. We would like to acknowledge the early childhood teachers from both government and independent schools, who are too many to name here, but who made a central contribution to this research. We would also like to thank Jan Brandreth and Tracie Wright for their dedication in the marking. The work was supported by an Australian Research Council Linkage grant with the Australian Curriculum and Reporting Authority and Western Australian Curriculum Council as Industry Partners, on which Stephen Humphry and David Andrich are chief investigators.