Weight-Based Classification of Raters and Rater Cognition in an EFL Speaking Test

Abstract

This study is an attempt to classify raters according to their weighting patterns and explore systematic differences between rater types in the rating process. In the context of an EFL speaking test, 126 raters were classified into three types—form-oriented, balanced, and content-oriented—through cluster analyses of their weighting patterns derived from a holistic judgment task. In subsequent verbal protocols, a smaller sample of raters from each type rated the performance of real test takers and justified their ratings. The procedure for classifying raters yielded reliable results that agreed broadly with self-perceived and actual weights of the raters in the rating process. Verbal protocol analyses found systematic differences across rater types in the distinction between criteria and conformity to rubric-implied criteria. The form-oriented raters were found to overweigh form-related criteria and underweigh content-related criteria, which suggested unbalanced construct representation and led to more halos. Preliminary evidence also suggested that the form-oriented raters might overrate digressive content due to lower sensitivity to digression.

Notes

¹ http://www.cambridgeenglish.org/exams/proficiency/

² http://www.ielts.org/

³ www.cambridgeenglish.org/exams/first

⁴ http://www.ets.org/toefl/

⁵ www.testdaf.de

Additional information

Funding

This article is funded by the project sponsored by SRF for ROCS, SEM, China. The article has evolved out of the author’s PhD dissertation, which was generously supported by Harry and Yvonne Lenart Graduate Travel Fellowship, TOEFL Small Grant for Doctoral Research in Second or Foreign Language Assessment (Educational Testing Service), the Institute of Oral English Studies of Nanjing University, and the Dissertation Year Fellowship at the University of California, Los Angeles.