Variability in ESL Essay Rating Processes: The Role of the Rating Scale and Rater Experience

REFERENCES

• Bacha , N. 2001 . Writing evaluation: What can analytic versus holistic essay scoring tell us? . System , 29 : 371 – 383 .
   Google Scholar
• Barkaoui , K. 2007a . Participants, texts, and processes in second language writing assessment: A narrative review of the literature . The Canadian Modern Language Review , 64 : 97 – 132 .
   Web of Science ®Google Scholar
• Barkaoui , K. 2007b . Rating scale impact on EFL essay marking: A mixed-method study . Assessing Writing , 12 : 86 – 107 .
   Google Scholar
• Barkaoui , K. 2008 . Effects of scoring method and rater experience on ESL essay rating processes and outcomes , Unpublished Ph.D. Thesis, University of Toronto .
   Google Scholar
• Barkaoui , K. forthcoming . Think-aloud protocols in research on essay rating: An empirical study of their veridicality and reactivity . Language Testing. ,
   Google Scholar
• Broad , B. 2003 . What we really value: Rubrics in teaching and assessing writing , Logan : Utah State University Press .
   Google Scholar
• Cumming , A. 1990 . Expertise in evaluating second language compositions . Language Testing , 7 : 31 – 51 .
   Google Scholar
• Cumming , A. , Kantor , R. and Powers , D. 2001 . Scoring TOEFL essays and TOEFL 2000 prototype writing tasks: An investigation into raters’ decision making and development of a preliminary analytic framework , Princeton, NJ : Educational Testing Service . TOEFL Monograph Series N 22
   Google Scholar
• Cumming , A. , Kantor , R. and Powers , D. 2002 . Decision making while rating ESL/EFL writing tasks: A descriptive framework . Modern Language Journal , 86 : 67 – 96 .
   Web of Science ®Google Scholar
• Davison , C. 2004 . The contradictory culture of teacher-based assessment: ESL teacher assessment practices in Australia and Hong Kong secondary schools . Language Testing , 21 : 305 – 334 .
   Google Scholar
• Delaruelle , S. 1997 . “ Text type and rater decision-making in the writing module ” . In Access: Issues in English language test design and delivery , Edited by: Brindley , G. and Wigglesworth , G. 215 – 242 . Sydney, , Australia : National Center for English Language Teaching and Research, Macquarie University .
   Google Scholar
• Erdosy , M. U. 2004 . Exploring variability in judging writing ability in a second language: A study of four experienced raters of ESL compositions , Princeton, NJ : Educational Testing Service . TOEFL Research Report No. RR-03-17
   Google Scholar
• Freedman , S. W. and Calfee , R. C. 1983 . “ Holistic assessment of writing: Experimental design and cognitive theory ” . In Research on writing: Principles and methods , Edited by: Mosenthal , P. , Tamor , L. and Walmsley , S. A. 75 – 98 . New York : Longman .
   Google Scholar
• Goulden , N. R. 1992 . Theory and vocabulary for communication assessments . Communication Education , 41 : 258 – –269 .
   Web of Science ®Google Scholar
• Goulden , N. R. 1994 . Relationship of analytic and holistic methods to raters’ scores for speeches . The Journal of Research and Development in Education , 27 : 73 – 82 .
   Google Scholar
• Hamp-Lyons , L. 1991 . “ Scoring procedures for ESL contexts ” . In Assessing second language writing in academic contexts , Edited by: Hamp-Lyons , L. 241 – 276 . Norwood, NJ : Ablex .
   Google Scholar
• Hamp-Lyons , L. and Kroll , B. 1997 . TOEFL 2000-writing: Composition, community and assessment , Princeton, NJ : Educational Testing Service . TOEFL Monograph Series N 5
   Google Scholar
• Homburg , T. J. 1984 . Holistic evaluation of ESL compositions: Can it be validated objectively? . TESOL Quarterly , 18 : 87 – 107 .
   Google Scholar
• Huot , B. A. 1993 . “ The influence of holistic scoring procedures on reading and rating student essays ” . In Validating holistic scoring for writing assessment: Theoretical and Empirical foundations , Edited by: Williamson , M. M. and Huot , B. A. 206 – 236 . Cresskill, NJ : Hampton .
   Google Scholar
• Lumley , T. 2005 . Assessing second language writing: The rater's perspective , New York : Peter Lang .
   Google Scholar
• Lumley , T. and McNamara , T. F. 1995 . Rater characteristics and rater bias: Implications for training . Language Testing , 12 : 54 – 71 .
   Google Scholar
• Milanovic , M. , Saville , N. and Shuhong , S. 1996 . “ A study of the decision-making behaviour of composition markers ” . In Performance testing, cognition and assessment: Selected papers from the 15th Language Testing Colloquium (LTRC), Cambridge and Arnhem , Edited by: Milanovic , M. and Saville , N. 92 – 114 . Cambridge, , UK : Cambridge University Press .
   Google Scholar
• Noldus Information Technology b.v . 2003 . Observer (Version 5.0). [Computer software] , Wageningen, , the Netherlands : Author .
   Google Scholar
• O'Loughlin , K. 1994 . The assessment of writing by English and ESL teachers . Australian Review of Applied Linguistics , 17 : 23 – 44 .
   Google Scholar
• Ruth , L. and Murphy , S. 1988 . Designing writing tasks for the assessment of writing , Norwood, NJ : Ablex .
   Google Scholar
• Sakyi , A. A. 2003 . A study of the holistic scoring behaviors of experienced and novice ESL instructors , Toronto, , Canada : Unpublished doctoral dissertation, University of Toronto .
   Google Scholar
• Schoonen , R. 2005 . Generalizability of writing scores: An application of structural equation modeling . Language Testing , 22 : 1 – 30 .
   Google Scholar
• Schoonen , R. , Vergeer , M. and Eiting , M. 1997 . The assessment of writing ability: Expert readers versus lay readers . Language Testing , 14 : 157 – 184 .
   Google Scholar
• Smith , D. 2000 . “ Rater judgments in the direct assessment of competency-based second language writing ability ” . In Studies in immigrant English language assessment, Volume 1 , Edited by: Brindley , G. 159 – 189 . Sydney, , Australia : Macquarie University .
   Google Scholar
• Song , C. B. and Caruso , I. 1996 . Do English and ESL faculty differ in evaluating the essays of native English-speaking and ESL students? . Journal of Second Language Writing , 5 : 163 – 182 .
   Google Scholar
• Torrance , H. 1998 . Learning from research in assessment: A response to writing assessment-raters’ elaboration of the rating task . Assessing Writing , 5 : 31 – 37 .
   Google Scholar
• Vaughan , C. 1991 . “ Holistic assessment: What goes on in the rater's mind? ” . In Assessing second language writing in academic contexts , Edited by: Hamp-Lyons , L. 111 – 125 . Norwood, NJ : Ablex .
   Google Scholar
• Weigle , S. C. 1994 . Effects of training on raters of ESL compositions . Language Testing , 11 : 197 – 223 .
   Google Scholar
• Weigle , S. C. 1998 . Using FACETS to model rater training effects . Language Testing , 15 : 263 – 287 .
   Google Scholar
• Weigle , S. C. 1999 . Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches . Assessing Writing , 6 : 145 – 178 .
   Google Scholar
• Weigle , S. C. 2002 . Assessing writing , Cambridge, , UK : Cambridge University Press .
   Google Scholar
• Wolfe , E. W. 2006 . Uncovering rater's cognitive processing and focus using think-aloud protocols . Journal of Writing Assessment , 2 : 37 – 56 .
   Google Scholar
• Wolfe , E. W. , Kao , C. and Ranney , M. 1998 . Cognitive differences in proficient and non-proficient essay scorers . Written Communication , 15 : 465 – 492 .
   Web of Science ®Google Scholar