Developing assessment scales for large-scale speaking tests: a multiple-method approach

References

• Alderson, J.C., N. Figueras, H. Kuijper, G. Nold, S. Takala, and S. Tardieu. 2004. The development of specifications for item development and classification within the Common European Framework of Reference for languages: Learning, teaching, assessment: Reading and listening. Final report of the Dutch CEF construct project. Lancaster University, Lancaster, UK. Unpublished.
   Google Scholar
• Brindley , G. 1998 . “ Describing language development? Rating scales and SLA ” . In Interfaces between second language acquisition and language testing research , Edited by: Bachman , L.F. and Cohen , A.D. 12 – 140 . Cambridge : Cambridge University Press .
   Google Scholar
• Brooks , L. 2009 . Interacting in pairs in a test of oral proficiency: Co-constructing a better performance . Language Testing , 26 ( 3 ) : 341 – 366 .
   Web of Science ®Google Scholar
• Brown , A. 2007 . “ An investigation of the rating process in the IELTS oral interview ” . In IELTS collected papers , Edited by: Taylor , L. and Falvey , P. 98 – 141 . Cambridge : Cambridge University Press .
   Google Scholar
• Canepa , A. 2006 . Review of speaking test criteria for Main Suite and BEC , Cambridge ESOL Internal Report Cambridge : UCLES .
   Google Scholar
• Chalhoub-Deville , M. 1995 . Deriving oral assessment scales across different tests and rater groups . Language Testing , 12 ( 1 ) : 16 – 33 .
   Google Scholar
• Christiansen , T. 2006 . Discourse Management assessment criteria in Main Suite and BEC , Cambridge ESOL Internal Report Cambridge : UCLES .
   Google Scholar
• Council of Europe . 2001 . Common European Framework of Reference for Languages: Learning, teaching, assessment , Cambridge : Cambridge University Press .
   Google Scholar
• Ducasse , A.M. and Brown , A. 2009 . Assessing paired orals: Raters’ orientation to interaction . Language Testing , 26 ( 3 ) : 423 – 443 .
   Web of Science ®Google Scholar
• Falvey , P. and Shaw , S. 2006 . IELTS writing: Revising assessment criteria and scales (Phase 4) . Cambridge ESOL Research Notes , 23 : 7 – 12 .
   Google Scholar
• ffrench , A. 2003 . “ The change process at the paper level. Paper 5, Speaking ” . In Continuity and innovation: Revising the Cambridge Proficiency in English examination 1913–2002 , Edited by: Weir , C. and Milanovic , M. 367 – 471 . Cambridge : Cambridge University Press .
   Google Scholar
• Field, J. 2011. Cognitive validity. In Examining speaking: Research and practice in assessing second language speaking , ed. L. Taylor. Cambridge: Cambridge University Press.
   Google Scholar
• Fulcher , G. 1996 . Does thick description lead to smart tests? A data-based approach to rating scale construction . Language Testing , 13 ( 1 ) : 208 – 238 .
   Google Scholar
• Galaczi , E.D. 2008 . Peer-peer interaction in a speaking test: The case of the First Certificate in English examination . Language Assessment Quarterly , 5 ( 2 ) : 89 – 119 .
   Web of Science ®Google Scholar
• Green , A. 2006 . Placing speaking descriptors on the Cambridge ESOL scale , Cambridge ESOL Internal Report Cambridge : UCLES .
   Google Scholar
• Green , A. 2011 . Language functions revisited: Theoretical and empirical bases for language construct definition across the ability range , Cambridge : Cambridge University Press .
   Google Scholar
• Gude , K. 2006 . Review of speaking test criteria for Main Suite and BEC , Cambridge ESOL Internal Report Cambridge : UCLES .
   Google Scholar
• Hawkins , J.A. and Filipović , L. 2011 . Criterial features in the learning of English: Specifying the reference levels of the Common European Framework , Cambridge : Cambridge University Press .
   Google Scholar
• Horner , D. 2006 . Review of speaking test criteria for Cambridge Main Suite and BEC: Interactive communication , Cambridge ESOL Internal Report Cambridge : UCLES .
   Google Scholar
• Hubbard , C. , Gilbert , S. and Pidcock , J. 2006 . Assessment processes in speaking tests: A pilot verbal protocol study . Cambridge ESOL Research Notes , 24 : 14 – 19 .
   Google Scholar
• Isaacs , T. 2008 . Towards defining a valid assessment criterion of pronunciation proficiency in non-native English-speaking graduate students . Canadian Modern Language Review , 64 ( 4 ) : 555 – 580 .
   Web of Science ®Google Scholar
• Jacoby , S. and Ochs , E. 1995 . Co-construction: An introduction . Research in Language and Social Interaction , 28 ( 3 ) : 171 – 183 .
   Web of Science ®Google Scholar
• Knoch , U. 2009 . Diagnostic assessment of writing: A comparison of two rating scales . Language Testing , 26 ( 2 ) : 275 – 304 .
   Web of Science ®Google Scholar
• Knoch , U. 2009 . Collaborating with ESP stakeholders in rating scale validation: The case of the ICAO rating scale . Spaan Fellow Working Papers in Second or Foreign Language Assessment , 7 : 21 – 46 .
   Google Scholar
• Lazaraton , A. 2002 . A qualitative approach to the validation of oral language tests , Cambridge : Cambridge University Press .
   Google Scholar
• Lazaraton , A. and Davis , L. 2007 . An analysis of candidate language output on the PET Speaking Test standardisation videos , Internal Cambridge ESOL Report Cambridge : UCLES .
   Google Scholar
• Linacre, M. 2006. Facets Rasch Measurement computer programme. Chicago, IL: Winsteps.
   Google Scholar
• Lunz , M.E. and Wright , B.D. 1997 . “ Latent trait models for performance examinations ” . In Applications of latent trait and latent class models in the social sciences , Edited by: Rost , J. and Langehene , R. 80 – 88 . Munster, , Germany : Waxmann .
   Google Scholar
• Lynch , B. and McNamara , T. 1998 . Using G theory and Many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants . Language Testing , 15 ( 2 ) : 158 – 180 .
   Google Scholar
• Matthews , A. 2006 . Review of speaking test criteria for Main Suite and BEC , Cambridge ESOL Internal Report Cambridge : UCLES .
   Google Scholar
• McNamara , T. 1996 . Measuring second language proficiency , London : Longman .
   Google Scholar
• Milanovich , M. , Saville , N. , Pollitt , A. and Cook , A. 1996 . “ Developing rating scales for CASE: Theoretical concerns and analyses ” . In Validation in language testing , Edited by: Cumming , A. and Berwick , R. 15 – 38 . Clevedon : Multilingual Matters .
   Google Scholar
• Myford , C.M. and Wolfe , E.W. 2003 . Detecting and measuring rater effects using Multi-Facet Rasch measurement: Part 1 . Journal of Applied Measurement , 4 ( 4 ) : 386 – 422 .
   PubMedGoogle Scholar
• Myford , C.M. and Wolfe , E.W. 2004 . Detecting and measuring rater effects using Multi-Facet Rasch measurement: Part 2 . Journal of Applied Measurement , 5 ( 2 ) : 189 – 227 .
   PubMedGoogle Scholar
• O’Sullivan, B. 2005. A practical introduction to using FACETS in language testing research. Unpublished manuscript. Roehampton University.
   Google Scholar
• O’Sullivan , B. , Weir , C.J. and Saville , N. 2002 . Using observation checklists to validate speaking-test tasks . Language Testing , 19 ( 1 ) : 33 – 56 .
   Google Scholar
• Saville , N. 2003 . “ The process of test development and revision within UCLES EFL ” . In Continuity and innovation: Revising the Cambridge Proficiency in English examination 1913–2002 , Edited by: Milanovic , M. and Weir , C. 57 – 120 . Cambridge : University of Cambridge Local Examinations Syndicate .
   Google Scholar
• Sawaki , Y. 2007 . Construct validation of analytic rating scales in a speaking assessment: Reporting a score profile and a composite . Language Testing , 24 ( 3 ) : 355 – 390 .
   Google Scholar
• Taylor, L. 1999. Study of quantitative differences between CPE individual and paired speaking tests. Internal Cambridge ESOL report. University of Cambridge, UK.
   Google Scholar
• Taylor, L. 2011. Introduction. In Examining speaking: Research and practice in assessing second language speaking, ed. L. Taylor. Cambridge: Cambridge University Press.
   Google Scholar
• Taylor , L. and Jones , N. 2006 . Cambridge ESOL exams and the Common European Framework of Reference (CEFR) . Cambridge ESOL Research Notes , 24 : 2 – 6 .
   Google Scholar
• Taylor , L. and Wigglesworth , G. 2009 . Are two heads better than one? Pair work in L2 assessment contexts . Language Testing , 26 ( 3 ) : 325 – 339 .
   Web of Science ®Google Scholar
• Turner , C.E. 2000 . Listening to the voices of rating scale developers: Identifying salient features of second language performance assessment . The Canadian Modern Language Review , 56 ( 4 ) : 555 – 584 .
   Web of Science ®Google Scholar
• Turner , C.E. and Upshur , J.A. 2002 . Rating scales derived from student samples: Effects of the scale maker and the student sample on scale content and student scores . TESOL Quarterly , 36 ( 1 ) : 49 – 70 .
   Web of Science ®Google Scholar
• University of Cambridge ESOL Examinations . 2007 . First Certificate in English handbook for teachers , Cambridge : UCLES .
   Google Scholar
• University of Cambridge ESOL Examinations . 2008 . Certificate in Advanced English handbook for teachers , Cambridge : UCLES .
   Google Scholar
• University of Cambridge ESOL Examinations . 2008 . Certificate of Proficiency in English handbook for teachers , Cambridge : UCLES .
   Google Scholar
• University of Cambridge ESOL Examinations . 2008 . Key English Test handbook for teachers , Cambridge : UCLES .
   Google Scholar
• University of Cambridge ESOL Examinations . 2008 . Preliminary English Test handbook for teachers , Cambridge : UCLES .
   Google Scholar
• van Moere , A. 2006 . Validity evidence in a university group oral test . Language Testing , 23 ( 4 ) : 411 – 440 .
   Google Scholar
• Weir , C.J. 2003 . “ A survey of the history of the Certificate of Proficiency in English (CPE) in the twentieth century ” . In Continuity and innovation: A history of the CPE Examination 1913–2002 , Edited by: Weir , C.J. and Milanovic , M. 1 – 56 . Cambridge : University of Cambridge Local Examinations Syndicate .
   Google Scholar
• Xi , X. 2010 . Automated scoring and feedback systems: Where are we and where are we heading? . Language Testing , 27 ( 3 ) : 291 – 300 .
   Web of Science ®Google Scholar