Designing and Scaling Level-Specific Writing Tasks in Alignment With the CEFR: A Test-Centered Approach

REFERENCES

• Adams , R. J. 2006 . Reliability as a measurement design effect . Studies in Educational Evaluation , 31 : 162 – 172 .
   Google Scholar
• Adams , R. J. and Wu , M. L. 2009 . The construction and implementation of user-defined fit tests for use with marginal maximum likelihood estimation and generalized item response models . Journal of Applied Measurement , 10 : 355 – 370 .
   Google Scholar
• Alderson , J. C. , Figueras , N. , Kuijper , H. , Nold , G. , Takala , S. and Tardieu , C. 2006 . Analysing tests of reading and listening in relation to the common European framework of reference: The experience of the Dutch CEFR construct project . Language Assessment Quarterly , 3 : 3 – 30 .
   Google Scholar
• Alderson , J. C. and Huhta , A. 2005 . The development of a suite of computer-based diagnostic tests based on the Common European Framework . Language Testing , 22 : 301 – 320 .
   Google Scholar
• Beck , B. and Klieme , E. , eds. 2007 . Sprachliche Kompetenzen – Konzepte und Messungen. DESI Studie [Linguistic competences—Constructs and their measurement. DESI –Study] , Weinheim, , Germany : Beltz .
   Google Scholar
• Brennan , R. L. 2001 . Generalizability theory , New York, NY : Springer .
   Google Scholar
• Brennan , R. L. 2007 . An NCME instructional module on generalizability theory items . Instructional Topics in Educational Measurement , : 225 – 232 .
   Google Scholar
• Briggs , D. C. and Wilson , M. 2007 . Generalizability in item response modeling . Journal of Educational Measurement , 44 : 131 – 155 .
   Web of Science ®Google Scholar
• Brindley , G. 2000 . “ Task difficulty and task generalizability in competency-based writing assessment ” . In Studies in immigrant English language assessment , Edited by: Brindley , G. Vol. 1 , 125 – 157 . Sydney, , Australia : National Centre for English Language Teaching and Research, Macquarie University .
   Google Scholar
• Brindley , G. 2001 . “ Investigating rater consistency in competency-based language assessment ” . In Studies in immigrant English language assessment , Edited by: Brindley , G. and Burrows , C. Vol. 2 , 59 – 80 . Sydney, , Australia : National Centre for English Language Teaching and Research, Macquarie University .
   Google Scholar
• Brown , G. , Anderson , A. , Shillcock , R. and Yule , G. 1984 . Teaching talk: Strategies for production and assessment , Cambridge, , UK : Cambridge University Press .
   Google Scholar
• Cambridge ESOL: Handbook for teachers. (n.d.) Cambridge, UK: University of Cambridge. https://www.teachers.cambridgeesol.org/ts/digitalAssets/109740_cae_hb_dec08.pdf (https://www.teachers.cambridgeesol.org/ts/digitalAssets/109740_cae_hb_dec08.pdf)
   Google Scholar
• Chiu , C. W. T. and Wolfe , E. W. 2002 . A method for analyzing sparse data matrices in the generalizability theory framework . Applied Psychological Measurement , 26 ( 3 ) : 321 – 338 .
   Google Scholar
• Cizek , G. J. , Bunch , M. B. and Koons , H. 2004 . Setting performance standards: Contemporary methods . Educational Measurement: Issues and Practice , 23 ( 4 ) : 31 – 50 .
   Google Scholar
• Congdon , P. J. and McQueen , J. 2000 . The stability of rater severity in large-scale assessment programs . Journal of Educational Measurement , 37 : 163 – 178 .
   Web of Science ®Google Scholar
• Council of Europe. (2001). Common European reference framework for languages. Strasbourg, France: Author. http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/common_framework.html (http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/common_framework.html)
   Google Scholar
• Council of Europe. (2003). Relating language examinations to the Common European Framework of Reference for Languages: Learning, Teaching, Assessment (CEF). Manual, preliminary pilot version. Strasbourg, France: Author. http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/Manual.html (http://www.coe.int/T/DG4/Portfolio/?L=E&M=/documents_intro/Manual.html)
   Google Scholar
• Council of Europe. (2008). The CEFR Grid for Writing Tasks. Strasbourg, France: Author. http://www.coe.int/T/DG4/Portfolio/documents/CEFRWritingGridv3_1_presentation.doc (http://www.coe.int/T/DG4/Portfolio/documents/CEFRWritingGridv3_1_presentation.doc)
   Google Scholar
• Council of Europe (2009). Manual for Relating Language Examinations to the Common European Framework of Reference for Languages (CEFR). Strasbourg, France: Author. http://www.coe.int/t/dg4/linguistic/Manuel1_EN.asp (http://www.coe.int/t/dg4/linguistic/Manuel1_EN.asp)
   Google Scholar
• Crick , J. E. and Brennan , R. L. 1983 . Manual for GENOVA: A generalized analysis of variance system (ACT Technical Bulletin No. 43) , Iowa City : American College Testing .
   Google Scholar
• de Ayala , R. J. 2009 . The theory and practice of item response theory , New York, NY : Guilford .
   Google Scholar
• De Jong , J. H. What is the role of the Common European Framework of Reference for Languages: Learning, teaching, assessment? . Paper presented at the EALTA conference . May , Kranjska Gora, Slovenia.
   Google Scholar
• Dutch Grid http://www.lancs.ac.uk/fss/projects/grid/grid.php (http://www.lancs.ac.uk/fss/projects/grid/grid.php)
   Google Scholar
• Eckes , T. 2005 . Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis . Language Assessment Quarterly , 2 : 197 – 221 .
   Google Scholar
• Eckes , T. 2008 . Rater types in writing performance assessments: A classification approach to rater variability . Language Testing , 25 : 155 – 185 .
   Web of Science ®Google Scholar
• Embretson , S. E. and Reise , S. P. 2000 . Item response theory for psychologists , Mahwah, NJ : Erlbaum .
   Google Scholar
• Frey , A. , Hartig , J. and Rupp , A. A. 2009 . An NCME instructional module on booklet designs in large-scale assessments of student achievement: Theory and practice . Educational Measurement: Issues and Practice , 28 ( 3 ) : 39 – 53 .
   Google Scholar
• Grabe , W. and Kaplan , R. B. 1996 . Theory & practice of writing: An applied linguistic perspective , Harlow, , England : Pearson Education .
   Google Scholar
• Hamp-Lyons , L. and Kroll , B. 1996 . Issues in ESL Writing Assessment: An overview . College ESL , 6 ( 1 ) : 52 – 72 .
   Google Scholar
• Harsch , C. 2007 . Der gemeinsame europäische Referenzrahmen für Sprachen: Leistung und Grenzen [The Common European Framework of Reference for Languages: Strengths and limitations] , Saarbrücken, , Germany : VDM .
   Google Scholar
• Harsch , C. 2010 . “ Schreibbewertung im Zuge der Normierung der KMK–Bildungsstandards: Der “niveauspezifische Ansatz” und ausgewählte Schritte zu seiner Validierung [Writing assessment in the context of evaluating the German educational standards: The level-specific approach and steps towards its validation] ” . In Fremdsprachliches Handeln beobachten, messen und evaluieren: Neue methodische Ansätze der Kompetenzforschung und Videographie , Edited by: Aguado , K. , Vollmer , H. and Schramm , K. 99 – 117 . Frankfurt, , Germany : Lang .
   Google Scholar
• Harsch , C. and Martin , G. 2011 . Coding the descriptors-a data–driven approach to improving rating validity , Manuscript submitted for publication .
   Google Scholar
• Harsch , C. , Neumann , A. , Lehmann , R. and Schröder , K. 2007 . “ Schreibfähigkeit [Writing proficiency] ” . In Sprachliche Kompetenzen: Konzepte und Messungen in der DESI–Studie , Edited by: Beck , B. and Klieme , E. 42 – 62 . Weinheim, , Germany : Beltz .
   Google Scholar
• Harsch , C. , Pant , A. and Köller , O. , eds. 2010 . Calibrating Standards-based Assessment Tasks for English as a First Foreign Language—Standard-setting procedures in Germany , Münster, , Germany : Waxmann .
   Google Scholar
• Hawkey , R. A. and Shaw , S.D. 2005 . The Common Scale for Writing project: Implications for the comparison of IELTS band scores and Main Suite exam levels . Research Notes , 19 : 19 – 24 .
   Google Scholar
• Hayes , J. R. 1996 . “ A new framework for understanding cognition and affect in writing ” . In The science of writing. Theories, methods, individual differences and applications , Edited by: Levy , C. M. and Ransdell , S. 1 – 27 . Hillsdale, NJ : Erlbaum .
   Google Scholar
• Hayes , J. R. and Flower , S. L. 1980 . “ Identifying the organization of writing processes ” . In Cognitive processes in writing , Edited by: Gregg , L. W. and Steinberg , E. R. 31 – 50 . Hillsdale, NJ : Erlbaum .
   Google Scholar
• Henson , R. and Templin , J. Implementation of standards setting for an algebra two end-of-course exam using cognitive diagnosis . Paper presented at the annual meeting of the American Educational Research Association . March , New York.
   Google Scholar
• Hulstijn , J. H. 2007 . The shaky ground beneath the CEFR: Quantitative and qualitative dimensions of language proficiency . The Modern Language Journal , 91 : 663 – 667 .
   Web of Science ®Google Scholar
• Iwashita , N. , McNamara , T. and Elder , C. 2001 . Can we predict task difficulty in an oral proficiency test? Exploring the potential of an information-processing approach to task design . Language Learning , 51 : 401 – 436 .
   Web of Science ®Google Scholar
• Kline , T. J. B. 2005 . “ Classical test theory: Assumptions, equations, limitations, and item analyses ” . In Psychological testing: A practical approach to design and evaluation , Edited by: Kline , T. J. B. 91 – 105 . Thousand Oaks, CA : Sage .
   Google Scholar
• Knoch , U. 2009 . Diagnostic assessment of writing: A comparison of two rating scales . Language Testing , 26 : 275 – 304 .
   Web of Science ®Google Scholar
• Kuiken , F. and Vedder , I. 2007 . Task complexity and measures of linguistics performance in L2 writing . IRAL , 45 : 261 – 284 .
   Google Scholar
• Lord , F. M. and Novick , M. R. 1968 . Statistical theories of mental test scores , Boston, MA : Addison-Wesley .
   Google Scholar
• Lumley , T. 2002 . Assessment criteria in a large-scale writing test: what do they really mean to the raters? . Language Testing , 19 : 246 – 276 .
   Google Scholar
• Lumley , T. 2005 . Assessing second language writing: the rater's perspective , Frankfurt, , Germany : Lang .
   Google Scholar
• Mislevy , J. and Rupp , A. A. 2009 . Using item response theory in survey research: Accommodating complex assessment and sampling designs , Manuscript submitted for publication .
   Google Scholar
• Morrow , K. 2004 . “ Background to the CEF ” . In Insights from the Common European Framework , Edited by: Morrow , K. 3 – 11 . Oxford, , UK : Oxford University Press .
   Google Scholar
• Norris , J. M. , Brown , J. D. , Hudson , T. D. and Bonk , W. 2002 . Examinee abilities and task difficulty in task-based second language performance assessment . Language Testing , 19 : 395 – 418 .
   Google Scholar
• North , B. 2004 . “ Relating assessments, examinations, and courses to the CEF ” . In Insights from the Common European Framework , Edited by: Morrow , K. 77 – 90 . Oxford, , UK : Oxford University Press .
   Google Scholar
• Prabhu , N. S. 1987 . Second language pedagogy , Oxford, , UK : Oxford University Press .
   Google Scholar
• Preece , D. A. 1990 . Fifty years of Youden squares: A review . Bulletin of the Institute of Mathematics and Its Applications , 26 : 65 – 75 .
   Google Scholar
• Robinson , P. 2001 . “ Task complexity, cognitive resources, and syllabus design: A triadic framework for examining task influences on SLA ” . In Cognition and second language instruction , Edited by: Robinson , P. 287 – 318 . Cambridge, , UK : Cambridge University Press .
   Google Scholar
• Rupp , A. A. and Porsch , R. 2010 . “ Standard-setting item pool ” . In Calibrating standards-based assessment tasks for English as a First Foreign Language—Standard-setting procedures in Germany , Edited by: Harsch , C. , Pant , H. A. and Köller , O. 39 – 60 . Münster, , Germany : Waxmann .
   Google Scholar
• Rupp , A. A. , Templin , J. and Henson , R. 2010 . “ Diagnostic measurement: Theory ” . In methods, and applications , New York, NY : Guilford .
   Google Scholar
• Rupp , A. A. , Vock , M. , Harsch , C. and Köller , O. 2008 . Developing standards-based assessment items for English as a first foreign language: Context, processes and outcomes in Germany , Münster, , Germany : Waxmann .
   Google Scholar
• Shaw , S. and Weir , C. J. 2007 . Examining writing in a second language. Studies in Language Testing 26 , Cambridge, , UK : Cambridge University Press and Cambridge ESOL .
   Google Scholar
• Skehan , P. 1998 . A cognitive approach to language learning , Oxford, , UK : Oxford University Press .
   Google Scholar
• Skehan , P. and Foster , P. 2001 . “ Cognition and tasks ” . In Cognition and second language instruction , Edited by: Robinson , P. 183 – 205 . Cambridge, , UK : Cambridge University Press .
   Google Scholar
• Smith , D. 2000 . “ Rater judgments in the direct assessment of competency-based second language writing ability ” . In Studies in immigrant English language assessment , Edited by: Brindley , G. Vol. 1 , 159 – 189 . Sydney, , Australia : National Centre for English Language Teaching and Research, Macquarie University .
   Google Scholar
• Tankó , G. 2005 . Into Europe: Prepare for modern English exams. The writing handbook , Budapest, , Hungary : Teleki László Foundation .
   Google Scholar
• Taylor , L. and Jones , N. 2006 . “ Cambridge ESOL exams and the Common European Framework of Reference (CEFR) ” . In Research Notes, 24 , 2 – 5 . Cambridge, , UK : Cambridge ESOL .
   Google Scholar
• von Davier , M. , Sinharay , S. , Oranje , A. and Beaton , A. 2006 . “ The statistical procedures used in National Assessment of Educational Progress: Recent developments and future directions ” . In Handbook of statistics (Vol. 26): Psychometrics , Edited by: Rao , C. R. and Sinharay , S. 1039 – 1056 . Amsterdam, , the Netherlands : Elsevier .
   Google Scholar
• Wainer , H. and Thissen , D . 2001 . “ True score theory: The traditional method ” . In Test scoring , Edited by: Thissen , D. and Wainer , H. 23 – 72 . Hillsdale, NJ : Erlbaum . InEds.pp.
   Google Scholar
• Weigle , S. C. 1998 . Using FACETS to model rater training effects . Language Testing , 15 : 264 – 288 .
   Google Scholar
• Weigle , S. C. 1999 . Investigating rater / prompt interactions in writing assessment: Quantitative and qualitative approaches . Assessing Writing , 6 : 145 – 178 .
   Google Scholar
• Weigle , S. C. 2002 . Assessing writing , Cambridge, , UK : Cambridge University Press .
   Google Scholar
• Weir , C. J. 2005 . Limitations of the Common European Framework for developing comparable examinations and tests . Language Testing , 22 : 281 – 300 .
   Google Scholar
• Wu , M. L. , Adams , R. J. and Wilson , M. R. 1998 . ACER ConQuest: Generalised item response modeling software , Melbourne : The Australian Council for Educational Research Limited . [Computer program]
   Google Scholar
• Zieky , M. and Perie , M. 2006 . A primer on setting cut scores on tests of educational achievement , Princeton, NJ : Educational Testing Service .
   Google Scholar