Comparing Yes/No Angoff and Bookmark Standard Setting Methods in the Context of English Assessment

REFERENCES

• Inc , ACT . 2007 . Developing achievement levels on the 2006 National Assessment of Educational Progress in grade 12 economics: Technical report , Iowa City , IA : Author .
   Google Scholar
• Angoff , W. H. 1971 . “ Scales, norms, and equivalent scores ” . In Educational measurement , Edited by: Thorndike , R. L. 508 – 600 . Washington , DC : American Council on Education .
   Google Scholar
• Bechger , T. M. , Kuijper , H. and Maris , G. 2009 . Standard setting in relation to the common European framework of reference for languages: The case of the state examination of Dutch as a second language . Language Assessment Quarterly , 6 : 126 – 150 .
   Web of Science ®Google Scholar
• Bejar , I. I. 1983 . Subject matter experts' assessment of item statistics . Applied Psychological Measurement , 7 : 303 – 310 .
   Web of Science ®Google Scholar
• Berk , R. A. 1986 . A consumer's guide to setting performance standards on criterion referenced tests . Review of Educational Research , 56 : 137 – 172 .
   Web of Science ®Google Scholar
• Buckendahl , C. W. , Smith , R. W. , Impara , J. C. and Plake , B. S. 2002 . A comparison of Angoff and Bookmark standard setting methods . Journal of Educational Measurement , 39 : 253 – 263 .
   Web of Science ®Google Scholar
• Cizek , G. J. and Bunch , M. B. 2007 . Standard setting: A guide to establishing and evaluating performance standards on tests , Thousand Oaks , CA : Sage .
   Google Scholar
• Cohen , J. 1988 . Statistical power analysis for the behavioral sciences , 2nd , Hillsdale , NJ : Erlbaum .
   Google Scholar
• Cohen , J. 1992 . A power primer . Psychological Bulletin , 112 : 155 – 159 .
   PubMed Web of Science ®Google Scholar
• Colton , D. A. and Hecht , J. T. A prelimary report on a study of three techniques for setting minimum passing scores . Symposium presentation at the annual meeting of the National Council on Measurement in Education . Los Angeles , CA .
   Google Scholar
• Council of Europe . 2001 . Common European Framework of Reference for Languages: Learning, teaching, assessment , Cambridge , UK : Cambridge University Press .
   Google Scholar
• Council of Europe. (2009). Relating language examinations to the common European Framework of Reference for Languages: Learning, teaching, assessment. A manual. http://www.coe.int/t/DG4/Portfolio/documents/Manual%20Revision%20-%20proofread%20-%20FINAL.pdf (http://www.coe.int/t/DG4/Portfolio/documents/Manual%20Revision%20-%20proofread%20-%20FINAL.pdf) (Accessed: 10 February 2012 ).
   Google Scholar
• Ferdous , A. A. and Plake , B. S. 2005 . Understanding the factors that influence decisions of panelists in a standard setting study . Applied Measurement in Education , 18 : 257 – 267 .
   Web of Science ®Google Scholar
• Figueras , N. , North , B. , Takala , S. , Verhelst , N. and Van Avermaet , P. 2005 . Relating examinations to the Common European Framework: A manual . Language Testing , 22 : 261 – 279 .
   Google Scholar
• Hambleton , R. K. 2001 . “ Setting performance standards on educational assessments nad criteria for evaluating the process ” . In Setting performance standardsL Concepts, methods, and perspectives , Edited by: Cizek , G. J. 89 – 116 . Mahwah , NJ : Erlbaum .
   Google Scholar
• Harsch , C. and Rupp , A. 2011 . Designing and scaling level-specific CEFR Writing Tasks . Language Assessment Quarterly , 8 : 1 – 33 .
   Web of Science ®Google Scholar
• Kane , M. 1994 . Validating the performance standards associated with passing scores . Review of Educational Research , 64 : 425 – 461 .
   Web of Science ®Google Scholar
• Kaftandjieva , F. and Takala , S. 2002 . “ Council of Europe scales of language proficiency: A validation study ” . In Common European Framework of Reference for Languages: Learning, teaching, assessment. Case studies , Edited by: Alderson , J. C. 106 – 129 . Strasbourg , France : Council of Europe .
   Google Scholar
• Impara , J. C. and Plake , B. S. 1998 . Teachers' ability to estimate item difficulty: A test of the assumptions in the Angoff Standard Setting Method . Journal of Educational Measurement , 35 : 69 – 81 .
   Web of Science ®Google Scholar
• Landis , J. R. and Koch , G. G. 1977 . Measurement of observer agreement for categorical data . Biometrics , 33 : 159 – 174 .
   PubMed Web of Science ®Google Scholar
• Lewis , D.M. , Mitzel , H. C. and Green , D. R. IRT based standard setting procedures using behavioral anchoring . Symposium conducted at the Council of Chief State School Officers National Conference on Large Scale Assessment . Phoenix , AZ . Standard setting: A bookmark approach. In D. R. Green (Chair) ,
   Google Scholar
• Lord , F. M. 1980 . Applications of item response theory to practical testing problems , Hillsdale , NJ : Erlbaum .
   Google Scholar
• Meara , K. C. , Hambleton , R. K. and Sireci , S. G. 2001 . Setting and validating standards on professional licensure and certification exams: A survey of current practice . CLEAR Exam Review , 7 ( 2 ) : 17 – 23 .
   Google Scholar
• Mehrens , W. A. Methodological issues in standard setting for educational exams . Proceedings of the joint conference on standard setting for large scale assessments of the National Assessment Governing Board (NAGB) and the National Center for Educational Statistics (MCES), Volume II . Washington , DC . pp. 221 – 263 . U.S. Government Printing Office .
   Google Scholar
• Mills , C. N. and Melican , G. J. 1988 . Estimating and adjusting cutoff scores: Features of selected methods . Applied Measurement in Education , 1 : 261 – 275 .
   Google Scholar
• O'Neill , T. R. , Buckendahl , C. W. , Plake , B. S. and Taylor , L. 2007 . Recommending a nursing specific passing standard for the IELTS examination . Language Assessment Quarterly , 4 : 295 – 317 .
   Google Scholar
• Papageorgiou , S. 2010 . Investigating the decision-making process of standard setting participants . Language Testing , 27 : 261 – 282 .
   Web of Science ®Google Scholar
• Perie , M. Angoff and Bookmark methods . Workshop presented at the annual meeting of the National Council on Measurement in Education . Montreal , Canada.
   Google Scholar
• Peterson , C. H. , Schulz , E. M. and Engelhard , J. G. 2011 . Reliability and validity of bookmark-based methods for standard setting: comparisons to Angoff-based methods in the national assessment of educational progress . Educational Measurement: Issues and Practice , 30 ( 2 ) : 3 – 14 .
   Google Scholar
• Plake , B. S. 1998 . Setting performance standards for professional licensure and certification . Applied Measurement in Education , 11 : 65 – 80 .
   Web of Science ®Google Scholar
• Shepard , L. A. Implication for standard setting of the National Academy of Education evaluation of the National Assessment of Educational Progress achievement levels . Proceedings of the joint conference on standard setting for large scale assessments of the National Assessment Governing Board (NAGB) and the National Center for Educational Statistics (NCES), Volume II . Washington , DC . pp. 143 – 160 . U.S. Government Printing Office .
   Google Scholar
• Shepard , L. , Glaser , R. , Linn , R. and Bohrnstedt , G. 1993 . Setting performance standards for student achievement , Stanford , CA : National Academy of Education .
   Google Scholar
• Sireci , S. G. and Biskin , B. J. 1992 . Measurement practices in national licensing examination programs: A survey . CLEAR Exam Review , 3 : 21 – 25 .
   Google Scholar
• Sireci , S. G. , Hauger , J. , Wells , C. S. , Lewis , C. , Delton , J. and Zenisky , A. Evaluation of the standard setting on the 2005 Grade 12 National Assessment of Educational Progress math test . Center for Educational Assessment Research Report No. 618 . 2007 . Amherst: Center for Educational Assessment, University of Massachusetts Amherst
   Google Scholar
• Sireci , S. G. , Hauger , J. B. , Wells , C. S. , Shea , C. and Zenisky , A. L. 2009 . Evaluation of the standard setting on the 2005 Grade 12 National Assessment of Educational Progress mathematics test . Applied Measurement in Education , 22 : 339 – 358 .
   Web of Science ®Google Scholar
• van der Linden , W. J. , Veldkamp , B. P. and Carlson , J. E. 2004 . Optimizing balanced incomplete block designs for large-scale educational assessments . Applied Psychological Measurement , 28 : 317 – 331 .
   Web of Science ®Google Scholar
• Van Moere , A. 2006 . Validity evidence in a group oral test . Language Testing , 23 : 411 – 440 .
   Google Scholar
• Wang , N. 2003 . Use of the Rasch IRT model in standard setting: An item mapping method . Journal of Educational Measurement , 40 : 231 – 253 .
   Web of Science ®Google Scholar
• Werner , E. 1978 . Cutting scores for occupational licensing tests, manual of considerations and methods , Sacramento : California Department of Consumer Affairs .
   Google Scholar
• Williams , N. and Schulz , E. M. An investigation of response probability values used in standard setting . Paper presented at the meeting of the National Council on Measurement in Education . Montreal , Canada.
   Google Scholar
• Wood , R. , Wilson , D. T. , Gibbons , R. D. , Schilling , S. G. , Muraki , E. and Bock , R. D. 2003 . TESTFACT 4 for Windows: Test Scoring, Item Statistics, and Full-information Item Factor Analysis [Computer software] , Lincolnwood , IL : Scientific Software International .
   Google Scholar
• Yin , P. , Schulz , M. and Sconing , J. A comparison of cut scores and cut score variability from Angoff-based and Bookmark procedure in standard setting . Paper presented at the meeting of the National Council on Measurement in Education . San Diego , CA .
   Google Scholar