The Effect of Data Format on Integration of Performance Data Into Angoff Judgments

REFERENCES

• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education . 1999 . Standards for educational and psychological tests , Washington , DC : American Educational Research Association .
   Google Scholar
• Angoff , W. H. 1971 . “ Scales, norms, and equivalent scores ” . In Educational measurement (2nd ed. , Edited by: Thorndike , R. L. 508 – 600 . Washington , DC : American Council on Education .
   Google Scholar
• Boursicot , K. A. M. , Roberts , T. E. and Pell , G. 2006 . Standard setting for clinical competence at graduation from medical school: A comparison of passing scores across five medical schools . Advances in Health Sciences Education , 11 : 173 – 183 .
   PubMed Web of Science ®Google Scholar
• Busch , J. C. and Jaeger , R. M. 1990 . Influence of type of judge, normative information, and discussion on standards recommended for the National Teacher Examinations . Journal of Educational Measurement , 27 : 145 – 163 .
   Web of Science ®Google Scholar
• Clauser , B. E. , Harik , P. , Margolis , M. J. , McManus , I. C. , Mollon , J. , Chis , L. and Williams , S. 2009 . Empirical evidence for the evaluation of performance standards estimated using the Angoff procedure . Applied Measurement in Education, 22 , : 1 – 21 .
   Google Scholar
• Clauser , B. E. , Mee , J. , Baldwin , S. G. , Margolis , M. J. and Dillon , G. F. 2009 . Judges’ use of examinee performance data in an Angoff standard-setting exercise for a medical licensing examination: An experimental study . Journal of Educational Measurement , 46 ( 4 ) : 390 – 407 .
   Web of Science ®Google Scholar
• Clauser , B. E. , Swanson , D. B. and Harik , P. 2002 . A multivariate generalizability analysis of the impact of training and examinee performance information on judgments made in an Angoff-style standard-setting procedure . Journal of Educational Measurement , 39 : 269 – 290 .
   Web of Science ®Google Scholar
• Cross , L. H. , Impara , J. C. , Frary , R. B. and Jaeger , R. M. 1984 . A comparison of three methods for establishing minimum standards on the National Teacher Examinations . Journal of Educational Measurement , 21 : 113 – 129 .
   Web of Science ®Google Scholar
• Davis-Becker , S. L. , Buckendahl , C. W. and Gerrow , J. 2011 . Evaluating the bookmark standard setting method: The impact of random item ordering . International Journal of Testing , 11 : 24 – 37 .
   Google Scholar
• Dillon , G. F. 1996 . The expectations of standard setting judges . CLEAR Exam Review , 7 ( 2 ) : 22 – 25 .
   Google Scholar
• Goodwin , L. D. 1999 . Relations between observed item difficulty levels and Angoff minimum passing levels for a group of borderline candidates . Applied Measurement in Education , 12 ( 1 ) : 13 – 28 .
   Web of Science ®Google Scholar
• Hambleton , R. K. 2001 . “ Setting performance standards on educational assessments and criteria for evaluating the process ” . In Standard setting: Concepts, methods and perspectives , Edited by: Cizek , G. 159 – 173 . Mahwah , NJ : Erlbaum Associates .
   Google Scholar
• Hambleton , R. K. , Brennan , R. L. , Brown , W. , Dodd , B. , Forsyth , R. A. , Mehrens , W. A. and … Zwick , R. 2000 . A response to “Setting reasonable and useful performance standards” in the National Academy of Sciences’ grading the nation's report card . Educational Measurement: Issues and Practice , 19 ( 2 ) : 5 – 14 .
   Google Scholar
• Hambleton , R. K. and Pitoniak , M. J. 2006 . “ Setting performance standards ” . In Educational measurement (4th ed. , Edited by: Brennan , R. L. 433 – 470 . Westport , CT : American Council on Education/Praeger .
   Google Scholar
• Hambleton , R. K. , Swaminathan , H. and Jane Rogers , H. 1991 . MMSS Fundamentals of item response theory , Newbury Park , CA : Sage .
   Google Scholar
• Hurtz , G. M. and Auerbach , M. A. 2003 . A meta-analysis of the effects of modifications to the Angoff method on cutoff scores and judgment consensus . Educational and Psychological Measurement , 63 : 584 – 601 .
   Web of Science ®Google Scholar
• Kane , M. 1994 . Validating the performance standards associated with passing scores . Review of Educational Research , 64 : 425 – 461 .
   Web of Science ®Google Scholar
• Maurer , T. J. and Alexander , R. A. 1992 . Methods of improving employment test critical scores derived by judging test content: A review and critique . Personnel Psychology , 45 : 727 – 762 .
   Web of Science ®Google Scholar
• Maurer , T. J. , Alexander , R. A. , Callahan , C. M. , Bailey , J. J. and Dambrot , F. H. 1991 . Methodological and psychometric issues in setting cutoff scores using the Angoff method . Personnel Psychology , 44 : 235 – 262 .
   Web of Science ®Google Scholar
• Norcini , J. J. , Shea , J. A. and Kanya , D. T. 1988 . The effects of various factors on standard setting . Journal of Educational Measurement , 25 : 57 – 65 .
   Web of Science ®Google Scholar
• Novakovic , N. 2008 . Thinking on the edge: The influence of discussion and statistical data on awarders’ perceptions of borderline candidates in an Angoff awarding meeting . International Journal of Training Research , 6 : 74 – 100 .
   Google Scholar
• Papageorgiou , S. 2010 . Investigating the decision-making process of standard setting participants . Language Testing , 27 : 261 – 282 .
   Web of Science ®Google Scholar
• Plake , B. S. , Impara , J. and Potenza , M. 1994 . Content specificity of expert judges in a standard setting study . Journal of Educational Measurement , 31 : 339 – 347 .
   Web of Science ®Google Scholar
• Raymond , M. R. and Reid , J. B. 2001 . “ Who made thee a judge: Selecting and training participants for standard setting ” . In Standard setting: Concepts, methods, and perspectives , Edited by: Cizek , G. 119 – 157 . Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Reckase , M. D. 2001 . “ Innovative methods for helping standard setting participants to perform their task: The role of feedback regarding consistency, accuracy, and impact ” . In Standard setting: Concepts, methods and perspectives , Edited by: Cizek , G. 159 – 173 . Mahwah , NJ : Erlbaum Associates .
   Google Scholar
• Shepard , L. A. 1995 . “ Implications for standard setting of the National Academy of Educational Evaluation of the National Assessment of Educational Progress achievement levels ” . In Proceedings of the joint conference on standard setting for large-scale assessments of the National Assessment Governing Board and the National Center for Educational Statistics , 143 – 159 . Washington , DC : U.S. Government Printing Office .
   Google Scholar
• Truxillo , D. M. , Donahue , L. M. and Sulzer , J. L. 1996 . Setting cutoff scores for personnel selection tests: Issues, illustrations, and recommendations . Human Performance , 9 : 275 – 295 .
   Web of Science ®Google Scholar