Selecting and Simplifying: Rater Performance and Behavior When Considering Multiple Competencies

References

• Huwendiek S, Mennin S, Dern P, Friedman Ben-David M, Van Der Vleuten C, Tönshoff B, et al. Expertise, needs and challanges of medical educators: Results of an international web survey. Medical Teacher 2010;32:912–8.
   PubMed Web of Science ®Google Scholar
• Eva KW, Hodges BD. Scylla or Charybdis? Can we navigate between objectification and judgement in assessment? Medical Education 2012;46:914–9.
   PubMed Web of Science ®Google Scholar
• Hodges B. Assessment in the post-psychometric era: Learning to love the subjective and collective. Medical Teacher 2013;35:564–8.
   PubMed Web of Science ®Google Scholar
• Gingerich A, Regehr G, Eva KW. Rater-based assessments as social judgments: Rethinking the etiology of rater errors. Academic Medicine 2011;86:S1–S7.
   PubMed Web of Science ®Google Scholar
• Yeates P, O'Neill P, Mann K, Eva KW. Seeing the same thing differently: Mechanisms that contribute to assessor differences in directly-observed performance assessments. Advances in Health Sciences Education 2012;18:325–41.
   PubMed Web of Science ®Google Scholar
• Govaerts M, Schuwirth L, van der Vleuten C, Muijtjens A. Workplace-based assessment: Effects of rater expertise. Advances in Health Sciences Education 2012;16:151–65.
   Web of Science ®Google Scholar
• Govaerts M, van de Wiel M, Schuwirth L, van der Vleuten C. Workplace-based assessment: Raters' performance theories and constructs. Advances in Health Sciences Education 2011;18:375–96.
   Web of Science ®Google Scholar
• Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. Opening the black box of clinical skills assessment via observation: A conceptual model. Medical Education 2011;45:1048–60.
   PubMed Web of Science ®Google Scholar
• Williams RG, Verhulst S, Colliver JA, Dunnington GL. Assuring the reliability of resident performance appraisals: More items or more observations? Surgery 2005;137:141–7.
   PubMed Web of Science ®Google Scholar
• Eva KW, Solomon P, Neville AJ, Ladouceur M, Kaufman K, Walsh A, et al. Using a sampling strategy to address psychometric challenges in tutorial-based assessments. Advances in Health Sciences Education 2007;12:19–33.
   PubMed Web of Science ®Google Scholar
• Frank JR. (Ed.). The CanMEDS 2005 Physician competency framework. Better standards. Better physicians. Better care. Ottawa, Canada: The Royal College of Physicians and Surgeons of Canada, 2005.
   Google Scholar
• Accreditation Council for Graduate Medical Education. Common program requirements. Chicago, IL: ACGME, 2003. Available at: http://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf. Accessed February 24, 2015.
   Google Scholar
• Tavares W, Eva KW. Exploring the impact of mental workload on rater-based assessments. Advances in Health Sciences Education 2012;18:291–303.
   PubMed Web of Science ®Google Scholar
• Byrne A, Tweed N, Halligan C. A pilot study of the mental workload of OSCE examiners. Medical Education 2014;48:262–7.
   PubMed Web of Science ®Google Scholar
• Tavares W, Eva KW. The impact of rating demands on rater-based assessment of clinical competence. Education in Primary Care 2014;25:308–18.
   PubMedGoogle Scholar
• Tavares W, Boet S, Theriault R, Mallette T, Eva KW. Global rating scale for the assessment of paramedic clinical competence. Prehospital Emergency Care 2012;17:57–67.
   PubMed Web of Science ®Google Scholar
• Tavares W, LeBlanc VR, Mausz J, Sun V, Eva KW. Simulation-based assessment of paramedics and performance in real clinical contexts. Prehospital Emergency Care 2014;18:116–22.
   PubMed Web of Science ®Google Scholar
• Wind LA, Van Dalen J, Muijtjens A, Rethans JJ. Assessing simulated patients in an educational setting: The MaSP (Maastricht Assessment of Simulated Patients). Medical Education 2004;38:39–44.
   PubMed Web of Science ®Google Scholar
• Paas F, van Merriënboer JJG. Measurement of cognitive load in instructional research. Perceptual and Motor Skills 1994;79:419–30.
   PubMed Web of Science ®Google Scholar
• Brennan RL. Generalizability theory. Berlin, Germany: Springer Verlag, 2001.
   Google Scholar
• Elo S, Kyngäs H. The qualitative content analysis process. Journal of Advanced Nursing 2008;62:107–15.
   PubMed Web of Science ®Google Scholar
• Charmaz K. Constructing grounded theory: A practical guide through qualitative analysis: London, UK: Sage, 2006.
   Google Scholar
• van der Vleuten CPM, Schuwirth LW, Scheele F, Driessen E, Hodges B. The assessment of professional competence: Building blocks for theory development. Best Practice & Research Clinical Obstetrics & Gynaecology 2010;24:703–19.
   PubMed Web of Science ®Google Scholar
• Baddeley A. Working memory. Science 1992;31:556–9.
   Web of Science ®Google Scholar
• Wickens CD, Carswell C. Handbook of human factors and ergonomics, third edition. (G Salvendy. Ed.). Hoboken, NJ: Wiley, 2006.
   Google Scholar
• Sweller J. Cognitive load during problem solving: Effects on learning. Cognitive Science 1988;12:257–85.
   Web of Science ®Google Scholar
• Melchers KG, Kleinmann M, Prinz MA. Do assessors have too much on their plates? The effects of simultaneously rating multiple assessment center candidates on rating quality. International Journal of Selection and Assessment 2010;18:329–41.
   Web of Science ®Google Scholar
• Williams R, Klamen D, McGaghie W. Special article: Cognitive, social and environmental sources of bias in clinical performance ratings. Teaching and Learning in Medicine 2003;15:270–92.
   PubMed Web of Science ®Google Scholar
• Hodges B. Validity and the OSCE. Medical Teacher 2003;25:250–4.
   PubMed Web of Science ®Google Scholar
• Streiner D, Norman G. Health measurement scales: a practical guide to their development and use, fourth ed. New York, NY: Oxford Univeristy Press, 2008.
   Google Scholar
• Downing S. Validity: On the meaningful interpretation of assessment data. Medical Education 2003;37:830–7.
   PubMed Web of Science ®Google Scholar
• Landy FJ, Farr JL. Performance rating. Psychological Bulletin 1980;87:72–107.
   Web of Science ®Google Scholar
• Gingerich A, Kogan J, Yeates P, Govaerts M, Holmboe E. Seeing the ‘black box’ differently: Assessor cognition from three different research perspectives. Medical Education 2014;48:1055–68.
   PubMed Web of Science ®Google Scholar
• Norman G, Vleuten C, Graaff E. Pitfalls in the pursuit of objectivity: Issues of validity, efficiency and acceptability. Medical Education 1991;25:119–26.
   PubMed Web of Science ®Google Scholar
• Crossley J, Humphris G, Jolly B. Assessing health professionals. Medical Education 2002;36:800–4.
   PubMed Web of Science ®Google Scholar
• van der Vleuten CPM. The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education 1996;1:41–67.
   PubMed Web of Science ®Google Scholar
• Nisbett RE, Wilson TD. Telling more than we can know: Verbal reports on mental processes. Psychological Review 1977;84:231–59.
   Web of Science ®Google Scholar
• van Merrienboer JJG, Sweller J. Cognitive load theory in health professional education: Design principles and strategies. Medical Education 2010;44:85–93.
   PubMed Web of Science ®Google Scholar