Building reliable and generalizable clerkship competency assessments: Impact of ‘hawk-dove’ correction

References

• AAMC MSPE Task Force. 2016. AAMC MSPE task force recommendations. AAMC group on student affairs; [accessed 2019 Oct 22]. https://www.aamc.org/professional-development/affinity-groups/gsa/medical-student-performance-evaluation.
   Google Scholar
• Accreditation Council for Graduate Medical Education [ACGME]. n.d. Milestones. http://www.acgme.org/What-We-Do/Accreditation/Milestones/Overview.
   Google Scholar
• Brennan RL. 2001. Generalizability theory. New York (NY): Springer-Verlag.
   Google Scholar
• Epstein RM. 2007. Assessment in medical education. N Engl J Med. 356(4):387–396.
   PubMed Web of Science ®Google Scholar
• Gauthier G, St‐Onge C, Tavares W. 2016. Rater cognition: review and integration of research findings. Med Educ. 50(5):511–522.
   PubMed Web of Science ®Google Scholar
• Gingerich A, Kogan J, Yeates P, Govaerts M, Holmboe E. 2014. Seeing the 'black box' differently: assessor cognition from three research perspectives. Med Educ. 48(11):1055–1068.
   PubMed Web of Science ®Google Scholar
• Gingerich A, Regehr G, Eva KW. 2011. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 86(10):S1–S7.
   PubMed Web of Science ®Google Scholar
• Green M, Jones P, Thomas JXJ. 2009. Selection criteria for residency: results of a national program directors survey. Acad Med. 84(3):362–367.
   PubMed Web of Science ®Google Scholar
• Griffeth B, Wiederman M. 2018. Faculty development effects on clerkship grades. Clin Teach. 15(2):151–155.
   PubMed Web of Science ®Google Scholar
• Harasym PH, Woloschuk W, Cunning L. 2008. Undesired variance due to examiner stringency/leniency effect in communication skill scores assessed in OSCEs. Adv Health Sci Educ Theory Pract. 13(5):617–632.
   PubMed Web of Science ®Google Scholar
• Holmboe ES. 2015. Realizing the promise of competency-based medical education. Acad Med. 90(4):411–413.
   PubMed Web of Science ®Google Scholar
• Houston WM, Raymond MR, Svec JC. 1991. Adjustments for rater effects in performance assessment. Appl Psychol Measurement. 15(4):409–421.
   Web of Science ®Google Scholar
• Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. 2011. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ. 45(10):1048–1060.
   PubMed Web of Science ®Google Scholar
• Kogan JR, Hess BJ, Conforti LN, Holmboe ES. 2010. What drives faculty ratings of residents’ clinical skills? The impact of faculty’s own clinical skills. Acad Med. 85(10):S25.
   PubMedGoogle Scholar
• Kreiter CD, Ferguson K, Lee WC, Brennan RL, Densen P. 1998. A generalizability study of a new standardized rating form used to evaluate students’ clinical clerkship performances. Acad Med. 73(12):1294.
   PubMed Web of Science ®Google Scholar
• Kreiter CD, Zaidi NL, Park YS. 2020. Chapter 4, generalizabililty theory. In: Yudkowsky R, Park YS, Downing SM, editors. Assessment in health professions education. 2nd ed. New York (NY): Routledge; p. 51–69.
   Google Scholar
• Lineberry M. 2020. Chapter 2, validity and quality. In: Yudkowsky R, Park YS, Downing SM, editors. Assessment in health professions education. 2nd ed. New York (NY): Routledge; p. 17-32.
   Google Scholar
• Lockyer J, Carraccio C, Chan MK, Hart D, Smee S, Touchie C, Holmboe ES, Frank JR, ICBME Collaborators. 2017. Core principles of assessment in competency-based medical education. Med Teach. 39(6):609–616.
   PubMed Web of Science ®Google Scholar
• McManus I, Thompson M, Mollon J. 2006. Assessment of examiner leniency and stringency ('hawk-dove effect') in the MRCP(UK) clinical examination (PACES) using multi-facet Rasch modelling. BMC Med Educ. 6(1):42–22.
   PubMedGoogle Scholar
• Moser S, Mayans L, Davis N. 2017. Improving interrater reliability of medical student assessment by clinical supervisors. MedEdPORTAL. 13:10609.
   PubMedGoogle Scholar
• National Resident Matching Program. 2016. Results of the 2016 NRMP program director survey; [accessed 2018 May 24]. www.nrmp.org/wp-content/uploads/…/NRMP-2016-Program-Director-Survey.pdf
   Google Scholar
• Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, Galbraith R, Hays R, Kent A, Perrott V, et al. 2011. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 33(3):206–214.
   PubMed Web of Science ®Google Scholar
• Ogden PE, Edwards J, Howell M, Via RM, Song J. 2008. The effect of two different faculty development interventions on third-year clerkship performance evaluations. Fam Med. 40(5):333–338.
   PubMed Web of Science ®Google Scholar
• Park YS. 2020. Chapter 3: reliability. In: Yudkowsky R, Park YS, Downing SM, editors. Assessment in health professions education. 2nd ed. New York (NY): Routledge; p. 33–50.
   Google Scholar
• Park YS, Hicks PJ, Carraccio C, Margolis M, Schwartz A, PMAC Module 2 Study Group. 2018. Does incorporating a measure of clinical workload improve workplace-based assessment scores? Insights for measurement precision and longitudinal score growth from ten pediatrics residency programs. Acad Med. 93(11S):S21–S29.
   PubMedGoogle Scholar
• Park YS, Xing K. 2019. Rater model using signal detection theory for latent differential rater functioning. Multivariate Behav Res. 54(4):492–504.
   PubMed Web of Science ®Google Scholar
• Raymond MR, Harik P, Clauser BE. 2011. The impact of statistically adjusting for rater effects on conditional standard errors for performance ratings. Appl Psych Measurement. 35(3):235–246.
   Web of Science ®Google Scholar
• Roberts C, Rothnie I, Zoanetti N, Crossley J. 2010. Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini-interview? Med Educ. 44(7):690–698.
   PubMed Web of Science ®Google Scholar
• van der Vleuten CP, Schuwirth LW, Driessen EW, Dijkstra J, Tigelaar D, Baartman LK, van Tartwijk J. 2012. A model for programmatic assessment fit for purpose. Med Teach. 34(3):205–214.
   PubMed Web of Science ®Google Scholar
• Williams RG, Klamen DA, McGaghie WC. 2003. SPECIAL ARTICLE: cognitive, social and environmental sources of bias in clinical performance ratings. Teach Learn Med. 15(4):270–292.
   PubMed Web of Science ®Google Scholar
• Wimmers PF, Kanter SL, Splinter TAW, Schmidt HG. 2008. Is clinical competence perceived differently for student daily performance on the wards versus clerkship grading? Adv Health Sci Educ Theory Pract. 13(5):693–707.
   PubMed Web of Science ®Google Scholar
• Yeates P, O’Neill P, Mann K, Eva K. 2013. Seeing the same thing differently: mechanisms that contribute to assessor differences in directly-observed performance assessments. Adv Health Sci Educ Theory Pract. 18(3):325–341.
   PubMed Web of Science ®Google Scholar
• Zaidi NLB, Kreiter CD, Castaneda PR, Schiller JH, Yang J, Grum CM, Hammoud MM, Gruppen LD, Santen SA. 2018. Generalizability of competency assessment scores across and within clerkships: how students, assessors, and clerkships matter. Acad Med. 93(8):1212–1217.
   PubMed Web of Science ®Google Scholar