The pursuit of fairness in assessment: Looking beyond the objective

References

• Apramian T, Cristancho S, Sener A, Lingard L. 2018. How do thresholds of principle and preference influence surgeon assessments of learner performance? Ann Surg. 268(2):385–390.
   PubMed Web of Science ®Google Scholar
• Apramian T, Cristancho S, Watling C, Ott M, Lingard L. 2015. Thresholds of principle and preference: exploring procedural variation in postgraduate surgical education. Acad Med. 90(11 Suppl):S70–S76.
   PubMed Web of Science ®Google Scholar
• Apramian T, Cristancho S, Watling C, Ott M, Lingard L. 2016. “Staying in the game”: how procedural variation shapes competence judgments in surgical education. Acad Med. 91(11):S37–S43.
   PubMed Web of Science ®Google Scholar
• Bacon R, Williams LT, Grealish L, Jamieson M. 2015. Competency-based assessment for clinical supervisors: design-based research on a web-delivered program. JMIR Res Protoc. 4(1):e26.
   PubMedGoogle Scholar
• Boulet JR, Durning SJ. 2019. What we measure … and what we should measure in medical education. Med Educ. 53(1):86–94.
   PubMed Web of Science ®Google Scholar
• Boursicot K. 2020. Consensus statement reports: performance assessment. Ottawa; Kuala Lumpur.
   Google Scholar
• Carmody JB. 2019. On residency selection and the quantitative fallacy. J Grad Med Educ. 11(4):420–421.
   PubMedGoogle Scholar
• Cilliers FJ, Schuwirth LW, Adendorff HJ, Herman N, Van der Vleuten CP. 2010. The mechanism of impact of summative assessment on medical students' learning. Adv Health Sci Educ Theory Pract. 15(5):695–715.
   PubMed Web of Science ®Google Scholar
• Cilliers FJ, Schuwirth LW, Herman N, Adendorff HJ, van der Vleuten CP. 2012. A model of the pre-assessment learning effects of summative assessment in medical education. Adv Health Sci Educ Theory Pract. 17(1):39–53.
   PubMed Web of Science ®Google Scholar
• Cohen GS, Blumberg P, Ryan NC, Sullivan PL. 1993. Do final grades reflect written qualitative evaluations of student performance? Teach Learn Med. 5(1):10–15.
   Google Scholar
• Dauphinee WD. 1995. Assessing clinical performance. Where do we stand and what might we expect? JAMA. 274(9):741–743.
   PubMed Web of Science ®Google Scholar
• Delandshere G, Petrosky AR. 1994. Capturing teachers' knowledge: performance assessment: a) and post-structuralist epistemology b) from a post-structuralist perspective c) and post-structuralism d) none of the above. Educ Res. 23(5):11–18.
   Google Scholar
• Dijkstra J, Galbraith R, Hodges BD, McAvoy PA, McCrorie P, Southgate LJ, Van der Vleuten CP, Wass V, Schuwirth LW. 2012. Expert validation of fit-for-purpose guidelines for designing programmes of assessment. BMC Med Educ. 12:20.
   PubMed Web of Science ®Google Scholar
• Dijkstra J, Van der Vleuten CP, Schuwirth LW. 2010. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. 15(3):379–393.
   PubMed Web of Science ®Google Scholar
• Downie R, Macnaughton J. 2013. In defence of professional judgement. Adv Psychiatr Treat. 15(5):322–327.
   PubMedGoogle Scholar
• Downing SM. 2004. Reliability: on the reproducibility of assessment data. Med Educ. 38(9):1006–1012.
   PubMed Web of Science ®Google Scholar
• Driessen E, Van der Vleuten C, Schuwirth L, van Tartwijk J, Vermunt J. 2005. The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: a case study. Med Educ. 39(2):214–220.
   PubMed Web of Science ®Google Scholar
• Durning SJ, Artino AR Jr., Schuwirth L, Van der Vleuten C. 2013. Clarifying assumptions to enhance our understanding and assessment of clinical reasoning. Acad Med. 88(4):442–448.
   PubMed Web of Science ®Google Scholar
• Durning SJ, Hanson J, Gilliland W, McManigle JM, Waechter D, Pangaro LN. 2010. Using qualitative data from a program director's evaluation form as an outcome measurement for medical school. Mil Med. 175(6):448–452.
   PubMed Web of Science ®Google Scholar
• Eva KW. 2008. On the limits of systematicity. Med Educ. 42(9):852–853.
   PubMed Web of Science ®Google Scholar
• Eva KW. 2015. Moving beyond childish notions of fair and equitable. Med Educ. 49(1):1–3.
   PubMed Web of Science ®Google Scholar
• Frambach JM, Van der Vleuten CP, Durning SJ. 2013. AM last page. Quality criteria in qualitative and quantitative research. Acad Med. 88(4):552.
   PubMedGoogle Scholar
• Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, Harris P, Glasgow NJ, Campbell C, Dath D, et al. 2010. Competency-based medical education: theory to practice. Med Teach. 32(8):638–645.
   PubMed Web of Science ®Google Scholar
• General Medical Council. 2017. Designing and maintaining postgraduate assessment programmes. London: General Medical Council.
   Google Scholar
• Gingerich A, Kogan J, Yeates P, Govaerts M, Holmboe E. 2014. Seeing the ‘black box’ differently: assessor cognition from three research perspectives. Med Educ. 48(11):1055–1068.
   PubMed Web of Science ®Google Scholar
• Gingerich A, Regehr G, Eva KW. 2011. Rater-based assessments as social judgments: rethinking the etiology of rater errors. Acad Med. 86(10 Suppl):S1–S7.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, Eva K, Regehr G. 2013. Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Acad Med. 88(10):1539–1544.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, Van der Vleuten CPM, Eva KW. 2017. The hidden value of narrative comments for assessment: a quantitative reliability analysis of qualitative data. Acad Med. 92(11):1617–1621.
   PubMed Web of Science ®Google Scholar
• Gipps C, Stobart G. 2009. Fairness in assessment. Educational assessment in the 21st century. Dordrecht (Netherlands): Springer; p. 105–118.
   Google Scholar
• Govaerts M, Van der Vleuten CP. 2013. Validity in work-based assessment: expanding our horizons. Med Educ. 47(12):1164–1174.
   PubMed Web of Science ®Google Scholar
• Govaerts MJ, Schuwirth LW, Van der Vleuten CP, Muijtjens AM. 2011. Workplace-based assessment: effects of rater expertise. Adv Health Sci Educ Theory Pract. 16(2):151–165.
   PubMed Web of Science ®Google Scholar
• Govaerts MJB, Van der Vleuten CPM, Holmboe ES. 2019. Managing tensions in assessment: moving beyond either-or thinking. Med Educ. 53(1):64–75.
   PubMed Web of Science ®Google Scholar
• Green SK, Johnson RL, Kim D-H, Pope NS. 2007. Ethics in classroom assessment practices: issues and attitudes. Teach Teach Educ. 23(7):999–1011.
   Web of Science ®Google Scholar
• Greenhalgh T, Howick J, Maskrey N, Evidence Based Medicine Renaissance Group. 2014. Evidence based medicine: a movement in crisis? BMJ. 348:g3725.
   PubMed Web of Science ®Google Scholar
• Greenhalgh T, Papoutsi C. 2018. Studying complexity in health services research: desperately seeking an overdue paradigm shift. BMC Med. 16(1):95.
   PubMed Web of Science ®Google Scholar
• Harden RM, Gleeson F. 1979. Assessment of clinical competence using an objective structured clinical examination (OSCE). Medical Educ. 13(1):39–54.
   Web of Science ®Google Scholar
• Hodges B. 2013. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 35(7):564–568.
   PubMed Web of Science ®Google Scholar
• Jones A. 1999. The place of judgement in competency-based assessment. J Vocat Educ. 51(1):145–160.
   Google Scholar
• Kaldjian LC. 2010. Teaching practical wisdom in medicine through clinical judgement, goals of care, and ethical reasoning. J Med Ethics. 36(9):558–562.
   PubMed Web of Science ®Google Scholar
• Kogan JR, Conforti L, Bernabeo E, Iobst W, Holmboe E. 2011. Opening the black box of clinical skills assessment via observation: a conceptual model. Med Educ. 45(10):1048–1060.
   PubMed Web of Science ®Google Scholar
• Kogan JR, Hess BJ, Conforti LN, Holmboe ES. 2010. What drives faculty ratings of residents' clinical skills? The impact of faculty's own clinical skills. Acad Med. 85(10 Suppl):S25–S28.
   PubMed Web of Science ®Google Scholar
• Marchese MC. 1992. Clinical versus actuarial prediction: a review of the literature. Percept Mot Skills. 75(2):583–594.
   PubMed Web of Science ®Google Scholar
• McCready T. 2007. Portfolios and the assessment of competence in nursing: a literature review. Int J Nurs Stud. 44(1):143–151.
   PubMed Web of Science ®Google Scholar
• McNamara R. 1995. In retrospect: the tragedy and lessons of Vietnam. New York (NY). Times Books, Random House.
   Google Scholar
• Mennin S. 2010. Self-organisation, integration and curriculum in the complex world of medical education. Med Educ. 44(1):20–30.
   PubMed Web of Science ®Google Scholar
• Moulton CA, Regehr G, Mylopoulos M, MacRae HM. 2007. Slowing down when you should: a new model of expert judgment. Acad Med. 82(10 Suppl):S109–S116.
   PubMed Web of Science ®Google Scholar
• Murad MH. 2017. Clinical practice guidelines: a primer on development and dissemination. Mayo Clin Proc. 92(3):423–433.
   PubMed Web of Science ®Google Scholar
• Newble DI, Hoare J, Sheldrake PF. 1980. The selection and training of examiners for clinical examinations. Med Educ. 14(5):345–349.
   PubMed Web of Science ®Google Scholar
• Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, Galbraith R, Hays R, Kent A, Perrott V, et al. 2011. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 33(3):206–214.
   PubMed Web of Science ®Google Scholar
• Norcini J, Shea J. 1997. The credibility and comparability of standards. App Meas Educ. 10(1):39–59.
   Web of Science ®Google Scholar
• Norcini JJ, Blank LL, Arnold GK, Kimball HR. 1995. The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 123(10):795–799.
   PubMed Web of Science ®Google Scholar
• O'Mahony S. 2017. Medicine and the McNamara fallacy. J R Coll Physicians Edinb. 47(3):281–287.
   PubMed Web of Science ®Google Scholar
• Oudkerk Pool A, Govaerts MJB, Jaarsma D, Driessen EW. 2018. From aggregation to interpretation: how assessors judge complex data in a competency-based portfolio. Adv Health Sci Educ Theory Pract. 23(2):275–287.
   PubMed Web of Science ®Google Scholar
• Plsek PE, Greenhalgh T. 2001. Complexity science: the challenge of complexity in health care. BMJ. 323(7313):625–628.
   PubMed Web of Science ®Google Scholar
• Reed JE, Howe C, Doyle C, Bell D. 2018. Simple rules for evidence translation in complex systems: a qualitative study. BMC Med. 16(1):92.
   PubMed Web of Science ®Google Scholar
• Rotthoff T. 2018. Standing up for subjectivity in the assessment of competencies. GMS J Med Educ. 35(3):Doc29.
   PubMedGoogle Scholar
• Schuwirth LW, Van der Vleuten CP. 2006. A plea for new psychometric models in educational assessment. Med Educ. 40(4):296–300.
   PubMed Web of Science ®Google Scholar
• Shaw S, Nisbet I. 2021. Attitudes to fair assessment in the light of COVID-19. Matters. 31:6–21.
   Google Scholar
• Ten Cate O. 2017. Competency-based postgraduate medical education: past, present and future. GMS J Med Educ. 34(5):Doc69.
   PubMed Web of Science ®Google Scholar
• Ten Cate O, Billett S. 2014. Competency-based medical education: origins, perspectives and potentialities. Med Educ. 48(3):325–332.
   PubMed Web of Science ®Google Scholar
• Ten Cate O, Regehr G. 2019. The power of subjectivity in the assessment of medical trainees. Acad Med. 94(3):333–337.
   PubMed Web of Science ®Google Scholar
• Ten Cate O, Scheele F. 2007. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med. 82(6):542–547.
   PubMed Web of Science ®Google Scholar
• Tierney RD. 2013. Fairness in classroom assessment. In: SAGE handbook of research on classroom assessment. CA: SAGE Publications, Inc. Chapter 8. p. 125.
   Google Scholar
• Tierney RD. 2014. Fairness as a multifaceted quality in classroom assessment. Stud Educ Evaluation. 43:55–69.
   Web of Science ®Google Scholar
• Valentine N, Schuwirth L. 2019. Identifying the narrative used by educators in articulating judgement of performance. Perspect Med Educ. 8(2):83–89.
   PubMed Web of Science ®Google Scholar
• Van der Vleuten CP, Norman GR, De Graaff E. 1991. Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ. 25(2):110–118.
   PubMed Web of Science ®Google Scholar
• Van der Vleuten CP, Schuwirth LW. 2005. Assessing professional competence: from methods to programmes. Med Educ. 39(3):309–317.
   PubMed Web of Science ®Google Scholar
• Van der Vleuten CP, Schuwirth LW, Driessen EW, Govaerts MJ, Heeneman S. 2015. Twelve tips for programmatic assessment. Med Teach. 37(7):641–646.
   PubMed Web of Science ®Google Scholar
• Watling CJ. 2014. Unfulfilled promise, untapped potential: feedback at the crossroads. Med Teach. 36(8):692–697.
   PubMed Web of Science ®Google Scholar
• Wayne DB, Green M, Neilson EG. 2020. Medical education in the time of COVID-19. Sci Adv. 6(31):eabc7110.
   PubMed Web of Science ®Google Scholar
• Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. 1999. Clinical guidelines: potential benefits, limitations, and harms of clinical guidelines. BMJ. 318(7182):527–530.
   PubMed Web of Science ®Google Scholar
• Yankelovich D. 1972. Corporate priorities: a continuing study of the new demands on business. Stamford, CT: Yankelovich Inc.
   Google Scholar
• Yeates P, O'Neill P, Mann K, Eva K. 2013. Seeing the same thing differently: mechanisms that contribute to assessor differences in directly-observed performance assessments. Adv Health Sci Educ Theory Pract. 18(3):325–341.
   PubMed Web of Science ®Google Scholar
• Young JQ, Van Merrienboer J, Durning S, Ten Cate O. 2014. Cognitive load theory: implications for medical education: AMEE Guide No. 86. Med Teach. 36(5):371–384.
   PubMed Web of Science ®Google Scholar