The Quality of Assessment of Learning (Qual) Score: Validity Evidence for a Scoring System Aimed at Rating Short, Workplace-Based Comments on Trainee Performance

References

• Iobst WF, Sherbino J, Cate OT, et al. Competency-based medical education in postgraduate medical education. Med Teach. 2010;32(8):651–656. doi:10.3109/0142159X.2010.500709.
   PubMed Web of Science ®Google Scholar
• Caccia N, Nakajima A, Kent N. Competency-based medical education: the wave of the future. J Obstet Gynaecol Canada. 2015;37(4):349–353. doi:10.1016/S1701-2163(15)30286-3.
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Sherbino J, Englander R, Snell L, Frank JR. A call to action: the controversy of and rationale for competency-based medical education. Med Teach. 2017;39(6):574–581. doi:10.1080/0142159X.2017.1315067.
   PubMed Web of Science ®Google Scholar
• Ferguson PC, Caverzagie KJ, Nousiainen MT, Snell L. Changing the culture of medical training: an important step toward the implementation of competency-based medical education. Med Teach. 2017;39(6):599–602. doi:10.1080/0142159X.2017.1315079.
   PubMed Web of Science ®Google Scholar
• Van Melle E, Frank JR, Holmboe ES, et al. A core components framework for evaluating implementation of competency-based medical education programs; 2019. doi:10.1097/ACM.0000000000002743.
   Google Scholar
• Lockyer J, Carraccio C, Chan MK, et al. Core principles of assessment in competency-based medical education. Med Teach. 2017;39(6):609–616. doi:10.1080/0142159X.2017.1315082.
   PubMed Web of Science ®Google Scholar
• Chan TM, Sherbino J, Mercuri M. Nuance and noise: lessons learned from longitudinal aggregated assessment data. J Grad Med Educ. 2017;9(6):724–729. doi:10.4300/JGME-D-17-00086.1.
   PubMedGoogle Scholar
• McConnell M, Sherbino J, Chan TM. Mind the gap: the prospects of missing data. J Grad Med Educ. 2016;8(5):708–712. doi:10.4300/JGME-D-16-00142.1.
   PubMedGoogle Scholar
• Ellaway RH, Pusic MV, Galbraith RM, et al. Developing the role of big data and analytics in health professional education. Med Teach. 2014;36(3):216–222. doi:10.3109/0142159X.2014.874553.
   PubMed Web of Science ®Google Scholar
• Cirigliano MM, Guthrie C, Pusic MV, et al. Yes, and …” exploring the future of learning analytics in medical education. Teach Learn Med. 2017;29(4):368–372. doi:10.1080/10401334.2017.1384731.
   PubMed Web of Science ®Google Scholar
• Chan T, Sebok-Syer S, Thoma B, et al. Learning analytics in medical education assessment: the past, the present, and the future. AEM Educ Train. 2018;2(2):178–187. doi:10.1002/aet2.10087.
   PubMedGoogle Scholar
• Bates J, Schrewe B, Ellaway RH, et al. Embracing standardisation and contextualisation in medical education. Med Educ. 2019;53(1):15–24. doi:10.1111/medu.13740.
   PubMed Web of Science ®Google Scholar
• Holmboe ES, Sherbino J, Long DM, et al. The role of assessment in competency-based medical education. Med Teach. 2010;32(8):676–682. doi:10.3109/0142159X.2010.500704.
   PubMed Web of Science ®Google Scholar
• Pell G, Fuller R, Homer M, et al. How to measure the quality of the OSCE: a review of metrics AMEE guide no. 49. Med Teach. 2010;32(10):802–811. doi:10.3109/0142159X.2010.507716.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, van der Vleuten C, Eva KW, et al. Hedging to save face: a linguistic analysis of written comments on in-training evaluation reports. Adv in Health Sci Educ. 2016;21(1):175–188. doi:10.1007/s10459-015-9622-0.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, Regehr G, Lingard L, et al. Reading between the lines: faculty interpretations of narrative evaluation comments. Med Educ. 2015;49(3):296–306. doi:10.1111/medu.12637.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, Eva K, Regehr G. Do in-training evaluation reports deserve their bad reputations? A study of the reliability and predictive ability of ITER scores and narrative comments. Acad Med. 2013;88(10):1539–1544. doi:10.1097/ACM.0b013e3182a36c3d.
   PubMed Web of Science ®Google Scholar
• Hodges B. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 2013;35(7):564–568. doi:10.3109/0142159X.2013.789134.
   PubMed Web of Science ®Google Scholar
• Chan T, Sherbino J. The McMaster Modular Assessment Program (McMAP). Acad Med. 2015;90(7):900–905. doi:10.1097/ACM.0000000000000707.
   PubMed Web of Science ®Google Scholar
• Cheung WJ, Dudek N, Wood TJ, et al. Daily encounter cards—evaluating the quality of documented assessments. J Grad Med Educ. 2016;8(4):601–604. doi:10.4300/JGME-D-15-00505.1.
   PubMedGoogle Scholar
• Sherbino J, Kulasegaram K, Worster A, et al. The reliability of encounter cards to assess the CanMEDS roles. Adv in Health Sci Educ. 2013;18(5):987–996. doi:10.1007/s10459-012-9440-6.
   PubMed Web of Science ®Google Scholar
• Shayne P, Gallahue F, Rinnert S, et al. Reliability of a core competency checklist assessment in the emergency department: the standardized direct observation assessment tool. Acad Emerg Med. 2006;13(7):727–732. doi:10.1197/j.aem.2006.01.030.
   PubMed Web of Science ®Google Scholar
• Hauer KE, Chesluk B, Iobst W, et al. Reviewing residents’ competence. Acad Med. 2015;90(8):1084–1092. doi:10.1097/ACM.0000000000000736.
   PubMed Web of Science ®Google Scholar
• Sebok-Syer SS, Klinger DA, Sherbino J, et al. Mixed messages or miscommunication? Investigating the relationship between assessors? Workplace-based assessment scores and written comments. Acad Med. 2017;92(12):1774–1779. doi:10.1097/ACM.0000000000001743.
   PubMed Web of Science ®Google Scholar
• Bandiera G, Lendrum D. Daily encounter cards facilitate competency-based feedback while leniency bias persists. Can J Emerg Med. 2008;10(1):44–50. doi:7df099179d2d4584a5d93bc828c06366 [pii] doi:10.1017/S1481803500010009.
   Web of Science ®Google Scholar
• LaMantia J, Kane B, Yarris L, et al. Real-time inter-rater reliability of the council of emergency medicine residency directors standardized direct observation assessment tool. Acad Emerg Med. 2009;16(SUPPL. 2):51–57. doi:10.1111/j.1553-2712.2009.00593.x.
   PubMedGoogle Scholar
• Chan T, Mercuri M, Van Dewark K, et al. Managing multiplicity: conceptualizing physician cognition in multipatient environments. Acad Med. 2018;93(5):786–793. doi:10.1097/ACM.0000000000002081.
   PubMed Web of Science ®Google Scholar
• Chan TM, Dewark K, Van Sherbino J, et al. Failure to flow: an exploration of learning and teaching in busy, multi-patient environments using an interpretive description method. Perspect Med Educ. 2017;6(6):380–387. doi:10.1007/s40037-017-0384-7.
   PubMed Web of Science ®Google Scholar
• Warm E, Kelleher M, Kinnear B, et al. Feedback on feedback as a faculty development tool. J Grad Med Educ. 2018;10(3):354–355. doi:10.4300/JGME-D-17-00876.1.
   PubMedGoogle Scholar
• Dudek NL, Marks MB, Wood TJ, et al. Assessing the quality of supervisors’ completed clinical evaluation reports. Med Educ. 2008;42:816–822. doi:10.1111/j.1365-2923.2008.03105.x.
   PubMed Web of Science ®Google Scholar
• Woods R, Chan T, Thoma B, et al. Education scholarship in Canadian emergency medicine: the past, present, and future. Can J Emerg Med. 2018;20(2):164–166. doi:10.1017/cem.2018.19.
   Web of Science ®Google Scholar
• Chan TM, Gottlieb M, Sherbino J, et al. The ALiEM faculty incubator: a novel online approach to faculty development in education scholarship. Acad. Med. 2018;93(10):1497–1502. doi:10.1097/ACM.0000000000002309.
   PubMed Web of Science ®Google Scholar
• Messick S. Validity. ETS Research Report Series. 1987;(2):i–208. doi:10.1002/j.2330-8516.1987.tb00244.x
   Google Scholar
• Lineberry M. Validity and its threats. In: Downing SM, Haladyna TM, eds. Assessment in Health Professions Education. New York: Routledge; 2019;21–55.
   Google Scholar
• Worster A, Kulasegaram K, Carpenter CR, et al. Consensus conference follow-up: inter-rater reliability assessment of the Best Evidence in Emergency Medicine (BEEM) rater scale, a medical literature rating tool for emergency physicians. Acad Emerg Med. 2011;18(11):1193–1200. doi:10.1111/j.1553-2712.2011.01214.x.
   PubMed Web of Science ®Google Scholar
• Chan TM, Grock A, Paddock M, Kulasegaram K, Yarris LM, Lin M. Examining reliability and validity of an online score (ALiEM AIR) for rating free open access medical education resources. Ann Emerg Med. 2016;68(6):729–735. doi:10.1016/j.annemergmed.2016.02.018.
   PubMed Web of Science ®Google Scholar
• Monteiro S, Sullivan GM, Chan TM. Generalizability theory made simple(r): an introductory primer to G-studies. J Grad Med Educ. 2019;11(4):365–370. doi:10.4300/JGME-D-19-00464.1.
   PubMedGoogle Scholar
• Cooper S, Endacott R. Generic qualitative research: a design for qualitative research in emergency care? Emerg Med J. 2007;24(12):816–819. doi:10.1136/emj.2007.050641.
   PubMed Web of Science ®Google Scholar
• Ginsburg S, van der Vleuten CPM, Eva KW. The hidden value of narrative comments for assessment. Acad Med. 2017;92(11):1. doi:10.1097/ACM.0000000000001669.
   PubMed Web of Science ®Google Scholar
• Watling CJ, Ginsburg S. Assessment, feedback and the alchemy of learning. Med Educ. 2019;53(1):76–85. doi:10.1111/medu.13645.
   PubMed Web of Science ®Google Scholar
• Ariaeinejad A, Samavi R, Chan T, et al. A performance predictive model for emergency medicine residents. In: 27th Annual International Conference on Computer Science and Software Engineering. Toronto, ON: ACM Digital Library; 2017.
   Google Scholar
• Bismil R, Dudek NL, Wood TJ. In-training evaluations: developing an automated screening tool to measure report quality. Med Educ. 2014;48(7):724–732. doi:10.1111/medu.12490.
   PubMed Web of Science ®Google Scholar
• Hart SG, Staveland LE. Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. Adv Psychol. 1988;52:139–183. doi:10.1016/S0166-4115(08)62386-9.
   Google Scholar