Key considerations in planning and designing programmatic assessment in competency-based medical education

References

• Ambady N, LaPlante D, Nguyen T, Rosenthal R, Chaumeton N, Levinson W. 2002. Surgeons’ tone of voice: a clue to malpractice history. Surgery. 132(1):5–9.
   PubMed Web of Science ®Google Scholar
• Baartman LK, Bastiaens TJ, Kirschner PA, Van der Vleuten CP. 2007. Evaluating assessment quality in competence-based education: a qualitative comparison of two frameworks. Educ Res Rev. 2(2):114–129.
   Google Scholar
• Bate F, Fyfe S, Griffiths D, Russell K, Skinner C, Tor E. 2020. Does an incremental approach to implementing programmatic assessment work? Reflections on the change process. MedEdPublish. 9(1), 55. doi:10.15694/mep.2020.000055.1.
   PubMedGoogle Scholar
• Bates J, Ellaway RH. 2016. Mapping the dark matter of context: a conceptual scoping review. Med Educ. 50(8):807–816.
   PubMed Web of Science ®Google Scholar
• Berkhout JJ, Helmich E, Teunissen PW, van der Vleuten CP, Jaarsma AD. 2018. Context matters when striving to promote active and lifelong learning in medical education. Med Educ. 52(1):34–44.
   PubMed Web of Science ®Google Scholar
• Bhanji F, Miller G, Cheung WJ, Puligandla PS, Winthrop A, Baird R, Davies D, Lopushinsky SR, Webber EM. 2020. The future is here! Pediatric surgery and the move to the Royal College of Physicians and Surgeons of Canada’s competence by design. J Pediatr Surg. 55(5):796–799.
   PubMed Web of Science ®Google Scholar
• Black P, Wiliam D. 1998. Inside the black box: raising standards through classroom assessment. Phi Delta Kappan. 80(2):139–148.
   Web of Science ®Google Scholar
• Bok HGJ, Teunissen PW, Favier RP, Rietbroek NJ, Theyse LFH, Brommer H, Haarhuis JCM, van Beukelen P, van der Vleuten CPM, Jaarsma DADC. 2013. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ. 13(1):123.
   PubMed Web of Science ®Google Scholar
• Deis JN, Smith KM, Warren MD, Throop PG, Hickson GB, Joers BJ, Deshpande JK. 2008. Transforming the morbidity and mortality conference into an instrument for systemwide improvement. In: Advances in patient safety: new directions and alternative approaches (vol. 2: culture and redesign). Rockville (MD): Agency for Healthcare Research and Quality.
   Google Scholar
• Dijkstra J, van der Vleuten CPM, Schuwirth LWT. 2010. A new framework for designing programmes of assessment. Adv Health Sci Educ Theory Pract. 15(3):379–393.
   PubMed Web of Science ®Google Scholar
• Donoff MG. 2009. Field notes: assisting achievement and documenting competence. Can Fam Physician. 55(12):1260–1262.
   PubMed Web of Science ®Google Scholar
• Donoff MG. 1990. The science of in-training evaluation: facilitating learning with qualitative research. Can Family Physician. 36:2002–2006.
   PubMed Web of Science ®Google Scholar
• Edgar L, McLean S, Hogan SO, Hanstra S, Holmboe ES. 2020. The milestones guidebook; [accessed 2020 Sept. 25]. Chicago (IL): Accreditation Council for Graduate Medical Education. http://www.acgme.org/Portals/0/MilestonesGuidebook.pdf.
   Google Scholar
• Eva KW, Bordage G, Campbell C, Galbraith R, Ginsburg S, Holmboe E, Regehr G. 2016. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ Theory Pract. 21(4):897–913.
   PubMed Web of Science ®Google Scholar
• Friedman KA, Raimo J, Spielmann K, Chaudhry S. 2016. Resident dashboards: helping your clinical competency committee visualize trainees' key performance indicators. Med Educ Online. 21(1):29838.
   PubMed Web of Science ®Google Scholar
• Hall AK, Rich J, Dagnone JD, Weersink K, Caudle J, Sherbino J, Frank JR, Bandiera G, Van Melle E. 2020. It’s a marathon, not a sprint: rapid evaluation of competency-based medical education program implementation . Acad Med. 95(5):786–793.
   PubMed Web of Science ®Google Scholar
• Harris P, Bhanji F, Topps M, Ross S, Lieberman S, Frank JR, Snell L, Sherbino J. 2017. Evolving concepts of assessment in a competency-based world. Med Teach. 39(6):603–608.
   PubMed Web of Science ®Google Scholar
• Hauer KE, O’Sullivan PS, Fitzhenry K, Boscardin C. 2018. Translating theory into practice: implementing a program of assessment. Acad Med. 93(3):444–450.
   PubMed Web of Science ®Google Scholar
• Heeneman S, Oudkerk Pool A, Schuwirth LW, van der Vleuten CP, Driessen EW. 2015. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 49(5):487–498.
   PubMed Web of Science ®Google Scholar
• Iobst WF, Holmboe ES. 2020. Programmatic assessment: the secret sauce of effective CBME implementation. J Grad Med Educ. 12(4):518–521.
   PubMedGoogle Scholar
• Kinnear B, Warm EJ, Hauer KE. 2018. Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education. Med Teach. 40(11):1110–1115.
   PubMed Web of Science ®Google Scholar
• Lacasse M, Théorêt J, Tessier S, Arsenault L. 2014. Expectations of clinical teachers and faculty regarding development of the CanMEDS-Family Medicine competencies: laval developmental benchmarks scale for family medicine residency training. Teach Learn Med. 26(3):244–251.
   PubMed Web of Science ®Google Scholar
• Lee AV, Moriarty JP, Borgstrom C, Horwitz LI. 2010. What can we learn from patient dissatisfaction? An analysis of dissatisfying events at an academic medical center. J Hosp Med. 5(9):514–520.
   PubMed Web of Science ®Google Scholar
• Levinson W, Roter DL, Mullooly JP, Dull VT, Frankel RM. 1997. Physician-patient communication. The relationship with malpractice claims among primary care physicians and surgeons. JAMA. 277(7):553–559.
   PubMed Web of Science ®Google Scholar
• Lockyer J, Carraccio C, Chan MK, Hart D, Smee S, Touchie C, Holmboe ES, Frank JR. 2017. Core principles of assessment in competency-based medical education. Med Teach. 39(6):609–616.
   PubMed Web of Science ®Google Scholar
• Lurie SJ. 2012. History and practice of competency-based assessment. Med Educ. 46(1):49–57.
   PubMed Web of Science ®Google Scholar
• McEwen LA, Griffiths J, Schultz K. 2015. Developing and successfully implementing a competency-based portfolio assessment system in a postgraduate family medicine residency program. Acad Med. 90(11):1515–1526.
   PubMed Web of Science ®Google Scholar
• Montini T, Noble AA, Stelfox HT. 2008. Content analysis of patient complaints. Int J Qual Health Care. 20(6):412–420.
   PubMed Web of Science ®Google Scholar
• Norcini J, Anderson MB, Bollela V, Burch V, Costa MJ, Duvivier R, Hays R, Palacios Mackay MF, Roberts T, Swanson D. 2018. 2018 Consensus framework for good assessment. Med Teach. 40(11):1102–1109.
   PubMed Web of Science ®Google Scholar
• Poulton A, Rose H. 2015. The importance of health advocacy in Canadian postgraduate medical education: current attitudes and issues. Can Med Educ J. 6(2):e54–e60.
   PubMedGoogle Scholar
• Redwood-Campbell L, Pakes B, Rouleau K, MacDonald CJ, Arya N, Purkey E, Schultz K, Dhatt R, Wilson B, Hadi A, et al. 2011. Developing a curriculum framework for global health in family medicine: emerging principles, competencies, and educational approaches. BMC Med Educ. 11(1):46.
   PubMed Web of Science ®Google Scholar
• Reiser RA, Dempsey JV, editors. 2012. Trends and issues in instructional design and technology. Boston: Pearson.
   Google Scholar
• Rich JV, Fostaty Young S, Donnelly C, Hall AK, Dagnone JD, Weersink K, Caudle J, Van Melle E, Klinger DA. 2020. Competency-based education calls for programmatic assessment: but what does this look like in practice? J Eval Clin Pract. 26(4):1087–1095.
   PubMed Web of Science ®Google Scholar
• Ross S, Poth C, Donoff M, Humphries P, Steiner I, Schipper S, Janke F, Nichols D. 2011. The competency-based achievement system: using formative feedback to teach and assess competencies with family medicine residents. Can Fam Physician. 57:e323–e330.
   PubMed Web of Science ®Google Scholar
• Ross S, Poth C, Donoff M, Papile C, Humphries P, Stasiuk S, Georgis R. 2012. Involving users in the refinement of the competency-based achievement system, an innovative approach to competency-based assessment. Med Teach. 34:e413–e417.
   Web of Science ®Google Scholar
• Schut S, Driessen E, van Tartwijk J, van der Vleuten C, Heeneman S. 2018. Stakes in the eye of the beholder: an international study of learners’ perceptions within programmatic assessment. Med Educ. 52(6):654–663.
   PubMed Web of Science ®Google Scholar
• Schuwirth L, Ash J. 2013. Assessing tomorrow’s learners: in competency-based education only a radically different holistic method of assessment will work. Six things we could forget . Med Teach. 35(7):555–559.
   PubMed Web of Science ®Google Scholar
• Schuwirth LW, Van der Vleuten CP. 2011. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 33(6):478–485.
   PubMed Web of Science ®Google Scholar
• Schuwirth LW, van der Vleuten CP. 2019. How ‘testing’ has become ‘programmatic assessment for learning. Health Prof Educ. 5(3):177–184.
   Google Scholar
• Swan Sein A, Rashid H, Meka J, Amiel J, Pluta W. 2020. Twelve tips for embedding assessment for and as learning practices in a programmatic assessment system. Med Teach. 43(3):300–306.
   PubMed Web of Science ®Google Scholar
• Ten Cate O. 2005. Entrustability of professional activities and competency-based training. Med Educ. 39(12):1176–1177.
   PubMed Web of Science ®Google Scholar
• Torrance H. 2007. Assessment as learning? How the use of explicit learning objectives, assessment criteria and feedback in post‐secondary education and training can come to dominate learning. Assess Educ. 14(3):281–294.
   Google Scholar
• Torre DM, Schuwirth LW, Van der Vleuten CP. 2020. Theoretical considerations on programmatic assessment. Med Teach. 42(2):213–220.
   PubMed Web of Science ®Google Scholar
• Trotter E. 2006. Student perceptions of continuous summative assessment. Assess Eval Higher Educ. 31(5):505–521.
   Google Scholar
• Tweed M, Wilkinson T. 2019. Student progress decision-making in programmatic assessment: can we extrapolate from clinical decision-making and jury decision-making? BMC Med Educ. 19(1):176.
   PubMed Web of Science ®Google Scholar
• Van Der Vleuten CP. 1996. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1(1):41–67.
   PubMed Web of Science ®Google Scholar
• van der Vleuten CPM, Schuwirth LWT. 2005. Assessing professional competence: from methods to programmes. Med Educ. 39(3):309–317.
   PubMed Web of Science ®Google Scholar
• van der Vleuten CP, Schuwirth LW, Driessen EW, Dijkstra J, Tigelaar D, Baartman LK, van Tartwijk J. 2012. A model for programmatic assessment fit for purpose. Med Teach. 34(3):205–214.
   PubMed Web of Science ®Google Scholar
• Van Der Vleuten CPM, Schuwirth LWT, Driessen EW, Govaerts MJB, Heeneman S. 2015. Twelve tips for programmatic assessment. Med Teach. 37(7):641–646.
   PubMed Web of Science ®Google Scholar
• Watling CJ, Ginsburg S. 2019. Assessment, feedback and the alchemy of learning. Med Educ. 53(1):76–85.
   PubMed Web of Science ®Google Scholar
• Zoanetti N, Pearce J. 2020. The potential use of Bayesian Networks to support committee decisions in programmatic assessment. Med Educ. [accessed 2020 Nov 5]:[10 p.]. https://doi.org/10.1111/medu.14407
   PubMed Web of Science ®Google Scholar