Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles

References

• Al Kadri H, Al-Moamary M, Van Der Vleuten C. 2009. Students' and teachers' perceptions of clinical assessment program: a qualitative study in a PBL curriculum. BMC Res Notes. 2(1):263.
   PubMedGoogle Scholar
• Anderson R, Demeulle L. 1998. Portfolio use in twenty-four teacher education programs. Teach Educ Quart. 25:23–31.
   Google Scholar
• Bala L, Van Der Vleuten C, Freeman A, Torre D, Heeneman S, Sam A. 2020. COVID-19 and programmatic assessment. Clin Teach. 17(4):420–422.
   PubMed Web of Science ®Google Scholar
• Bate F, Fyfe S, Griffiths D, Russell K, Skinner C, Tor E. 2020. Does an incremental approach to implementing programmatic assessment work? Reflections on the change process. MedEdPublish. 9(1).
   PubMedGoogle Scholar
• Biggs J. 1996. Enhancing teaching through constructive alignment. High Educ. 32(3):347–364.
   Web of Science ®Google Scholar
• Bok H, Teunissen P, Favier RP, Rietbroek N, Theyse L, Brommer H, Haarhuis J, Van Beukelen P, Van Der Vleuten C, Jaarsma D. 2013. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ. 13:123.
   PubMed Web of Science ®Google Scholar
• Boud D, Molloy E. 2012. Rethinking models of feedback for learning: the challenge of design. Assess Eval High Educ. 2012:1–15.
   Google Scholar
• Carless D, Salter D, Yang M, Lam J. 2011. Developing sustainable feedback practices. Stud High Educ. 36(4):395–407.
   Web of Science ®Google Scholar
• Castanelli D, Weller J, Molloy E, Bearman M. 2020. Shadow systems in assessment: how supervisors make progress decisions in practice. Adv Health Sci Educ. 25(1):131–147.
   PubMed Web of Science ®Google Scholar
• Chan T, Sherbino J. 2015. The McMaster Modular Assessment Program (McMAP): a theoretically grounded work-based assessment system for an emergency medicine residency program. Acad Med. 90(7):900–905.
   PubMed Web of Science ®Google Scholar
• Cilliers F, Schuwirth L, Herman N, Adendorff H, Van Der Vleuten C. 2012a. A model of the pre-assessment learning effects of summative assessment in medical education. Adv Health Sci Educ. 17(1):39–53.
   PubMed Web of Science ®Google Scholar
• Cilliers F, Schuwirth L, Van Der Vleuten C. 2012b. Modelling the pre-assessment learning effects of assessment: evidence in the validity chain. Med Educ. 46(11):1087–1098.
   PubMed Web of Science ®Google Scholar
• Dannefer EF, Henson L. 2007. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Acad Med. 82:493–502.
   PubMed Web of Science ®Google Scholar
• Dannefer EF, Prayson RA. 2013. Supporting students in self-regulation: use of formative feedback and portfolios in a problem-based learning setting. Med Teach. 35:1–6.
   PubMed Web of Science ®Google Scholar
• Dart J, Twohig C, Anderson A, Bryce A, Collins J, Gibson S, Kleve S, Porter J, Volders E, Palermo C. 2021. The value of programmatic assessment in supporting educators and students to succeed: a qualitative evaluation. J Acad Nutr Diet. DOI:https://doi.org/10.1016/j.jand.2021.01.013.
   PubMed Web of Science ®Google Scholar
• Dawson L, Mason B, Balmer C, Jimmieson P. 2015. Developing professional competence using integrated technology-supported approaches: a case study in dentistry. In: Fry H, Ketteridge S, Marshall S, editors. A handbook for teaching and learning in higher education enhancing academic practice. London; New York: Routledge.
   Google Scholar
• De Jong L, Bok H, Kremer W, Van Der Vleuten C. 2019. Programmatic assessment: can we provide evidence for saturation of information? Med Teach. 41(6):678–682.
   PubMed Web of Science ®Google Scholar
• De La Croix A, Veen M. 2018. The reflective zombie: problematizing the conceptual framework of reflection in medical education. Perspect Med Educ. 7(6):394–400.
   PubMed Web of Science ®Google Scholar
• Driessen E, Van Tartwijk J, Govaerts M, Teunissen P, Van Der Vleuten C. 2012. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 34(3):226–231.
   PubMed Web of Science ®Google Scholar
• Duitsman ME, Fluit CRMG, van Alfen-van der Velden JAEM, de Visser M, Ten Kate-Booij M, Dolmans DHJM, Jaarsma DADC, de Graaf J. 2019. Design and evaluation of a clinical competency committee. Perspect Med Educ. 8(1):1–8.
   PubMed Web of Science ®Google Scholar
• Ellaway R, Chou C, Kalet A. 2018. Situating remediation: accommodating success and failure in medical education systems. Acad Med. 93(3):391–398.
   PubMed Web of Science ®Google Scholar
• Frambach J, Driessen E, Chan L, Van Der Vleuten C. 2012. Rethinking the globalisation of problem-based learning: how culture challenges self-directed learning. Med Educ. 46(8):738–747.
   PubMed Web of Science ®Google Scholar
• Frank JR, Snell LS, Cate OT, Holmboe ES, Carraccio C, Swing SR, Harris P, Glasgow NJ, Campbell C, Dath D, et al. 2010. Competency-based medical education: theory to practice. Med Teach. 32(8):638–645.
   PubMed Web of Science ®Google Scholar
• Freeman A, Ricketts C. 2010. Choosing and designing knowledge assessments: experience at a new medical school. Med Teach. 32(7):578–581.
   PubMed Web of Science ®Google Scholar
• Friedman Ben David M, Davis MH, Harden RM, Howie PW, Ker J, Pippard MJ. 2001. AMEE Medical Education Guide No. 24: portfolios as a method of student assessment. Med Teach. 23(6):535–551.
   PubMed Web of Science ®Google Scholar
• Govaerts M, Van Der Vleuten C. 2013. Validity in work-based assessment: expanding our horizons. Med Educ. 47(12):1164–1174.
   PubMed Web of Science ®Google Scholar
• Harrison CJ, Konings KD, Molyneux A, Schuwirth L, Wass V, Van Der Vleuten C. 2013. Web-based feedback after summative assessment: how do students engage? Med Educ. 47(7):734–744.
   PubMed Web of Science ®Google Scholar
• Harrison C, Könings K, Schuwirth L, Wass V, Van Der Vleuten C. 2015. Barriers to the uptake and use of feedback in the context of summative assessment. Adv Health Sci Educ. 20(1):229–245.
   PubMed Web of Science ®Google Scholar
• Hauer K, Cate O, Boscardin C, Iobst W, Holmboe E, Chesluk B, Baron R, O'Sullivan P. 2016. Ensuring resident competence: a narrative review of the literature on group decision making to inform the work of clinical competency committees. J Grad Med Educ. 8(2):156–164.
   PubMedGoogle Scholar
• Heeneman S, De Grave W. 2017. Tensions in mentoring medical students toward self-directed and reflective learning in a longitudinal portfolio-based mentoring system – an activity theory analysis. Med Teach. 39(4):368–376.
   PubMed Web of Science ®Google Scholar
• Heeneman S, Oudkerk Pool A, Schuwirth L, Van Der Vleuten C, Driessen E. 2015. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 49(5):487–498.
   PubMed Web of Science ®Google Scholar
• Jamieson J, Hay M, Gibson S, Palermo C. 2021. Implementing programmatic assessment transforms supervisor attitudes: an explanatory sequential mixed methods study. Med Teach. 43:1–9.
   PubMed Web of Science ®Google Scholar
• Jamieson J, Jenkins G, Beatty S, Palermo C. 2017. Designing programmes of assessment: a participatory approach. Med Teach. 39(11):1182–1188.
   PubMed Web of Science ®Google Scholar
• Khan R. 2018. Measuring learning of medical students through ‘programmatic assessment’. Pak J Med Sci. 34:3–4.
   PubMed Web of Science ®Google Scholar
• Kinnear B, Warm E, Hauer K. 2018. Twelve tips to maximize the value of a clinical competency committee in postgraduate medical education. Med Teach. 40(11):1110–1115.
   PubMed Web of Science ®Google Scholar
• Knowles M. 1975. A guide for learners and teachers. England Cliffs: Prentice Hall/Cambridge.
   Google Scholar
• Kulasegaram K, Mylopoulos M, Tonin P, Bernstein S, Bryden P, Law M, Lazor J, Pittini R, Sockalingam S, Tait G, et al. 2018. The alignment imperative in curriculum renewal. Med Teach. 40(5):443–448.
   PubMed Web of Science ®Google Scholar
• Laughlin T, Brennan A, Brailovsky C. 2012. Effect of field notes on confidence and perceived competence: survey of faculty and residents. Can Fam Phys. 58:e352–e356.
   PubMed Web of Science ®Google Scholar
• Lingard L. 2007. The rhetorical ‘turn’ in medical education: what have we learned and where are we going? Adv Health Sci Educ Theory Pract. 12(2):121–133.
   PubMed Web of Science ®Google Scholar
• Lingard L. 2009. What we see and don’t see when we look at ‘competence’: notes on a god term. Adv Health Sci Educ. 14(5):625–628.
   PubMed Web of Science ®Google Scholar
• McDowell L. 2012. Programme focused assessment – a short guide. Northumbria University. https://www.bradford.ac.uk/pass/resources/short-guide.pdf.
   Google Scholar
• Mookherjee S, Chang A, Boscardin C, Hauer K. 2013. How to develop a competency-based examination blueprint for longitudinal standardized patient clinical skills assessments. Med Teach. 35(11):883–890.
   PubMed Web of Science ®Google Scholar
• Murdoch-Eaton D, Sandars J. 2014. Reflection: moving from a mandatory ritual to meaningful professional development. Arch Dis Child. 99(3):279–283.
   PubMed Web of Science ®Google Scholar
• Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, Galbraith R, Hays R, Kent A, Perrott V, et al. 2011. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 33(3):206–214.
   PubMed Web of Science ®Google Scholar
• Norcini J, Anderson M, Bollela V, Burch V, Costa M, Duvivier R, Hays R, Palacios Mackay M, Roberts T, Swanson D. 2018. 2018 consensus framework for good assessment. Med Teach. 40(11):1102–1109.
   PubMed Web of Science ®Google Scholar
• Norman G, Swanson D, Case SM. 1996. Conceptual and methodological issues in studies comparing assessment formats. Teach Learn Med. 8(4):208–216.
   Web of Science ®Google Scholar
• Pool AO, Jaarsma A, Driessen E, Govaerts M. 2020. Student perspectives on competency-based portfolios: does a portfolio reflect their competence development? Perspect Med Educ. 9(3):166–172.
   PubMed Web of Science ®Google Scholar
• Panadero E. 2017. A review of self-regulated learning: six models and four directions for research. Front Psychol. 8:422.
   PubMed Web of Science ®Google Scholar
• Pearce J, Reid K, Chiavaroli N, Hyam D. 2021. Incorporating aspects of programmatic assessment into examinations: aggregating rich information to inform decision-making. Med Teach. 43:1–8.
   PubMed Web of Science ®Google Scholar
• Pilling-Cormick J. 1997. Transformative and self-directed learning in practice. New Dir Adult Cont Educ. 1997(74):69–77.
   Google Scholar
• Prentice S, Benson J, Kirkpatrick E, Schuwirth L. 2020. Workplace‐based assessments in postgraduate medical education – a hermeneutic review. Med Educ. 54(11):981–992.
   PubMed Web of Science ®Google Scholar
• Price M, Rust C, O'Donovan B, Handley K, Bryant R. 2012. Assessment literacy: the foundation for improving student learning. Oxford: Oxford Brookes University.
   Google Scholar
• Ryan A, McColl G, O'Brien R, Chiavaroli N, Judd T, Finch S, Swanson D. 2017. Tensions in post-examination feedback: information for learning versus potential for harm. Med Educ. 51(9):963–973.
   PubMed Web of Science ®Google Scholar
• Sargeant J, Mann K, Van Der Vleuten C, Metsemakers J. 2009. Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ. 14(3):399–410.
   PubMed Web of Science ®Google Scholar
• Schut S, Driessen E, Van Tartwijk J, Van Der Vleuten C, Heeneman S. 2018. Stakes in the eye of the beholder: an international study of learners' perceptions within programmatic assessment. Med Educ. 52(6):654–663.
   PubMed Web of Science ®Google Scholar
• Schut S, Heeneman S, Bierer B, Driessen E, Van Tartwijk J, Van Der Vleuten C. 2020. Between trust and control: teachers' assessment conceptualisations within programmatic assessment. Med Educ. 54(6):528–537.
   PubMed Web of Science ®Google Scholar
• Schut S, Maggio L, Heeneman S, Van Tartwijk J, Van Der Vleuten C, Driessen E. 2021. Where the rubber meets the road – an integrative review of programmatic assessment in health care professions education. Perspect Med Educ. 10(1):6–13.
   PubMed Web of Science ®Google Scholar
• Schuwirth L, Valentine N, Dilena P. 2017. An application of programmatic assessment for learning (PAL) system for general practice training. GMS J Med Educ. 34:Doc56.
   PubMed Web of Science ®Google Scholar
• Schuwirth L, Van Der Vleuten C. 2011. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 33(6):478–485.
   PubMed Web of Science ®Google Scholar
• Schuwirth L, Van Der Vleuten C. 2019. Current assessment in medical education: programmatic assessment. J Appl Test Technol. 20:2–10.
   Google Scholar
• Sutton P. 2012. Conceptualizing feedback literacy: knowing, being, and acting. Inn Educ Teach Int. 49(1):31–40.
   Web of Science ®Google Scholar
• Tekian A, Watling C, Roberts T, Steinert Y, Norcini J. 2017. Qualitative and quantitative feedback in the context of competency-based education. Med Teach. 39(12):1245–1249.
   PubMed Web of Science ®Google Scholar
• Ten Cate O. 2005. Entrustability of professional activities and competency-based training. Med Educ. 39(12):1176–1177.
   PubMed Web of Science ®Google Scholar
• Ten Cate O, Scheele F. 2007. Viewpoint: competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med. 82:542–547.
   PubMed Web of Science ®Google Scholar
• Tillema H. 2001. Portfolios as developmental assessment tools. Int J Train Dev. 5(2):126–135.
   Google Scholar
• Tochel C, Haig A, Hesketh A, Cadzow A, Beggs K, Colthart I, Peacock H. 2009. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12. Med Teach. 31(4):299–318.
   PubMed Web of Science ®Google Scholar
• Torre D, Rice N, Ryan A, Bok H, Dawson L, Bierer B, Wilkinson T, Tait G, Laughlin T, Veerapen K, et al. 2021. Ottawa 2020 consensus statements for programmatic assessment 2: implementation and practice. Med Teach. in press.
   Google Scholar
• Tsuei S, Lee D, Ho C, Regehr G, Nimmon L. 2019. Exploring the construct of psychological safety in medical education. Acad Med. 94:S28–S35.
   PubMed Web of Science ®Google Scholar
• Tweed M, Wilkinson T. 2019. Student progress decision-making in programmatic assessment: can we extrapolate from clinical decision-making and jury decision-making? BMC Med Educ. 19(1):176.
   PubMed Web of Science ®Google Scholar
• Van Der Vleuten C. 1996. The assessment of professional competence: developments, research and practical implications. Adv Health Sci Educ Theory Pract. 1(1):41–67.
   PubMed Web of Science ®Google Scholar
• Van Der Vleuten C, Heeneman S, Schut S. 2019. Programmatic assessment: an avenue to a different assessment culture. In: Yudkowsky R, Soo Park Y, Downing S, editors. Assessment in health professions education. New York: Routledge.
   Google Scholar
• Van Der Vleuten C, Heeneman S, Schuwirth L. 2017. Programmatic assessment. In: Dent J, Harden R, Hunt D, editors. A practical guide for medical teachers. Edinburgh: Elsevier.
   Google Scholar
• Van Der Vleuten C, Schuwirth L. 2005. Assessing professional competence: from methods to programmes. Med Educ. 39(3):309–317.
   PubMed Web of Science ®Google Scholar
• Van Der Vleuten C, Schuwirth L, Driessen E, Dijkstra J, Tigelaar D, Baartman L, Van Tartwijk J. 2012. A model for programmatic assessment fit for purpose. Med Teach. 34(3):205–214.
   PubMed Web of Science ®Google Scholar
• Van Der Vleuten C, Schuwirth L, Driessen E, Govaerts M, Heeneman S. 2015. Twelve tips for programmatic assessment. Med Teach. 37(7):641–646.
   PubMed Web of Science ®Google Scholar
• Van Der Vleuten C, Schuwirth L, Scheele F, Driessen E, Hodges B. 2010. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. 24(6):703–719.
   PubMed Web of Science ®Google Scholar
• Van Tartwijk J, Driessen E. 2009. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach. 31(9):790–801.
   PubMed Web of Science ®Google Scholar
• Villarroel V, Bloxham S, Bruna D, Bruna C, Herrera-Seda C. 2018. Authentic assessment: creating a blueprint for course design. Assess Eval High Educ. 43(5):840–854.
   Web of Science ®Google Scholar
• Watling C, Ginsburg S, Ladonna K, Lingard L, Field E. 2021. Going against the grain: an exploration of agency in medical learning. Med Educ.
   Web of Science ®Google Scholar
• Wilkinson T, Tweed M. 2018. Deconstructing programmatic assessment. Adv Med Educ Pract. 9:191–197.
   PubMed Web of Science ®Google Scholar
• Wilkinson T, Tweed M, Egan T, Ali A, Mckenzie J, Moore M, Rudland J. 2011. Joining the dots: conditional pass and programmatic assessment enhances recognition of problems with professionalism and factors hampering student progress. BMC Med Educ. 11(1):1–9.
   PubMed Web of Science ®Google Scholar
• Wong A. 2011. Culture in medical education: comparing a Thai and a Canadian residency programme. Med Educ. 45(12):1209–1219.
   PubMed Web of Science ®Google Scholar
• Zimmerman BJ. 1989. A social cognitive view of self-regulated academic learning. J Educ Psychol. 81(3):329–339.
   Web of Science ®Google Scholar
• Zoanetti N, Pearce J. 2021. The potential use of Bayesian networks to support committee decisions in programmatic assessment. Med Educ. 55(7):808–817.
   PubMed Web of Science ®Google Scholar