Ottawa 2020 consensus statement for programmatic assessment – 1. Agreement on the principles

Figures & data

Table 1. Principles of programmatic assessment as based on literature, used as the starting point of the consensus process (Van der Vleuten et al. 2010, 2012, 2017, 2019; van der Vleuten and Schuwirth 2005).

1	Every (part of an) assessment is but a data-point
2	Every data-point is optimised for learning by giving meaningful feedback to the learner
3	Pass/fail decisions are not given on a single data-point
4	There is a mix of methods of assessment
5	The choice of method depends on the educational justification for using that method
6	The distinction between summative and formative is replaced by a continuum of stakes
7	Stakes and decision-making learner progress are proportionally related to the stakes
8	Assessment information is triangulated across data-points towards a competency framework
9	High-stakes decisions (promotion, graduation) are made in competence committees
10	Intermediate decisions are made with the purpose of informing the learner on their progress
11	Learners have recurrent learning meetings with (faculty) mentors using a self-analysis of all assessment data
12	The assessment is optimally bespoke or tailored to the individual learner

Table 2. Final Ottawa 2020 consensus principles of programmatic assessment after input of the expert group and Ottawa attendees, changes as compared to Table 1 are indicated in bold.

1	Every (part of an) assessment is but a data-point
2	Every data-point is optimised for learning by giving meaningful feedback to the learner
3	Pass/fail decisions are not given on a single data-point
4	There is a mix of methods of assessment
5	The method chosen should depend on the educational justification for using that method
6	The distinction between summative and formative is replaced by a continuum of stakes
7	Decision-making on learner progress is proportionally related to the stake
8	Assessment information is triangulated across data-points towards an appropriate framework
9	High-stakes decisions (promotion, graduation) are made by in a credible and transparent manner, using a holistic approach
10	Intermediate review is made to discuss and decide with the learner on their progression
11	Learners have recurrent learning meetings with (faculty) mentors/coaches using a self-analysis of all assessment data
12	Programmatic assessment seeks to gradually increase the learner’s agency and accountability for their own learning through the learning being tailored to support individual learning priorities

Table 3. Characteristics of the programmes of members of the expert group, who completed the survey on agreement, importance, adherence, and implementation of the principles of programmatic assessment.

	Discipline	Undergraduate (UG) post-graduate/speciality training (PG)	Number of learners	Country	Reference, in which the programme is described, if available
1	Medicine	UG	280/year	United Kingdom	Freeman and Ricketts (2010)
2	Medicine	UG	32/year	USA	Dannefer and Henson (2007) and Dannefer and Prayson (2013)
3	Dentistry	UG	377/year	United Kingdom	Dawson et al. (2015)
4	Medicine	PG	38	Canada	Chan and Sherbino (2015)
5	Medicine	UG	300/year	New Zealand	Wilkinson et al. (2011)
6	Medicine	UG	50/year	Netherlands	Heeneman et al. (2015) and Schut et al. (2018)
7	Veterinary	UG	225/year	Netherlands	Bok et al. (2013) and de Jong et al. (2019)
8	Medicine	UG	260/year	Canada	Kulasegaram et al. (2018)
9	Nutrition	PG	20	Australia	Jamieson et al. (2017)
10	Medicine	UG	350/year	Netherlands	Partly in Driessen et al. (2012)
11	Medicine	UG	360/year	Australia	Ryan et al. (2017)
12	Medicine	UG	200/year	Australia	–
13	Medicine	PG	462	Australia	Schuwirth et al. (2017)
14	Medicine	UG	288/year	Canada	–
15	Medicine	PG	136	Canada	Laughlin et al. (2012)

Table 4. Overview of the data by categorising the responses for questions on agreement, importance, adherence, and implementation of the principles of programmatic assessment.

Principle	Agreement	Importance	Easy to adhere	Implemented
	Yes/partially/no	Yes/mixed/no	Yes/partially/no	Yes/partially/no
1/2/3	14/1/0	10/4/1	8/0/7	13/2/0
4/5	15/0/0	9/6/0	12/0/2	12/2/1
6/7	15/0/0	6/5/4	7/0/8	13/0/2
8	14/1/0	8/4/3	9/0/5	12/2/1
9	15/0/0	7/6/2	11/3/0	13/2/0
9^a	13/0/0	12/0/0	12/1/0	13/0/0
10	13/1/1	8/5/2	9/0/4	13/1/1
11	12/3/0	9/3/3	10/0/5	10/4/1
12	7/5/1	2/4/8	5/0/9	8/3/3
12^a	11/2/0	10/2/0	5/6/1	8/3/0

^a The responses to rephrased principle 9 and 12 were recollected after the first analysis and discussion with the experts, prior to the conference.

White and bold: majority of programs confirmed. Light grey and italics: programs were mixed in their responses. Dark grey and underlined: majority of programs did not confirm.

Van Der Vleuten C, Schuwirth L, Scheele F, Driessen E, Hodges B. 2010. The assessment of professional competence: building blocks for theory development. Best Pract Res Clin Obstet Gynaecol. 24(6):703–719.

 PubMed Web of Science ®Google Scholar

Van Der Vleuten C, Schuwirth L, Driessen E, Dijkstra J, Tigelaar D, Baartman L, Van Tartwijk J. 2012. A model for programmatic assessment fit for purpose. Med Teach. 34(3):205–214.

 PubMed Web of Science ®Google Scholar

Van Der Vleuten C, Heeneman S, Schuwirth L. 2017. Programmatic assessment. In: Dent J, Harden R, Hunt D, editors. A practical guide for medical teachers. Edinburgh: Elsevier.

 Google Scholar

Van Der Vleuten C, Heeneman S, Schut S. 2019. Programmatic assessment: an avenue to a different assessment culture. In: Yudkowsky R, Soo Park Y, Downing S, editors. Assessment in health professions education. New York: Routledge.

 Google Scholar

Van Der Vleuten C, Schuwirth L. 2005. Assessing professional competence: from methods to programmes. Med Educ. 39(3):309–317.

 PubMed Web of Science ®Google Scholar

Freeman A, Ricketts C. 2010. Choosing and designing knowledge assessments: experience at a new medical school. Med Teach. 32(7):578–581.

 PubMed Web of Science ®Google Scholar

Dannefer EF, Henson L. 2007. The portfolio approach to competency-based assessment at the Cleveland Clinic Lerner College of Medicine. Acad Med. 82:493–502.

 PubMed Web of Science ®Google Scholar

Dannefer EF, Prayson RA. 2013. Supporting students in self-regulation: use of formative feedback and portfolios in a problem-based learning setting. Med Teach. 35:1–6.

 PubMed Web of Science ®Google Scholar

Dawson L, Mason B, Balmer C, Jimmieson P. 2015. Developing professional competence using integrated technology-supported approaches: a case study in dentistry. In: Fry H, Ketteridge S, Marshall S, editors. A handbook for teaching and learning in higher education enhancing academic practice. London; New York: Routledge.

 Google Scholar

Chan T, Sherbino J. 2015. The McMaster Modular Assessment Program (McMAP): a theoretically grounded work-based assessment system for an emergency medicine residency program. Acad Med. 90(7):900–905.

 PubMed Web of Science ®Google Scholar

Wilkinson T, Tweed M, Egan T, Ali A, Mckenzie J, Moore M, Rudland J. 2011. Joining the dots: conditional pass and programmatic assessment enhances recognition of problems with professionalism and factors hampering student progress. BMC Med Educ. 11(1):1–9.

 PubMed Web of Science ®Google Scholar

Heeneman S, Oudkerk Pool A, Schuwirth L, Van Der Vleuten C, Driessen E. 2015. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 49(5):487–498.

 PubMed Web of Science ®Google Scholar

Schut S, Driessen E, Van Tartwijk J, Van Der Vleuten C, Heeneman S. 2018. Stakes in the eye of the beholder: an international study of learners' perceptions within programmatic assessment. Med Educ. 52(6):654–663.

 PubMed Web of Science ®Google Scholar

Bok H, Teunissen P, Favier RP, Rietbroek N, Theyse L, Brommer H, Haarhuis J, Van Beukelen P, Van Der Vleuten C, Jaarsma D. 2013. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ. 13:123.

 PubMed Web of Science ®Google Scholar

De Jong L, Bok H, Kremer W, Van Der Vleuten C. 2019. Programmatic assessment: can we provide evidence for saturation of information? Med Teach. 41(6):678–682.

 PubMed Web of Science ®Google Scholar

Kulasegaram K, Mylopoulos M, Tonin P, Bernstein S, Bryden P, Law M, Lazor J, Pittini R, Sockalingam S, Tait G, et al. 2018. The alignment imperative in curriculum renewal. Med Teach. 40(5):443–448.

 PubMed Web of Science ®Google Scholar

Jamieson J, Jenkins G, Beatty S, Palermo C. 2017. Designing programmes of assessment: a participatory approach. Med Teach. 39(11):1182–1188.

 PubMed Web of Science ®Google Scholar

Driessen E, Van Tartwijk J, Govaerts M, Teunissen P, Van Der Vleuten C. 2012. The use of programmatic assessment in the clinical workplace: a Maastricht case report. Med Teach. 34(3):226–231.

 PubMed Web of Science ®Google Scholar

Ryan A, McColl G, O'Brien R, Chiavaroli N, Judd T, Finch S, Swanson D. 2017. Tensions in post-examination feedback: information for learning versus potential for harm. Med Educ. 51(9):963–973.

 PubMed Web of Science ®Google Scholar

Schuwirth L, Valentine N, Dilena P. 2017. An application of programmatic assessment for learning (PAL) system for general practice training. GMS J Med Educ. 34:Doc56.

 PubMed Web of Science ®Google Scholar

Laughlin T, Brennan A, Brailovsky C. 2012. Effect of field notes on confidence and perceived competence: survey of faculty and residents. Can Fam Phys. 58:e352–e356.

 PubMed Web of Science ®Google Scholar