Ottawa 2020 consensus statements for programmatic assessment – 2. Implementation and practice

Abstract

Introduction

Programmatic assessment is a longitudinal, developmental approach that fosters and harnesses the learning function of assessment. Yet the implementation, a critical step to translate theory into practice, can be challenging. As part of the Ottawa 2020 consensus statement on programmatic assessment, we sought to provide descriptions of the implementation of the 12 principles of programmatic assessment and to gain insight into enablers and barriers across different institutions and contexts.

Methods

After the 2020 Ottawa conference, we surveyed 15 Health Profession Education programmes from six different countries about the implementation of the 12 principles of programmatic assessment. Survey responses were analysed using a deductive thematic analysis.

Results and Discussion

A wide range of implementations were reported although the principles remained, for the most part, faithful to the original enunciation and rationale. Enablers included strong leadership support, ongoing faculty development, providing students with clear expectations about assessment, simultaneous curriculum renewal and organisational commitment to change. Most barriers were related to the need for a paradigm shift in the culture of assessment. Descriptions of implementations in relation to the theoretical principles, across multiple educational contexts, coupled with explanations of enablers and barriers, provided new insights and a clearer understanding of the strategic and operational considerations in the implementation of programmatic assessment. Future research is needed to further explore how contextual and cultural factors affect implementation.

Keywords:

• Assessment
• feedback
• theory
• general
• portfolio

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Acknowledgements

This work was made possible thanks to the efforts of individuals, including the authors, who took the time to complete the survey questionnaire describing implementation experiences in their programmes. As such, the authors would like to acknowledge and thank the following: Marjan Govaerts, (Maastricht, the Netherlands) Lambert Schuwirth (Flinders University, Australia), Teresa Chan (McMaster, Canada), Paul Dilena (General Practice, Parkside, Australia), Vincent Arockiasamy (University of British Columbia, Canada) and Kent Hecker (University of Calgary, Canada).

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Additional information

Notes on contributors

Dario Torre

Dario Torre, MD, MPH, PhD, Professor of Medicine and Associate Director for Program Evaluation and Long Term Outcomes at Uniformed Services University of Health Sciences in Bethesda, Maryland, USA.

Neil E. Rice

Neil Rice, Head of Psychometrics and Informatics, University of Exeter, College of Medicine and Health, Exeter, UK.

Anna Ryan

Anna Ryan, MBBS, PhD, Associate Professor, Director of Assessment, Department of Medical Education, Melbourne Medical School, University of Melbourne, Australia.

Harold Bok

Harold Bok, Faculty for Veterinary Medicine, Centre for Quality Improvement in Veterinary education, Department of Veterinary Medicine, Utrecht University, Utrecht, The Netherlands.

Luke J. Dawson

Luke J. Dawson, BSc, BDS, PhD, FDSRCS(Eng), FHEA, MA (TLHE) NTF, Professor of Dental Education, Director of Undergraduate Education, School of Dentistry, Liverpool, UK.

Beth Bierer

Beth Beirer, MD, Director of Assesmsnent and Evaluation, Cleveland Clinic Learner College of Medicine of Case Western University, Cleveland, OH, USA.

Tim J. Wilkinson

Tim J. Wilkinson, MB, ChB, MD, PhD(Otago), M Clin, Ed(UNSW), FRACP FRCP(London), FANZAHPE, Director of the University of Otago MBChB programme, Education unit, University of Otago, Christchurch, New Zealand.

Glendon R. Tait

Tom Laughlin, MD, Associate Professor of Family Medicine, Department of Family Medicine, Dalhousie University, Halifax, Canada.

Tom Laughlin

Kieran Veerapen, MD, Associate Professor of medicine, Assistant Dean for Faculty Development, University of British Columbia, Vancouver, BC, Canada.

Kiran Veerapen

Glendon R. Tait, MD, MSc, Director of Student Assessment, MD Program at the University of Toronto; Associate Professor Department of Psychiatry and practices Consultation-Liaison Psychiatry with Sinai Health System, and The Wilson Centre, University of Toronto, Canada.

Sylvia Heeneman

Sylvia Heeneman, PhD, Professor of Health Profession Education at the School of Health Profession Education, Department of Pathology, Faculty of Health, Medicine and Life Sciences, Maastricht University, the Netherlands.

Adrian Freeman

Adrian Freeman, MD, MMedSci, FRCGP, FAcadMed, Professor of Medical Education at University of Exeter Medical School, examiner for the Royal College of General Practitioners, President of the European Board of Medical Assessors and Deputy chair of the GMC Panel for Tests of Competence, University of Exeter, College of Medicine and Health, Exeter, UK.

Cees van der Vleuten

Cees van der Vleuten, PhD, Professor of Education, Faculty of Health Medicine and Life Sciences, Maastricht University, The Netherlands.