The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10

Abstract

Review date: Literature search January 1990 to February 2005 (with update February 2006). Analysis completed January 2007.

Background: Health professionals are increasingly expected to identify their own learning needs through a process of ongoing self-assessment. Self-assessment is integral to many appraisal systems and has been espoused as an important aspect of personal professional behaviour by several regulatory bodies and those developing learning outcomes for clinical students.

In this review we considered the evidence base on self-assessment since Gordon's comprehensive review in 1991.

The overall aim of the present review was to determine whether specific methods of self-assessment lead to change in learning behaviour or clinical practice.

Specific objectives sought evidence for effectiveness of self-assessment interventions to:

a. improve perception of learning needs;

b. promote change in learning activity;

c. improve clinical practice;

d. improve patient outcomes.

Methods: The methods for this review were developed and refined in a series of workshops with input from an expert BEME systematic reviewer, and followed BEME guidance. Databases searched included Medline, CINAHL, BNI, Embase, EBM Collection, Psychlit, HMIC, ERIC, BEI, TIMElit and RDRB. Papers addressing self-assessment in all professions in clinical practice were included, covering under- and post-graduate education, with outcomes classified using an extended version of Kirkpatrick's hierarchy. In addition we included outcome measures of accuracy of self-assessment and factors influencing it. 5,798 papers were retrieved, 194 abstracts were identified as potentially relevant and 103 papers coded independently by pairs using an electronic coding sheet adapted from the standard BEME form. This total included 12 papers identified by hand-searches, grey literature, cited references and updating. The identification of a further 12 papers during the writing-up process resulted in a total of 77 papers for final analysis.

Results: Although a large number of papers resulted from our original search only a small proportion of these were of sufficient academic rigour to be included in our review. The majority of these focused on judging the accuracy of self-assessment against some external standard, which raises questions about assumed reliability and validity of this ‘gold standard’. No papers were found which satisfied Kirkpatrick's hierarchy above level 2, or which looked at the association between self-assessment and resulting changes in either clinical practice or patient outcomes.

Thus our review was largely unable to answer the specific research questions and provide a solid evidence base for effective self-assessment.

Despite this, there was some evidence that the accuracy of self-assessment can be enhanced by feedback, particularly video and verbal, and by providing explicit assessment criteria and benchmarking guidance. There was also some evidence that the least competent are also the least able to self-assess accurately. Our review recommends that these areas merit future systematic research to further our understanding of self-assessment.

Conclusion: As in other BEME reviews, the methodological issues emerging from this review indicate a need for more rigorous study designs. In addition, it highlights the need to consider the potential for combining qualitative and quantitative data to further our understanding of how self-assessment can improve learning and professional clinical practice.

Table

Contribution of reviewers

Brian McKinstry led the review. All reviewers conceived and designed the review. All reviewers evaluated the abstracts and relevant full text papers. All reviewers abstracted data, analysed the findings and wrote the report.

Sources of support

LTSN (now part of the Higher Education Academy), NHS Education for Scotland, University of Newcastle, University of Leeds, University of Edinburgh. Brian McKinstry is supported by the Chief Scientist Office of the Scottish Executive.

Conflicts of interest

None known

Additional information

Notes on contributors

Iain Colthart

IAIN COLTHART is Research & Information Officer at NHS Education for Scotland, Edinburgh, UK.

Gellisse Bagnall

GELLISSE BAGNALL is Educational Development Manager, NHS Education for Scotland, Glasgow, UK.

Alison Evans

ALISON EVANS is Associate Director in Postgraduate General Practice Education (Yorkshire), NHS PG Medical & Dental Education, University of Leeds, UK.

Helen Allbutt

HELEN ALLBUTT is Research and Information Officer, NHS Education for Scotland, UK.

Alex Haig

ALEX HAIG is e-Portfolio Projects Manager, NHS Education for Scotland, Edinburgh, UK.

Jan Illing

JAN ILLING is a Senior Research Associate at the Postgraduate Institute for Medicine and Dentistry at the University of Newcastle upon Tyne, UK.

Brian McKinstry

BRIAN MCKINSTRY is Senior Research Fellow in the General Practice section of the Division of Community Health Sciences, The University of Edinburgh, UK.