Impact of rating demands on rater-based assessments of clinical competence

Abstract

WHAT IS ALREADY KNOWN IN THIS AREA

•	Judgements based on observation of colleagues' performance are commonplace in medical education.
•	Increasing the complexity of the judgements needing to be made is known to increase fallibility and reduce reliability.

WHAT THIS WORK ADDS

•	Increasing the number of dimensions of performance to consider simultaneously increases fallibility and decreases reliability.
•	By contrast, extraneous distractors had little effect on judgements in this study.

SUGGESTIONS FOR FUTURE RESEARCH

•	Further assessment research is needed to take forward the investigation of assessor performance under conditions of multi-task resource overload.
•	It will then be appropriate to recommend design changes for assessments.

Keywords:

• assessment of competence
• clinical performance
• mental workload
• performance assessment
• rater cognition