Are rating scales really better than checklists for measuring increasing levels of expertise?

Abstract

Background: It is a doctrine that OSCE checklists are not sensitive to increasing levels of expertise whereas rating scales are. This claim is based primarily on a study that used two psychiatry stations and it is not clear to what degree the finding generalizes to other clinical contexts. The purpose of our study was to reexamine the relationship between increasing training and scoring instruments within an OSCE.

Approach: A 9-station OSCE progress test was administered to Internal Medicine residents in post-graduate years (PGY) 1–4. Residents were scored using checklists and rating scales. Standard scores from three administrations (27 stations) were analyzed.

Findings: Only one station produced a result in which checklist scores did not increase as a function of training level, but the rating scales did. For 13 stations, scores increased as a function of PGY equally for both checklists and rating scales.

Conclusion: Checklist scores were as sensitive to the level of training as rating scales for most stations, suggesting that checklists can capture increasing levels of expertise. The choice of which measure is used should be based on the purpose of the examination and not on a belief that one measure can better capture increases in expertise.

Acknowledgements

The authors wish to thank Katherine Scrowcroft for her help with this manuscript.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Glossary

Progress tests, checklists, rating scales: Progress tests are an assessment design in which learners at all levels are tested simultaneously on the same test but the content focuses on the knowledge or skills of a learner who has completed their studies. Feedback allows comparisons between groups of learners at different stages of learning and within any learner across their training.

Additional information

Funding

This study was funded by a research grant awarded to Timothy Wood by the Faculty of Medicine, University of Ottawa.

Notes on contributors

Timothy J. Wood

Timothy J. Wood, PhD, is a Professor at the Department of Innovation in Medical Education, University of Ottawa, Ottawa, Canada.

Debra Pugh

Debra Pugh, MD, MHPE, is an Associate Professor at the Department of Medicine, University of Ottawa, at the Ottawa Hospital, and Medical Education Advisor at Medical Council of Canada, Ottawa Canada.