Composition of the panel of reference for concordance tests: Do teaching functions have an impact on examinees’ ranks and absolute scores?

Abstract

Background: Concordance tests are designed to assess the component of uncertainty of clinical reasoning. Scoring is based on a comparison of examinees’ answers with those of a panel of reference, including their variability. This allows construction of tests that are close to real clinical life, with all its complexity and ambiguity.

Aim: This study was carried out to determine the effect of teaching functions of members composing the reference panels on students’ scores and ranking.

Methods: A group of 80 residents in family medicine from a French University (Bobigny) completed a 72-item concordance test. The answers of two panels, each made up of 29 family physicians (teaching function versus non-teaching function), were used to generate the correction keys.

Results: Correlation between the sets of data obtained with the two panels is high (ICC = 0.98). Concordance scores obtained from the teaching-function panel are higher than scores obtained from the non-teaching-function panel (72.0 versus 76.3; p < 0.001). Ranking provided by the two panels was very similar.

Conclusions: This legitimizes the use of non-teaching physicians on panels. Panel composition influenced absolute score values: Residents showed more concordance with their academic trainers than with community-based physicians.

Additional information

Notes on contributors

Bernard Charlin

BERNARD CHARLIN is Professor of Surgery and Head of the Unit of Research and Development in Health Sciences Education at the University of Montreal.

Robert Gagnon

ROBERT GAGNON is a psychometrician. He works as a research associate with the Continuing Medical Education Division at the University of Montreal.

Evelyne Sauvé

ÉVELYNE SAUVÉ is a research assistant at the Unit of Research and Development in Health Sciences Education of the University of Montreal.

Michel Coletti

MICHEL COLETTI is a professor in the unit of Family Medicine of the University of Bobigny, France.