Impact of reference panel composition on scores of script concordance test assessing basic nephrology knowledge in undergraduate medical education

Abstract

Purpose

In the assessment of basic medical knowledge, the composition of the reference panel between specialists and primary care (PC) physicians is a contentious issue. We assessed the effect of panel composition on the scores of undergraduate medical students in a script concordance test (SCT).

Methods

The scale of an SCT on basic nephrology knowledge was set by a panel of nephrologists or a mixed panel of nephrologists and PC physicians. The results of the SCTs were compared with ANOVA for repeated measurements. Concordance was assessed with Bland and Altman plots.

Results

Forty-five students completed the SCT. Their scores differed according to panel composition: 65.6 ± 9.73/100 points for nephrologists, and 70.27 ± 8.82 for the mixed panel, p < 0.001. Concordance between the scores was low with a bias of −4.27 ± 2.19 and a 95% limit of agreement of −8.96 to −0.38. Panel composition led to a change in the ranking of 71% of students (mean 3.6 ± 2.6 places).

Conclusion

The composition of the reference panel, either specialist or mixed, for SCT assessment of basic knowledge has an impact on test results and student rankings.

Keywords:

• Script concordance test
• clinical reasoning
• undergraduate medical education
• basic knowledge
• evaluation test

Author contributions

JA: Conceptualization, Data curation, Formal analysis, Writing – Original Draft JT: Investigation GT: Investigation SB: Investigation BS: Investigation BP: Formal analysis CG: Investigation CP: Investigation PC: Supervision, Writing Reviewing and Editing AEH: Supervision, Writing Reviewing and Editing AL: Conceptualization, Supervision, Writing Reviewing and Editing.

Acknowledgements

The authors would like to thank Jeffrey Watts for editing the manuscript.

Disclosure statement

All authors have no conflicts of interest to this study.

Additional information

Funding

This study did not receive any funding.

Notes on contributors

Julien Aniort

Julien Aniort, MD, PhD, Department of Nephrology, University Hospital, School of Medicine, Clermont-Ferrand, France.

Jeromine Trefond

Jeromine Trefond, MD, Department of General Medicine, School of Medicine, Clermont-Ferrand, France

Gilles Tanguy

Gilles Tanguy, MD Department of General Medicine, School of Medicine, Clermont-Ferrand, France.

Stanislas Bataille

Stanislas Bataille, MD, PhD Department of Nephrology, University Hospital, School of Medicine, Marseille, France.

Stephane Burtey

Stephane Burtey, MD, PhD Department of Nephrology, Bouchard Clinic, Marseille, France.

Bruno Pereira

Bruno Pereira, PhD, Department of Biostatistics, University Hospital, Clermont-Ferrand, France.

Cyril Garrouste

Cyril Garrouste, MD, Department of Nephrology, School of Medicine, Clermont-Ferrand, France.

Carole Philipponnet

Carole Philipponnet, MD, Department of Nephrology, University Hospital, Clermont-Ferrand, France.

Pierre Clavelou

Pierre Clavelou, MD, PhD, Department of Neurology, University Hospital, dean of the school of Medicine Clermont-Ferrand France.

Anne-Elisabeth Heng

Anne-Elisabeth Heng, MD, PhD, Department of Nephrology, School of Medicine, Clermont-Ferrand, France.

Alexandre Lautrette

Alexandre Lautrette, MD, PhD, ICU, Jean-Perrin Hospital, School of Medicine, Clermont-Ferrand, France.