A prediction-based method to estimate student learning outcome: Impact of response rate and gender differences on evaluation results

Abstract

Background

Low response rates threaten the reliability and validity of student evaluations of teaching. Previous research has shown that asking students to predict how satisfied their fellow students were with a course produces reliable results at lower response rates. The aim of this study was to investigate whether this prediction-based method can also be used to evaluate student learning outcome.

Methods

Before and after a cardiorespiratory module, 128 fourth-year medical students provided self-assessments and predictions of performance on 27 specific learning objectives and took formative tests on the respective contents. Pre-post performance gain was compared across all three modalities.

Results

Formative exam results indicated a performance gain of 63.0%. Self-assessed and prediction-based performance gains were identical (67.8%) but both slightly overestimated actual performance gain. Irrespective of the method used, a response rate of 20% was sufficient to produce reliable results. Compared to male students, females greatly overestimated their peers’ performance which led to inflated performance gain values.

Conclusions

Student self-assessments and predictions are equally valid sources of learning outcome measures, and low response rates are sufficient to produce stable results. When using a prediction-based approach, a tendency to overestimate learning outcome in female students needs to be taken into account.

Keywords:

• Evaluation
• learning outcome
• self-assessment
• predictions
• gender

Ethical approval

This study was approved by the local Ethics Committee (application number 14/9/16), and all participants provided written consent.

Acknowledgements

We would like to thank all medical students who devoted their time to this study.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

Binia-Laureen Grebener

Binia-Laureen Grebener is a former medical student at Göttingen University. She is currently preparing her doctoral thesis on course evaluation and working as an assistant physician in the Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy at Asklepios Hospital Harburg.

Janina Barth

Janina Barth, MMel, is a medical education researcher based at Göttingen University Medical Centre. She is in charge of course and teacher evaluations. In addition, she is involved in the design and the delivery of the local faculty development programme.

Sven Anders

Sven Anders, MD, MME, works as a consultant in the Department of Legal Medicine at Hamburg University, co-ordinating the department’s teaching activities. Main research areas are forensic pathology, clinical forensic medicine, and medical education.

Tim Beißbarth

Tim Beißbarth, PhD, heads the Department of Medical Bioinformatics at University Medical Center Göttingen.

Tobias Raupach

Tobias Raupach, MD, MME, is a cardiologist and head of the Department of Medical Education at Bonn University Medical Centre. His current research focuses on test-enhanced learning, assessment formats and evaluation.