Using longitudinal progress test data to determine the effect size of learning in undergraduate medical education – a retrospective, single-center, mixed model analysis of progress testing results

ABSTRACT

Medical education research focuses on the development of efficient learning methods promoting the acquisition of student’s knowledge and competencies. Evaluation of any modification of educational approaches needs to be evaluated accordingly and a reliable effect size needs to be reached. Our aim is to provide a methodological basis to calculate effect sizes from longitudinal progress test data that can be used as reference values in further research. We used longitudinally collected progress test data and evaluated the increasing knowledge of medical students from the first to the fifth academic year. Students were asked to participate in the progress test, which consists of 200 multiple-choice questions in single best answer format with an additional ‘don’t know’ option. All available individual test scores of all progress tests (n = 10) administered between April 2012 and October 2017 were analyzed. Due to the large amount of missing test results, e.g., from students at the beginning of their studies, a linear mixed model was fitted to include all collected data. In total, we analyzed 6324 test scores provided by 2587 medical students. Mean score for medical knowledge (% correct answers) increases from 16.6% (SD: 10.8%) to 51.0% (SD: 15.7%, overall effects size using linear mixed models d = 1.55). Medical students showed a learning effect of d = 0.54 (total gain: 6.9%) between the 1^st and 2^nd, d = 0.88 (total gain: 12.0%) between the 2^nd and 3^rd, d = 0.60 (total gain: 7.9%) between the 3^rd and 4^th and d = 0.58 (total gain: 7.9%) between the 4^th and 5^th study year. We demonstrated that incomplete data from longitudinally collected progress tests can be used to acquire reliable effect size estimates. The demonstrated effects size between d = 0.53–0.9 by study year may help researchers to design studies in medical education.

KEYWORDS:

• Medical education
• undergraduate
• progress testing
• learning
• learning curve
• students
• medical

Disclosure statement

The authors declare that they have no competing interests.

Author contributions

HF designed the work and acquired data. DG planned and conducted the statistical data analysis. HF and DG interpreted the results, drafted the manuscript and approved the final version. All authors are accountable for all aspects of the work.

Supplementary material

Supplemental data for this article can be accessed here.