How Many Responses Do We Need? Using Generalizability Analysis to Estimate Minimum Necessary Response Rates for Online Student Evaluations

Abstract

Construct:The study compares paper and online ratings of instructional units and analyses, with the G-study using the symmetry principle, the response rates needed to ensure acceptable precision of the measure when compliance is low. Background: Students' ratings of teaching contribute to the quality of medical training programs. To date, many schools have replaced pen-and-paper questionnaires with electronic forms, despite the lower response rates consistently reported with the latter. Few available studies have examined the effects of low response rates on the reliability and precision of the evaluation measure. Moreover, the minimum number of raters to target when response rates are low remains unclear. Approach: Descriptive data were derived from 799 students' paper and online ratings of 11 preclinical instructional units (PIUs). Reliability was assessed by Cronbach's alpha coefficients. The generalizability method applying the symmetry principle approach was used to analyze the precision of the measure with a reference standard error of mean (SEM) set at 0.10; optimization models were built to estimate minimum response rates. Results: Overall, response rates were 74% and 30% (p < .001) and PIUs ratings were 3.8 ± 0.5 and 3.6 ± 0.5 (p = .02), respectively in paper and online questionnaires. Higher SEM levels and significantly larger 95% confidence intervals of PIUs rating scores were observed with online evaluations. To keep the SEM within preset limits of precision, a minimum of 48% response rate was estimated for online formats. Conclusions: The proposed generalizability analysis allowed estimating the minimum response needed to maintain acceptable precision in online evaluations. The effects of response rates on accuracy are discussed.

Keywords:

• compliance
• response rates
• online evaluations
• evaluation of teaching and learning
• undergraduate medical education

ACKNOWLEDGMENTS

We express our gratitude to the students who responsibly evaluated all preclinical instructional units, to the tutors who teach in the preclinical years, and to the faculty in charge of the preclinical instructional units for their engagement throughout the years. The technical advice and critical review of the manuscript by Professor Jean Cardinet are kindly acknowledged.

FUNDING

This study received partial support from the Gabriella Giorgi-Cavaglieri Foundation. The opinions expressed here are those of the authors and should not be attributed to the Foundation.