Can we trust teaching evaluations when response rates are not high? Implications from a Monte Carlo simulation

References

• Anderson, H. J., J. Cain, and E. Bird. 2005. “Online Student Course Evaluations: Review of Literature and a Pilot Study.” American Journal of Pharmaceutical Education 69: 34–43. doi: 10.5688/aj690234
   Web of Science ®Google Scholar
• Anderson, J., G. Brown, and S. Spaeth. 2006. “Online Student Evaluations and Response Rates Reconsidered.” Innovate: Journal of Online Education 2 (6), 1–7.
   Google Scholar
• Avery, R. J., W. K. Bryant, A. Mathios, H. Kang, and D. Bell. 2006. “Electronic Course Evaluations: Does an Online Delivery System Influence Student Evaluations?” The Journal of Economic Education 37 (1): 21–37. doi: 10.3200/JECE.37.1.21-37
   Web of Science ®Google Scholar
• Berkson, J. 1946. “Limitations of the Application of Fourfold Table Analysis to Hospital Data.” Biometrics Bulletin 2 (3): 47–53. doi: 10.2307/3002000
   Google Scholar
• Burton, W. B., A. Civitano, and P. Steiner-Grossman. 2012. “Online versus Paper Evaluations: Differences in Both Quantitative and Qualitative Data.” Journal of Computing in Higher Education 24 (1): 58–69. doi: 10.1007/s12528-012-9053-3
   Web of Science ®Google Scholar
• Campbell, H. E. 2019. “JPAE at 25: Looking Back and Moving Forward on Teaching Evaluations.” Journal of Public Affairs Education 25 (1): 23–29. doi: 10.1080/15236803.2018.1558823
   Web of Science ®Google Scholar
• Capa-Aydin, Y. 2016. “Student Evaluation of Instruction: Comparison Between In-Class and Online Methods.” Assessment & Evaluation in Higher Education 41 (1): 112–126. doi: 10.1080/02602938.2014.987106
   Web of Science ®Google Scholar
• Dommeyer, C. J., P. Baum, R. W. Hanna, and K. S. Chapman. 2004. “Gathering Faculty Teaching Evaluations by In-Class and Online Surveys: Their Effects on Response Rates and Evaluations.” Assessment & Evaluation in Higher Education 29 (5): 611–623. doi: 10.1080/02602930410001689171
   Google Scholar
• Donovan, J., C. Mader, and J. Shinsky. 2007. “Online vs. Traditional Course Evaluation Formats: Student Perceptions.” Journal of Interactive Online Learning 6 (3): 158–180.
   Google Scholar
• Estelami, H. 2015. “The Effects of Survey Timing on Student Evaluation of Teaching Measures Obtained Using Online Surveys.” Journal of Marketing Education 37 (1): 54–64. doi: 10.1177/0273475314552324
   Web of Science ®Google Scholar
• Evans, S. J. 1991. “Good Surveys Guide.” The BMJ 302: 302–303. doi: 10.1136/bmj.302.6772.302
   PubMed Web of Science ®Google Scholar
• Fike, D. S., D. J. Doyle, and R. J. Connelly. 2010. “Online vs. Paper Evaluations of Faculty: When Less is Just as Good.” The Journal of Effective Teaching 10 (2): 42–54.
   Google Scholar
• Gerbase, M. W., M. Germond, B. Cerutti, N. V. Vu, and A. Baroffio. 2015. “How Many Responses Do We Need? Using Generalizability Analysis to Estimate Minimum Necessary Response Rates for Online Student Evaluations.” Teaching and Learning in Medicine 27 (4): 395–403. doi: 10.1080/10401334.2015.1077126
   PubMed Web of Science ®Google Scholar
• Guder, F., and M. Malliaris. 2010. “Online and Paper Course Evaluations.” American Journal of Business Education 3 (2): 131–137.
   Google Scholar
• Kreiter, C. D., and V. Lakshman. 2005. “Investigating the Use of Sampling for Maximising the Efficiency of Student-Generated Faculty Teaching Evaluations.” Medical Education 39: 171–175. doi: 10.1111/j.1365-2929.2004.02066.x
   PubMed Web of Science ®Google Scholar
• Kroese, D. P., T. Brereton, T. Taimre, and Z. I. Botev. 2014. “Why the Monte Carlo Method is So Important Today.” WIRES Computational Statistics 6: 386–392. doi: 10.1002/wics.1314
   Google Scholar
• Layne, B. H., J. R. DeCristoforo, and D. McGinty. 1999. “Electronic Versus Traditional Student Ratings of Instruction.” Research in Higher Education 40: 221–232. doi: 10.1023/A:1018738731032
   Web of Science ®Google Scholar
• McPherson, M. A. 2006. “Determinants of How Students Evaluate Teachers.” The Journal of Economic Education 37 (1): 3–20. doi: 10.3200/JECE.37.1.3-20
   Web of Science ®Google Scholar
• Morrison, K. 2013. “Online and Paper Evaluations of Courses: A Literature Review and Case Study.” Educational Research and Evaluation 19 (7): 585–604. doi: 10.1080/13803611.2013.834608
   Google Scholar
• Nulty, D. D. 2008. “The Adequacy of Response Rates to Online and Paper Surveys: What Can Be Done?” Assessment & Evaluation in Higher Education 33 (3): 301–314. doi: 10.1080/02602930701293231
   Web of Science ®Google Scholar
• Oliver, R. L., and E. P. Sautter. 2005. “Using Course Management Systems to Enhance the Value of Student Evaluations of Teaching.” Journal of Education for Business 80 (4): 231–234. doi: 10.3200/JOEB.80.4.231-234
   Google Scholar
• Rea, L. M., and R. A. Parker. 2014. Designing and Conducting Survey Research: A Comprehensive Guide. San Francisco, CA: Wiley.
   Google Scholar
• Sheng, Y., and Z. Sheng. 2012. “Is Coefficient Alpha Robust to Non-Normal Data?” Frontiers in Psychology 3: 1–13, Article 34. doi: 10.3389/fpsyg.2012.00034
   PubMed Web of Science ®Google Scholar
• Sullivan, G. M. 2001. “A Primer on the Validity of Assessment Instruments.” Journal of Graduate Medical Education 3 (2): 119–120. doi: 10.4300/JGME-D-11-00075.1
   Google Scholar
• Spooren, P., B. Brockx, and D. Mortelmans. 2013. “On the Validity of Student Evaluation of Teaching: The State of the Art.” Review of Educational Research 83 (4): 598–642. doi: 10.3102/0034654313496870
   Web of Science ®Google Scholar
• Thorpe, S. W. 2002. “Online Student Evaluation of Instruction: An Investigation of Nonresponse Bias.” Paper presented at the 42nd annual forum of the association for institutional research, Toronto, June 1–6.
   Google Scholar
• Vanacore, A., and M. S. Pellegrino. 2019. “How Reliable are Students’ Evaluations of Teaching (SETs)? A Study to Test Student’s Reproducibility and Repeatability.” Social Indicators Research 146 (1-2): 77–89. doi: 10.1007/s11205-018-02055-y
   Web of Science ®Google Scholar
• Vogt, W. P. 2005. Dictionary of Statistics & Methodology: A Nontechnical Guide for the Social Sciences. 3rd ed. Los Angeles, CA: Sage.
   Google Scholar
• Webb, N. M., R. J. Shavelson, and E. H. Haertel. 2006. “Reliability Coefficients and Generalizability Theory.” Handbook of Statistics 26: 81–124. doi: 10.1016/S0169-7161(06)26004-8
   Google Scholar