How Many Responses Do We Need? Using Generalizability Analysis to Estimate Minimum Necessary Response Rates for Online Student Evaluations

REFERENCES

• Irby DM, Shannon NF, Scher M, Peckham P, Ko G, Davis E. The use of student ratings in multiinstructor courses. Journal of Medical Education 1977;52:668–73.
   PubMedGoogle Scholar
• Irby DM, Gillmore GM, Ramsey PG. Factors affecting ratings of clinical teachers by medical students and residents. Journal of Medical Education 1987;62:1–7.
   PubMedGoogle Scholar
• Albanese MA, Schuldt SS, Case DE, Brown D. The validity of lecture ratings by students and trained observers. Academic Medicine 1991;66:26–8.
   PubMed Web of Science ®Google Scholar
• Solomon DJ, Speer AJ, Rosenbraugh CJ, DiPette DJ. The reliability of medical student ratings of clinical teaching. Evaluation of Health Professions 1997;20:343–52.
   PubMed Web of Science ®Google Scholar
• Marriott DJ, Litzelman DK. Students' global assessments of clinical teachers: A reliable and valid measure of teaching effectiveness. Academic Medicine 1998;73(Suppl):S72–4.
   Google Scholar
• Anderson H, Cain J, Bird E. Online student course evaluations: Review of the literature and a pilot study. American Journal of Pharmaceutical Education 2005;69:34–43.
   Web of Science ®Google Scholar
• Leung DYP, Kember D. Comparability of data gathered from evaluation questionnaires on paper through the Internet. Research in Higher Education 2005;46:571–91.
   Web of Science ®Google Scholar
• Johnson T. Online student ratings: Will students respond? Paper presented at: The Annual Conference of the American Educational Research Association; 2002; New Orleans, LA. Retrieved from http://www.armstrong.edu/images/institutional_research/onlinesurvey_will_students_respond.pdf. Accessed October 20, 2014.
   Google Scholar
• Caruso JB, Kvavik RB. ECAR study of students and information technology, 2005: Convenience, connection, control and learning. EDUCAUSE Center for Applied Research. Retrieved from https://net.educause.edu/ir/library/pdf/ERS0506/ekf0506.pdf. Accessed October 20, 2014.
   Google Scholar
• Venette S, Sellnow D, McIntyre K. Charting new territory: Assessing the online frontier of student ratings of instruction. Assessment & Evaluation in Higher Education 2010;35:101–15.
   Web of Science ®Google Scholar
• Johnson TD. Online student ratings: Will students respond? In TD Johnson, DL Sorenson (Eds.), Online student ratings of instruction: New directions for teaching and learning (No. 96, pp. 49–59). San Francisco, CA: Jossey-Bass; 2003.
   Google Scholar
• Berk RA. Top 20 strategies to increase the online response rates of student rating scales. International Journal of Technology in Teaching and Learning 2012;8:98–107.
   Google Scholar
• Paolo AM, Bonaminio GA, Gibson C, Partridge T, Kallail K. Response rate comparisons of e-mail and mail-distributed student evaluations. Teaching and Learning in Medicine 2000;12:81–4.
   PubMed Web of Science ®Google Scholar
• Sax LJ, Gilmartin SK, Bryant AN. Assessing response rates and nonresponse bias in web and paper surveys. Research in Higher Education 2003;44:409–32.
   Web of Science ®Google Scholar
• Kongsved SM, Basnov M, Holm-Christensen K, et al. Response rate and completeness of questionnaires: a randomized study of internet versus paper-and-pencil versions. Journal of Medical Internet Research 2007;9:1–6.
   Web of Science ®Google Scholar
• van den Berg MH, Overbeek A, van der Pal HJ, Versluys AB, Bresters D, van Leeuwen FD, et al. Using web-based and paper-based questionnaires for collecting data on fertility issues among female childhood cancer survivors: Differences in response characteristics. Journal of Medical Internet Research 2011;13:e76.
   PubMed Web of Science ®Google Scholar
• Leamon MH, Fields L. Measuring teaching effectiveness in a pre-clinical multi-instructor course: A case study in the development and application of a brief instructor rating scale. Teaching and Learning in Medicine 2005;17:119–29.
   PubMed Web of Science ®Google Scholar
• Kreiter CD, Lakshman V. Investigating the use of sampling for maximizing the efficiency of student-generated faculty teaching evaluation. Medical Education 2005;39:171–5.
   PubMed Web of Science ®Google Scholar
• Bloch R, Norman G. Generalizability for the perplexed: A practical introduction and guide: AMEE Guide No. 68. Medical Teacher 2012;34:960–92.
   PubMed Web of Science ®Google Scholar
• Vu NV, van der Vleuten CPM, Lacombe G. Medical students' learning processes: A comparative and longitudinal study. Academic Medicine 1998;73:S25–7.
   PubMed Web of Science ®Google Scholar
• Gerbase MW, Germond M, Nendaz MR, Vu NV. When the evaluated becomes evaluator: What counts in students' evaluations of clerkships. Academic Medicine 2009;84:877–85.
   PubMed Web of Science ®Google Scholar
• Cardinet J, Tourneur Y, Allal L. Extension of generalizability theory and its applications in educational measurement. Journal of Educational Measurement 1981;18:183–204.
   Web of Science ®Google Scholar
• Cardinet J, Johnson S, Pini G. Applying generalizability theory using EduG. New York, NY: Taylor & Francis Group, 2009.
   Google Scholar
• Costin F, Greenough WT, Menges RJ. Student ratings of college teaching: Reliability, validity, and usefulness. Review of Educational Research 1971;41:511–35.
   Web of Science ®Google Scholar
• Albanese MA. Challenges in using rater judgments in medical education. Journal of Evaluation in Clinical Practice 2000;6:305–19.
   PubMed Web of Science ®Google Scholar
• Basnov M, Kongsved SM, Bech P, Hjollund NH. Reliability of short form-36 in an Internet- and a pen-and-paper version. Informatics for Health & Social Care, 2009;34:53–8.
   PubMed Web of Science ®Google Scholar
• Dommeyer CJ, Baum P, Hanna RW, Chapman KS. Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education 2004;29:611–23.
   Google Scholar
• Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, et al. Methods to increase response to postal and electronic questionnaires (The Cochrane Library). Cochrane Database System Review 2009;3:MR000008. doi:10.1002/14651858.MR000008.pub4.
   PubMed Web of Science ®Google Scholar
• Greenwald AG, Gillmore GM. No pain, no gain? The importance of measuring course workload in student ratings of instruction. Journal of Educational Psychology 1997;89:743–51.
   Web of Science ®Google Scholar
• Nowell C, Gale LR, Handley B. Assessing faculty performance using student evaluations of teaching in an uncontrolled setting. Assessment & Evaluation in Higher Education 2010;35:463–75.
   Web of Science ®Google Scholar
• Benton SL, Cashin WE. Student ratings on teaching: A summary of research and literature (IDEA Paper No. 50). 2012. Manhattan, KS: The IDEA Center. Retrieved from http://ideaedu.org/sites/default/files/idea-paper_50.pdf. Accessed October 15, 2014.
   Google Scholar
• Thorpe SW. Online student evaluation of instruction: An investigation on non-response bias. Paper presented at: The 42nd Annual Forum of the Association for Institutional Research; 2002; Toronto, Canada. Retrieved from http://files.eric.ed.gov/fulltext/ED472469.pdf. Accessed October 15, 2014.
   Google Scholar