Abstract
This paper reviews the literature on comparing online and paper course evaluations in higher education and provides a case study of a very large randomised trial on the topic. It presents a mixed but generally optimistic picture of online course evaluations with respect to response rates, what they indicate, and how to increase them. The paper presents a case study of 1 university and finds that means for paper course evaluations tend to be higher than for online evaluations, that the standard deviations of online evaluations are typically larger than for paper evaluations, that online evaluations take longer to complete than their paper counterparts, that students prefer online evaluations, and that factor analysis shows similar and different numbers of factors for the 2 types of evaluation, even with the same instrument and the same population. Caution is advocated in assuming that the same online and paper evaluations yield similar results.
Notes
1. H. M. Anderson, Cain, and Bird (Citation2005); Donovan, Mader, and Shinsky (Citation2007); Fike, Doyle, and Connelly (Citation2010); Johnson (Citation2003); Layne, DeCristoforo, and McGinty (Citation1999); Stowell, Addison, and Smith (Citation2012).
2. Baum, Chapman, Dommeyer, and Hanna (2001); Collings and Ballantyne (Citation2004); Crews and Curtis (Citation2011); Dommeyer, Baum, Chapman, and Hanna (Citation2002); Dommeyer, Baum, Hanna, and Chapman (Citation2004); Donovan, Mader, and Shinsky (Citation2006, 2007); Fike et al. (Citation2010); Guder and Malliaris (Citation2010); Ha, Marsh, and Jones (1998); Hardy (Citation2003); Hmieleski and Champagne (Citation2000); Johnson (Citation2003); Kasiar, Schroeder, and Holstad (Citation2002); Kuhtman (Citation2004); Layne et al. (Citation1999); Ravelli (2000); Rhea, Rovai, Ponton, Derrick, and Davis (Citation2007); Sorenson and Johnson (Citation2003); Sorenson and Reiner (Citation2003); Stowell et al. (Citation2012); Tucker, Jones, Straker, and Cole (Citation2003).
Anderson, H. M., Cain, J., & Bird, E. (2005). Online student course evaluations: Review of literature and a pilot study. American Journal of Pharmaceutical Education, 61, 34–43. Donovan, J., Mader, C., & Shinsky, J. (2007). Online vs. traditional course evaluation formats: Student perceptions. Journal of Interactive Online Learning, 6, 158–180. Fike, D. S., Doyle, D. J., & Connelly, R. J. (2010). Online vs. paper evaluations of faculty: When less is good. The Journal of Effective Teaching, 10(2), 42–54. Johnson, T. D. (2003). Online student ratings: Will students respond? New Directions for Teaching and Learning, 96, 49–59. doi:10.1002/tl.122 doi: 10.1002/tl.122 Layne, B. H., DeCristoforo, J. R., & McGinty, D. (1999). Electronic versus traditional student ratings of instruction. Research in Higher Education, 40, 221–232. doi: 10.1023/A:1018738731032 Stowell, J. R., Addison, W. E., & Smith, J. L. (2012). Comparison of online and classroom-based student evaluations of instruction. Assessment & Evaluation in Higher Education, 37, 465–473. doi: 10.1080/02602938.2010.545869 Crews, T. B., & Curtis, D. F. (2011). Online course evaluations: Faculty perspective and strategies for improved online response rates. Assessment & Evaluation in Higher Education, 36, 865–878. doi: 10.1080/02602938.2010.493970 Dommeyer, C. J., Baum, P., Chapman, K. S., & Hanna, R. W. (2002). Attitudes of business faculty towards two methods of collective teaching evaluations: Paper vs. online. Assessment & Evaluation in Higher Education, 27, 455–462. doi: 10.1080/0260293022000009320 Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29, 611–623. doi: 10.1080/02602930410001689171 Donovan, J., Mader, C. E., & Shinsky, J. (2006). Constructive student feedback: Online vs. traditional course evaluations. Journal of Interactive Online Learning, 5, 283–296. Fike, D. S., Doyle, D. J., & Connelly, R. J. (2010). Online vs. paper evaluations of faculty: When less is good. The Journal of Effective Teaching, 10(2), 42–54. Guder, F., & Malliaris, M. (2010). Online and paper course evaluations. American Journal of Business Education, 3, 131–138. Hardy, N. (2003). Online ratings: Fact and fiction. New Directions for Teaching and Learning, 96, 31–38. doi:10.1002/tl.120 doi: 10.1002/tl.120 Hmieleski, K., & Champagne, M. V. (2000, September/October). Plugging into course evaluation. The Technology Source. Retrieved from http://technologysource.org/article/plugging_in_to_ course_evaluation/ Johnson, T. D. (2003). Online student ratings: Will students respond? New Directions for Teaching and Learning, 96, 49–59. doi:10.1002/tl.122 doi: 10.1002/tl.122 Kasiar, J. B., Schroeder, S. L., & Holstad, S. G. (2002). Comparison of traditional and web-based course evaluation processes in a required, team-taught pharmacotherapy course. American Journal of Pharmaceutical Education, 66, 268–270. Kuhtman, M. (2004). Review of online ratings of instruction. College and University Journal, 80, 64–67. Layne, B. H., DeCristoforo, J. R., & McGinty, D. (1999). Electronic versus traditional student ratings of instruction. Research in Higher Education, 40, 221–232. doi: 10.1023/A:1018738731032 Rhea, N., Rovai, A., Ponton, D., Derrick, G., & Davis, J. (2007). The effect of computer-mediated communication on anonymous end-of-course teaching evaluations. International Journal on E-Learning, 6, 581–592. Sorenson, D. L., & Johnson, D. (Eds.). (2003). Online student ratings of instruction [Special issue]. New Directions for Teaching and Learning, 96, 1–112. doi: 10.1002/tl.118 Sorenson, D. L., & Reiner, C. (2003). Charting the uncharted seas of online student ratings of instruction. New Directions for Teaching and Learning, 96, 1–24. doi:10.1002/tl.118 doi: 10.1002/tl.118 Stowell, J. R., Addison, W. E., & Smith, J. L. (2012). Comparison of online and classroom-based student evaluations of instruction. Assessment & Evaluation in Higher Education, 37, 465–473. doi: 10.1080/02602938.2010.545869 Tucker, B., Jones, S., Straker, L., & Cole, J. (2003). Course evaluation on the Web: Facilitating students and teacher reflection to improve learning. New Directions for Teaching and Learning, 96, 81–93. doi:10.1002/tl.125 doi: 10.1002/tl.125 Anderson, H. M., Cain, J., & Bird, E. (2005). Online student course evaluations: Review of literature and a pilot study. American Journal of Pharmaceutical Education, 61, 34–43. Bothell, T. W., & Henderson, T. (2003). Do online ratings of instruction make sense? New Directions for Teaching and Learning, 96, 69–79. doi:10.1002/tl.124 doi: 10.1002/tl.124 Crews, T. B., & Curtis, D. F. (2011). Online course evaluations: Faculty perspective and strategies for improved online response rates. Assessment & Evaluation in Higher Education, 36, 865–878. doi: 10.1080/02602938.2010.493970 Cummings, R., Ballantyne, C., & Fowler, L. (2000, August). Online student feedback surveys: Encouraging staff and student use. Paper presented at the Teaching Evaluation Forum, Perth, Western Australia. Dommeyer, C. J., Baum, P., Chapman, K. S., & Hanna, R. W. (2002). Attitudes of business faculty towards two methods of collective teaching evaluations: Paper vs. online. Assessment & Evaluation in Higher Education, 27, 455–462. doi: 10.1080/0260293022000009320 Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29, 611–623. doi: 10.1080/02602930410001689171 Donovan, J., Mader, C. E., & Shinsky, J. (2006). Constructive student feedback: Online vs. traditional course evaluations. Journal of Interactive Online Learning, 5, 283–296. Donovan, J., Mader, C., & Shinsky, J. (2007). Online vs. traditional course evaluation formats: Student perceptions. Journal of Interactive Online Learning, 6, 158–180. Hmieleski, K., & Champagne, M. V. (2000, September/October). Plugging into course evaluation. The Technology Source. Retrieved from http://technologysource.org/article/plugging_in_to_ course_evaluation/ Johnson, T. D. (2003). Online student ratings: Will students respond? New Directions for Teaching and Learning, 96, 49–59. doi:10.1002/tl.122 doi: 10.1002/tl.122 Kuhtman, M. (2004). Review of online ratings of instruction. College and University Journal, 80, 64–67. Layne, B. H., DeCristoforo, J. R., & McGinty, D. (1999). Electronic versus traditional student ratings of instruction. Research in Higher Education, 40, 221–232. doi: 10.1023/A:1018738731032 Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment & Evaluation in Higher Education, 33, 301–314. doi: 10.1080/02602930701293231 Sorenson, D. L., & Reiner, C. (2003). Charting the uncharted seas of online student ratings of instruction. New Directions for Teaching and Learning, 96, 1–24. doi:10.1002/tl.118 doi: 10.1002/tl.118 Stowell, J. R., Addison, W. E., & Smith, J. L. (2012). Comparison of online and classroom-based student evaluations of instruction. Assessment & Evaluation in Higher Education, 37, 465–473. doi: 10.1080/02602938.2010.545869 Ballantyne, C. (2003). Online evaluations of teaching: An examination of current practice and considerations for the future. New Directions for Teaching and Learning, 96, 103–112. doi:10.1002/tl.127 doi: 10.1002/tl.127 Bothell, T. W., & Henderson, T. (2003). Do online ratings of instruction make sense? New Directions for Teaching and Learning, 96, 69–79. doi:10.1002/tl.124 doi: 10.1002/tl.124 Crews, T. B., & Curtis, D. F. (2011). Online course evaluations: Faculty perspective and strategies for improved online response rates. Assessment & Evaluation in Higher Education, 36, 865–878. doi: 10.1080/02602938.2010.493970 Fraze, S., Hardin, K., Brashears, T., Smith, J., & Lockaby, J. (2002, December). The effects of delivery mode upon survey response rate and perceived attitudes of Texas Agri-Science teachers. Paper presented at the National Agricultural Education Research Conference, Las Vegas, NV. Hmieleski, K., & Champagne, M. V. (2000, September/October). Plugging into course evaluation. The Technology Source. Retrieved from http://technologysource.org/article/plugging_in_to_ course_evaluation/ Johnson, T. D. (2003). Online student ratings: Will students respond? New Directions for Teaching and Learning, 96, 49–59. doi:10.1002/tl.122 doi: 10.1002/tl.122 Sorenson, D. L., & Reiner, C. (2003). Charting the uncharted seas of online student ratings of instruction. New Directions for Teaching and Learning, 96, 1–24. doi:10.1002/tl.118 doi: 10.1002/tl.118 Anderson, H. M., Cain, J., & Bird, E. (2005). Online student course evaluations: Review of literature and a pilot study. American Journal of Pharmaceutical Education, 61, 34–43. Badri, M. A., Abdulla, M., Kamali, M. A., & Dodeen, H. (2006). Identifying potential biasing variables in student evaluation of teaching in a newly accredited business program in UAE. International Journal of Educational Management, 20, 43–59. doi: 10.1108/09513540610639585 Barkhi, R., & Williams, P. (2010). The impact of electronic media on faculty evaluation. Assessment & Evaluation in Higher Education, 35, 241–262. doi: 10.1080/02602930902795927 Crews, T. B., & Curtis, D. F. (2011). Online course evaluations: Faculty perspective and strategies for improved online response rates. Assessment & Evaluation in Higher Education, 36, 865–878. doi: 10.1080/02602938.2010.493970 Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29, 611–623. doi: 10.1080/02602930410001689171 Donovan, J., Mader, C., & Shinsky, J. (2007). Online vs. traditional course evaluation formats: Student perceptions. Journal of Interactive Online Learning, 6, 158–180. Anderson, H. M., Cain, J., & Bird, E. (2005). Online student course evaluations: Review of literature and a pilot study. American Journal of Pharmaceutical Education, 61, 34–43. Anderson, J., Brown, G., & Spaeth, S. (2006). Online student evaluations and response rates reconsidered. Innovate, 2. Retrieved from http://www.innovateonline.info/index.php?view=article&id=301 (The article is reprinted here with permission of the publisher, The Fischler School of Education and Human Services at Nova Southeastern University.) Crews, T. B., & Curtis, D. F. (2011). Online course evaluations: Faculty perspective and strategies for improved online response rates. Assessment & Evaluation in Higher Education, 36, 865–878. doi: 10.1080/02602938.2010.493970 Donovan, J., Mader, C., & Shinsky, J. (2007). Online vs. traditional course evaluation formats: Student perceptions. Journal of Interactive Online Learning, 6, 158–180. Hmieleski, K., & Champagne, M. V. (2000, September/October). Plugging into course evaluation. The Technology Source. Retrieved from http://technologysource.org/article/plugging_in_to_ course_evaluation/ Watt, S., Simpson, C., McKillop, C., & Nunn, V. (2002). Electronic course surveys: Does automating feedback and reporting give better results? Assessment & Evaluation in Higher Education, 27, 325–337. doi: 10.1080/0260293022000001346 Avery, R. J., Bryant, W. K., Mathios, A., Kang, H., & Bell, D. (2006). Electronic course evaluations: Does an online delivery system influence student evaluations? The Journal of Economic Education, 37, 21–37. doi: 10.3200/JECE.37.1.21-37 Chang, T. (2003, April). The results of student ratings: The comparison between paper and online survey. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL. Crews, T. B., & Curtis, D. F. (2011). Online course evaluations: Faculty perspective and strategies for improved online response rates. Assessment & Evaluation in Higher Education, 36, 865–878. doi: 10.1080/02602938.2010.493970 Dommeyer, C. J., Baum, P., Chapman, K. S., & Hanna, R. W. (2002). Attitudes of business faculty towards two methods of collective teaching evaluations: Paper vs. online. Assessment & Evaluation in Higher Education, 27, 455–462. doi: 10.1080/0260293022000009320 Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29, 611–623. doi: 10.1080/02602930410001689171 Guder, F., & Malliaris, M. (2010). Online and paper course evaluations. American Journal of Business Education, 3, 131–138. Laubsch, P. (2006). Online and in-person evaluations: A literature review and exploratory comparison. MERLOT Journal of Online Learning and Teaching, 2, 62–73. Layne, B. H., DeCristoforo, J. R., & McGinty, D. (1999). Electronic versus traditional student ratings of instruction. Research in Higher Education, 40, 221–232. doi: 10.1023/A:1018738731032 Liegle, J., & McDonald, D. (2005). Lessons learned from online vs. paper-based computer information students' evaluation system. Information Systems Education Journal, 3(37), 3–14. Norris, J., & Conn, C. (2005). Investigating strategies for increasing student response rates to online-delivered course evaluations. Quarterly Review of Distance Education, 6, 13–29. Richardson, J. T. E. (2005). Instruments for obtaining student feedback: A review of the literature. Assessment & Evaluation in Higher Education, 30, 387–415. doi: 10.1080/02602930500099193 Schawitch, M. (2005, June). Online course evaluations: One institute's success in transitioning from a paper process to a completely electronic process. Paper presented at the Association for Institutional Research Forum, Atlanta, GA. Sorenson, D. L., & Johnson, D. (Eds.). (2003). Online student ratings of instruction [Special issue]. New Directions for Teaching and Learning, 96, 1–112. doi: 10.1002/tl.118 Stowell, J. R., Addison, W. E., & Smith, J. L. (2012). Comparison of online and classroom-based student evaluations of instruction. Assessment & Evaluation in Higher Education, 37, 465–473. doi: 10.1080/02602938.2010.545869 Avery, R. J., Bryant, W. K., Mathios, A., Kang, H., & Bell, D. (2006). Electronic course evaluations: Does an online delivery system influence student evaluations? The Journal of Economic Education, 37, 21–37. doi: 10.3200/JECE.37.1.21-37 Barkhi, R., & Williams, P. (2010). The impact of electronic media on faculty evaluation. Assessment & Evaluation in Higher Education, 35, 241–262. doi: 10.1080/02602930902795927 Carini, R. M., Hayek, J. C., Kuh, G. D., Kennedy, J. M., & Ouimet, J. A. (2003). College student responses to web and paper surveys: Does mode matter? Research in Higher Education, 44, 1–19. doi: 10.1023/A:1021363527731 Dommeyer, C. J., Baum, P., Chapman, K. S., & Hanna, R. W. (2002). Attitudes of business faculty towards two methods of collective teaching evaluations: Paper vs. online. Assessment & Evaluation in Higher Education, 27, 455–462. doi: 10.1080/0260293022000009320 Dommeyer, C. J., Baum, P., Hanna, R. W., & Chapman, K. S. (2004). Gathering faculty teaching evaluations by in-class and online surveys: Their effects on response rates and evaluations. Assessment & Evaluation in Higher Education, 29, 611–623. doi: 10.1080/02602930410001689171 Donovan, J., Mader, C. E., & Shinsky, J. (2006). Constructive student feedback: Online vs. traditional course evaluations. Journal of Interactive Online Learning, 5, 283–296. Fike, D. S., Doyle, D. J., & Connelly, R. J. (2010). Online vs. paper evaluations of faculty: When less is good. The Journal of Effective Teaching, 10(2), 42–54. Gamliel, E., & Davidovitz, L. (2005). Online versus traditional teaching evaluation: Mode can matter. Assessment & Evaluation in Higher Education, 30, 581–592. doi: 10.1080/02602930500260647 Guder, F., & Malliaris, M. (2010). Online and paper course evaluations. American Journal of Business Education, 3, 131–138. Heath, N. M., Lawyer, S. R., & Rasmussen, E. B. (2007). Web-based versus paper-and-pencil course evaluations. Teaching of Psychology, 34, 259–261. doi: 10.1080/00986280701700433 Johnson, T. D. (2003). Online student ratings: Will students respond? New Directions for Teaching and Learning, 96, 49–59. doi:10.1002/tl.122 doi: 10.1002/tl.122 Layne, B. H., DeCristoforo, J. R., & McGinty, D. (1999). Electronic versus traditional student ratings of instruction. Research in Higher Education, 40, 221–232. doi: 10.1023/A:1018738731032 Perrett, J. J. (2013). Exploring graduate and undergraduate course evaluations administered on paper and online: A case study. Assessment & Evaluation in Higher Education, 38, 85–93. doi:10.1080/02602938.2011.604123 doi: 10.1080/02602938.2011.604123 Sorenson, D. L., & Johnson, D. (Eds.). (2003). Online student ratings of instruction [Special issue]. New Directions for Teaching and Learning, 96, 1–112. doi: 10.1002/tl.118 Stowell, J. R., Addison, W. E., & Smith, J. L. (2012). Comparison of online and classroom-based student evaluations of instruction. Assessment & Evaluation in Higher Education, 37, 465–473. doi: 10.1080/02602938.2010.545869 Turhan, K., Yaris, F., & Nural, E. (2005). Does instructor evaluation by students using a web-based questionnaire impact instructor performance? Advances in Health Sciences Education, 10, 5–13. doi: 10.1007/s10459-004-0943-7