Will artificial intelligence revolutionise the student evaluation of teaching? A big data study of 1.6 million student reviews

Abstract

This article presents the first-ever big data study of the student evaluation of teaching (SET) using artificial intelligence (AI). We train natural language processing (NLP) models on 1.6 million student evaluations from the US and the UK. We address two research questions: (1) are these models able to predict student ratings from the student textual feedback, and (2) can AI-powered SET eliminate the problems of the traditional SET based on Likert scale surveys. We test these NLP models on a new dataset of 12,386 university reviews from the UK and on 155 SET reviews from a university that agreed to run a pilot AI project. The trained NLP models exhibited high prediction accuracy, and they learnt two biases from humans, those of extreme responding and assigning higher ratings to less demanding courses. In the future, universities will likely adopt many AI-based tools that have proved successful in client management in other sectors. However, our results make a strong case against using AI as a black box for performativity purposes. It should remain a useful tool for university administrators who are aware of the AI weaknesses documented here.

Keywords:

• Student evaluation of teaching
• university rating
• artificial intelligence
• big data

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Krzysztof Rybinski

Prof. Krzysztof Rybinski is Data science professor at Vistula University in Warsaw and a former rector (vice-chancellor) of two universities in two countries. His recent research concentrates on applications of machine learning and NLP in particular in higher education, economics and finance.

Elzbieta Kopciuszewska

Dr. Elzbieta Kopciuszewska is a dean of the Department of Art, Technology and Communication at Vistula University. Her research concentrates on the development of quality management systems, quality assessment, computer simulations for process optimisation, data modelling and forecasting.