Advancing text-analysis to tap into the student voice: a proof-of-concept study

Abstract

Student evaluations of teaching and courses (SETs) are part of the fabric of tertiary education and quantitative ratings derived from SETs are highly valued by tertiary institutions. However, many staff do not engage meaningfully with SETs, especially if the process of analysing student feedback is cumbersome or time-consuming. To address this issue, we describe a proof-of-concept study to automate aspects of analysing student free text responses to questions. Using Quantext text analysis software, we summarise and categorise student free text responses to two questions posed as part of a larger research project which explored student perceptions of SETs. We compare human analysis of student responses with automated methods and identify some key reasons why students do not complete SETs. We conclude that the text analytic tools in Quantext have an important role in assisting teaching staff with the rigorous analysis and interpretation of SETs and that keeping teachers and students at the centre of the evaluation process is key.

Keywords:

• Text analytics
• student voice
• natural language processing
• student evaluation of teaching

Disclosure statement

The first and second authors of this paper developed the Quantext software referred to in this paper.

Additional information

Notes on contributors

Jenny McDonald

Jenny McDonald holds an honorary academic position at the University of Auckland's Centre for Learning and Research in Higher Education and is an Adjunct Research Fellow at the Centre for Academic Development, Victoria University of Wellington. Her primary research interest is in the applications of text analytics for higher education.

 

Adon Christian Michael Moskal

Adon Christian Michael Moskal is a senior lecturer in Information Technology at Otago Polytechnic in Dunedin, New Zealand. His research is centred on improving teaching and learning, with a particular focus on practical applications of research for teachers “at the chalkface”. His research interests include learning analytics, educational technology and student evaluation.

 

Allen Goodchild

Allen Goodchild is the student evaluation coordinator at the University of Otago. With over 20 years of data science, questionnaire design and IT experience, his analytical work has informed student evaluation systems, financial regulation, accident insurance, and psychological research. Current topics of special interest include engaging students to improve teaching, system development, and text analysis. Prior to his current role he worked at the United Kingdom’s Financial Conduct Authority, Barclay’s Bank, and New Zealand’s Accident Compensation Corporation. He has an MSc and BSc in Psychology.

 

Sarah Stein

Sarah J. Stein is the Director, Distance Learning at the University of Otago, New Zealand. She researches and publishes in the area of teaching, learning and curriculum across education levels, but most recently in higher education. She has a particular interest in design and technology education, distance learning, student evaluation and the professional development of educators.

 

Stuart Terry

Stuart Terry has been the Organisational Researcher at Otago Polytechnic. In this role he is responsible for institutional evaluation, quality systems, listening to the student voice and providing evidence to improve student satisfaction with their learning experience. Stuart has a strong interest in quality systems and evaluation based on data and evidence within the Institute of Technology and Polytechnic (ITP) sector of tertiary education in New Zealand. In his role he has overseen the redesign of methods to gather and report feedback from students, and how teaching staff use the data to inform teaching practices. Since 2012 Stuart has also coordinated benchmarking projects on student satisfaction and graduate outcomes with six ITPs. Stuart has published journal articles and presented at a number of conferences in the USA and Australasia on engaging students and teachers in feedback and evaluation. He is currently undertaking doctoral studies on professional identity in the vocation educations sector of higher education.