Evaluating automatic speech recognition-based language learning systems: a case study

ABSTRACT

The purpose of this research was to evaluate a prototype of an automatic speech recognition (ASR)-based language learning system that provides feedback on different aspects of speaking performance (pronunciation, morphology and syntax) to students of Dutch as a second language. We carried out usability reviews, expert reviews and user tests to gain insight into the potential of this prototype and the possible ways in which it could be further adapted or improved, with a view to developing specific language learning products. The evaluation revealed that domain experts and users (teachers and students) are generally positive about the system and intend to use it if they get the opportunity. In addition, recommendations have been made which range from specific changes and additions to the system to more general statements about the pedagogical and technological issues involved. These recommendations can be useful to improve this prototype and to develop other ASR-based systems, which can be deployed either as language courseware or as research tools to investigate design hypotheses and language acquisition processes.

KEYWORDS:

• Computer-assisted language learning
• automatic speech recognition
• computer-assisted pronunciation training

Acknowledgements

The DISCO project was funded by the Dutch and Flemish Governments through the STEVIN program (http://taalunieversum.org/taal/technologie/stevin/). We would like to thank the experts from Radboud in'to Languages, Arcus College and ROC Nijmegen for their valuable feedback and the students who participated in this research for their cooperation. We are indebted to two anonymous reviewers for their useful comments.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Joost van Doremalen

Joost van Doremalen received his PhD in 2014 based on research on speech technology for second language learning. He is currently CTO of NovoLanguage, where he designs and develops innovative language learning applications.

Lou Boves

Lou Boves is emeritus professor of Language and Speech Technology at Radboud University. He has published widely on advanced methods for speech recognition and synthesis, and on novel applications of these technologies. His current interest is focused on applying insights gained from successful automatic speech recognition in models of human speech comprehension.

Jozef Colpaert

Jozef Colpaert teaches Instructional Design, Educational Technology and Computer Assisted Language Learning at the University of Antwerp, Belgium. He is the editor of Computer Assisted Language Learning (Taylor and Francis) and organizer of the International CALL Research Conferences. He is currently working on the empirical and theoretical validation of Educational Engineering, a novel instructional design and research method.

Catia Cucchiarini

Dr. Catia Cucchiarini is a senior researcher at the Centre for Language and Speech Technology of the Radboud University Nijmegen. Her research activities address speech processing, computer assisted language learning (CALL), and the application of ASR to language learning and testing.

Helmer Strik

Dr. Helmer Strik is an associate professor in Speech Science and Technology at the Radboud University Nijmegen. His fields of expertise include phonetics, speech production, automatic speech recognition, spoken dialogue systems, e-learning and e-health.