https://orcid.org/0000-0002-6962-2735
![Sample our Education journals, sign in here to start your access, latest two full volumes FREE to you for 14 days]({"type":"image" , "src" : "/sda/5932/TOC-Subject-Sample-2021-Education.jpg"})
Abstract
This study presents the development of an automated tool for assessing the speech of non-native Japanese speakers. The tool integrated an accent-trained automatic speech recording system based on deep neural networks, audio-text transcription, and natural language processing technologies to perform annotation, analysis, calculation, and scoring. A dataset consisting of audio recordings from 1000 Japanese learners across 16 countries was used for machine training. Validity and reliability tests were parallelly developed to evaluate: (a) the measures’ ability to distinguish test takers with different oral proficiency levels; (b) the agreement between the annotations and scoring performed by the tool and human raters. The results, obtained through precision, recall, F-Score, quadratically-weighted kappa, proportional reduction of mean squared error, Pearson correlation, and multi-faceted Rasch measurement, demonstrated a relatively high reliability of the automated tool. The tool provides users with both a holistic score and individual scores for each criterion, allowing them to gain insights into their overall performance as well as detailed evaluations for each aspect of their proficiency. Consequently, it can be utilized for employment, education, and self-learning purposes.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Notes
1 J-CAT and TTBJ are two computerized adaptive tests used to assess Japanese language proficiency.
SPOT is a specific component of the TTBJ test.
2 We extend our sincere gratitude to the reviewer for bringing this matter to our attention.
3 English translation: Uh, at that time, um, I, uh, I cooked, cooked food for him.
4 Please refer to for the specific names of the measures associated with each code.
5 The JLPT consists of five independent levels of certification, with level 5 being the lowest and level 1 being the highest.
Additional information
Funding
Notes on contributors
Wenchao Li
Wenchao Li is an Associate Professor and Deputy Director of the Department of Japanese Studies at Zhejiang University. She obtained her M.Phil. and PhD in Linguistics from the University of Tohoku in Japan, and she also holds an M.St. in Japanese Studies with a linguistic track from the University of Oxford. Her research focuses primarily on mathematical linguistics and the incorporation of natural language processing technologies into Japanese language oral and writing assessments. Wenchao Li has published 44 academic papers in international journals and authored three books published by LINCOM Press in Germany.
Zhentao Zhong
Zhentao Zhong is a graduate student enrolled in the Graduate Schoo’s Department of Japanese Studies at Zhejiang University.
Haitao Liu
Haitao Liu holds the distinguished title of Qiushi Distinguished Professor of Linguistics and Applied Linguistics at Zhejiang University. His research is primarily focused on quantitative linguistics, corpus linguistics, cognitive linguistics, and language planning. Dr. Liu’s contributions to the field have been recognized by esteemed journals, including Lingua, Journal of Quantitative Linguistics, Journal of Chinese Linguistics, Language Problems and Language Planning, Corpus Linguistics and Linguistic Theory, Language Sciences, Complexity, Journal of Linguistics, Cognitive Linguistics, Physics of Life Reviews, EPL, PLoS One, Physica A, Phonetica, Poznań Studies in Contemporary Linguistics, Chinese Science Bulletin, Computational Linguistics, Natural Language Engineering, TEXT & TALK, Digital Scholarship in the Humanities, Discourse Studies, Discourse & Society, and Behavioral and Brain Sciences, among numerous others.