Impact of automated writing evaluation on teacher feedback, student revision, and writing improvement

Abstract

Recent years have witnessed an increasing interest in the use of automated writing evaluation (AWE) in second language writing classrooms. This increase is partially due to the belief that AWE can assist teachers by allowing them to devote more feedback to higher-level (HL) writing skills, such as content and organization, while the technology addresses lower-level (LL) skills, such as grammar. As is speculated, student revisions will then be positively impacted. However, little evidence has supported these claims, calling into question the impact of AWE on teaching and learning. The current study explored these claims by comparing two second language writing classes that were assigned to either an AWE + teacher feedback condition or a teacher-only-feedback condition. Findings suggest that using AWE as a complement to teacher feedback did not have a significant impact on the amount of HL teacher feedback, but the teacher who did not use AWE tended to provide a greater amount of LL feedback than AWE alone. Furthermore, students seemed to revise the teacher’s LL feedback more frequently than LL feedback from the computer. Interestingly, students retained their improvement in accuracy in the long-term when they had access to AWE, but students who did not have access appeared to have lower retention. We explain the relevance of our findings in relation to an argument-based validation framework to align our work with state-of-the-art research in the field and contribute to a broader discussion about how AWE can be best provided to support second language writing development.

Keywords:

• Automated writing evaluation
• automated feedback
• teacher feedback
• student revision
• long-term language gains

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 Literature recommends that the delayed test be conducted with a longer interval than two weeks that has been the typical practice in research thus far (Karim & Nassaji, 2018). Nevertheless, after that interval, the winter break, the Persian New Year holiday, and few weeks for introducing and practicing journalistic writing (the course in which the delayed posttest was administered) did not provide us with an opportunity to give the delayed test sooner than 3 months.

2 In Wilson and Czik, (2016) coding scheme, sentences structure is described as “Pertaining to the correct expression of syntax” (e.g. Don’t start a sentence with a conjunction, fragment, or run-ons) whereas grammar is defined as “Pertaining to the correct usage of parts of speech at the word-, phrase-, or sentence-levels” (e.g. verb tense, pronoun usage and pronoun agreement, conjunctions (adding, deleting, substituting), or prepositions). We have thus separated the two in our research.

Additional information

Notes on contributors

Stephanie Link

Stephanie Link is an Assistant Professor of TESOL/Applied Linguistics at Oklahoma State University. Her research involves technology-mediated language learning with a focus on development and use of automated writing evaluation tools and intelligent tutoring systems for second language writing and written scientific communication. Her work can be found in top journals, such as CALICO Journal, Language Learning and Technology, System, and Journal of Second Language Writing. Her recent co-edited volume through Equinox Publishing, Assessment Across Online Language Education, explores critical issues in the field.

Mohaddeseh Mehrzad

Mohaddeseh Mehrzad is a Ph.D. candidate of TEFL at Shiraz University, Iran. Her research interests include computer-assisted language learning with a focus on technology-mediated second language writing, automated writing evaluation, and electronic written corrective feedback.

Mohammad Rahimi

Mohammad Rahimi is an Associate Professor of TEFL at Shiraz University, Iran, and a former Assistant Professor at Université du Québec À Montreal, Canada. His research focuses on L2 Writing Evaluation, Written Corrective Feedback, and Dynamic Assessment of Writing. He has published in Language Teaching Research, System, Reading and Writing, Journal of Response to Writing, and Language and Sociocultural Theory.