Human versus machine: investigating L2 learner output in face-to-face versus fully automated role-plays

Abstract

To examine the utility of spoken dialog systems (SDSs) for learning and low-stakes assessment, we administered the same role-play task in two different modalities to a group of 47 tertiary-level learners of English. Each participant completed the task in an SDS setting with a fully automated agent and engaged in the same task with a human interlocutor in a face-to-face format. Additionally, we gauged students’ perceptions of the two delivery formats. Elicited oral performances were examined for linguistic complexity (syntactic complexity, lexical variety, fluency) and pragmatic functions (number and type of requests). Learner performance data across the two delivery modes were comparable although learners spoke slightly longer in the SDS task and used significantly more turns in the face-to-face setting—a finding that may be due to participants deploying more social rapport building moves, clarification requests, and backchanneling. The attitudinal data indicate that, while many learners liked both delivery formats, there was a slight preference for the face-to-face format, mainly due to the presence of body language. Overall, results show that fully automated SDS tasks may constitute a feasible alternative to face-to-face role-plays. Nevertheless, when possible, learners should be given a choice in task format for both learning and assessment.

Keywords:

• Spoken dialogue system
• role play
• linguistic complexity
• pragmatics
• perceptions

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Veronika Timpe-Laughlin

Veronika Timpe-Laughlin (Ph.D., TU Dortmund University, Germany) is a research scientist in the Center for Language Education and Assessment Research at Educational Testing Service (ETS) in Princeton, NJ. Her research interests include pragmatics, language assessment, task-based language teaching, and technology in L2 instruction and assessment for both young and adult learners.

Tetyana Sydorenko

Tetyana Sydorenko (Ph.D., Michigan State University) is an Associate Professor of Applied Linguistics at Portland State University. Her research interests include L2 pragmatics, computer-assisted language learning, multimodality, and assessment. Her most recent investigations include the use of computer simulations for the teaching of L2 pragmatics.

Judit Dombi

Judit Dombi (Ph.D., dr. habil, University of Pécs, Hungary) is an Associate Professor of Linguistics at the University of Pécs, Hungary, where she teaches undergraduate and graduate courses in linguistics. Her fields of interest include theoretical and applied aspects of intercultural communication and pragmatics in intercultural and ELF contexts. Her recent research has focused on the communicative functions of directness and indirectness, communication asymmetries and human-machine interactions.