Scenario-Based Spoken Interaction with Virtual Agents

Abstract

This paper describes a CALL approach which integrates software for speaker independent continuous speech recognition with embodied virtual agents and virtual worlds to create an immersive environment in which learners can converse in the target language in contextualised scenarios. The result is a self-access learning package: SPELL (Spoken Electronic Language Learning). The SPELL system has been implemented to run in real time on a standard desktop PC. A prototype of SPELL is in user testing with learners of Italian and learners of Japanese in five high schools in Scotland. The speech recogniser is programmed to recognise grammatical and some ungrammatical utterances so that the learner can receive feedback on their language production. The dialogues can be modified in cases of communication difficulties, and are reactive to the learner's spoken input so that the learner receives relevant and immediate feedback to their utterance. Feedback takes two key forms: reformulations, where the system modifies the initial input, and recasts, where the system repeats the learner's utterance, implicitly correcting any errors. The SPELL system will offer a test bed for the effectiveness of the feedback within a self-access virtual learning environment.

Acknowledgements

The authors wish to acknowledge project funding from the Scottish Enterprise Proof of Concept Fund which has made this development possible; and the generous support of Nuance Communications Inc in this work.