Toward a neuroscience-informed evaluation of language technology

Abstract

With the ever-increasing number of available language technology products, there is also a need to evaluate them objectively. Unsubstantiated beliefs about what language technology can and cannot do inside or outside the language classroom often influence decisions about the choice of language technology to be used. The declarative/procedural model, which makes a clear distinction between the declarative and procedural memory systems, can help to provide an objective, neuroscience-informed evaluation of language technology. The central argument in this paper is that language technology caters predominantly to the declarative memory system. This system is very effective in forming explicit metalinguistic knowledge but does not lead to automatic production or procedural ability. For technology to promote procedural ability, it should instead cater to the procedural memory, which involves the implicit neurofunctional computational system. This paper provides a language technology evaluation flowchart to help professionals evaluate the language technology products they will use and gauge their expectations of those products more realistically. It also provides a list of factors to be taken into account in maximizing the benefits of technology.

Keywords:

• Declarative
• procedural
• implicit
• explicit
• evaluation of technology
• language education
• apps

Acknowledgements

The authors would like to thank Dr. Anita Bowles and the anonymous reviewers for their comments and suggestions, Mr. Michael McGuire for English-editing the manuscript, and Mr. Jeff Hansman for English-editing an earlier draft.

Disclosure statement

The views expressed in this article are those of the authors and are not necessarily the official policies or views of, or endorsed by, the Defense Language Institute Foreign Language Center (DLIFLC), or any other institutions or companies that are mentioned in this paper.

Notes

1 Note that, though important, this paper is not concerned with the technical or economic aspects of technology, such as temporary or permanent unavailability, or level of user-friendliness or complexities of websites and software, or subscription fees, among many others. The discussion applies to situations in which users do not encounter such issues and they have full access to technology.

2 Feedback and its importance are discussed later in the paper.

3 The online learning that they evaluated was not necessarily for language teaching; it was online learning in general.

4 ‘Flipped classroom’ refers to an instruction model in which students work outside the classroom on content that would traditionally be delivered in a lecture style by a teacher in the classroom. In the flipped classroom model, the bulk of class time is devoted to meaningful application activities with the focus on target language communication (Bergmann & Sams, 2012).

5 Also referred to as Paradis/Ullman perspective (Libben, 2006), Ullman/Paradis account (Clahsen & Felser, 2006), declarative/procedural model of Paradis (Falk, Lindqvist, & Bardel, 2015), among others, we interpret the two to be compatible. Any differences that may exist between Paradis’ and Ullman’s accounts are negligible for the purposes of this paper.

6 Anderson (1982), unlike Lewicki, Czyzewska, and Hoffman (1987) and Reber, Allen, and Regan (1985), theorizes that procedural knowledge develops from declarative knowledge. For Bialystok and Ryan (1985) and Bialystok and Smith (1985) these are independent. For them, declarative knowledge develops along a continuum from unanalyzed to analyzed, whereas procedural knowledge develops along a continuum from controlled to automatic. In the DP model, they are independent memory systems.

7 The most renowned, of course, is the ‘WUG’ test, devised by Berko-Gleason (1958). As part of this test, children are asked to complete sentences, such as this one: This is a WUG. Now there is another one. There are two of them. There are two________.

8 In fact, such limitations, coupled with problematic peer evaluation (Ware & O’Dowd, 2008), lead some researchers to question the value of even pursuing research on social networks with respect to language learning (Lamy & Mangenot, 2013).

9 This is not to say that technology will never reach a level of sophistication that incorporates all the current limitations stated above. In fact, applications that employ natural language processing are now capable of providing more detailed feedback when compared to those that are more traditional (see, e.g. Nagata, 2002). Also, see Tegmark (2017) for a brilliant account of the incredible progress made so far in computer technology and artificial intelligence, and for a prediction about the levels of sophistication they may reach in the future.

10 We left out technology in its ‘tool’ and ‘medium’ roles since, as the names of the roles indicate, they either provide access to resources or an ability to connect participants. Such technologies do not contribute to or interfere with participants’ language learning in the sense that they do not ‘teach’ content (see Garrett (2009) and Kern (2006) for their categorizations of technology). This is not to say that technology should not be used, for example, for distance education, or for conducting language classes over the Internet; in fact, there may be many reasons for doing so. See, for example, Schwienhorst (2011) for a thorough discussion about the role multi-user domains, object oriented may play in language learning, and in helping establish or support learner autonomy. However, when technology is used as a medium, we should keep in mind that communicating over the Internet is not intrinsically superior to face-to-face communication, as it will not necessarily involve procedural memory to any greater degree. For example, in an action research involving three separate groups, one engaging in technology-mediated task-based instructional design, one engaging in the same task-based – but no technology – design, and a third that engaged in equivalent textbook activities, Solares (2014) found that the three groups achieved similar linguistic gains.

11 Here, the verb ‘learn’ is not used contrastively with ‘acquire,’ which would have been a better choice. ‘Learn,’ here, should be interpreted to mean as forming neuronal connections.

12 The involvement of the procedural memory system is possible here because at least some of the sentences that are repeated within the same or different modules of a given technology product, such as Duolingo, are likely to share the same underlying syntactic structure. For example, the two sentences, La bicicleta no es pequeña ‘The bicycle is not small’ and El tren no es rojo ‘The train is not red,’ share the same underlying structure even though their surface forms differ. Being exposed to sentences with different surface forms but the same underlying syntactic structure repeatedly could potentially involve the procedural memory, which tallies ‘the number of times the underlying structure is implicitly abstracted’ (Paradis, 2009, p. 54). In fact, the ‘procedural memory system […] is well suited for learning implicit knowledge about rules, sequences, and categories’ (Ullman & Lovelett, 2018, p. 43) and the system ‘proceeds gradually through repeated exposure’ (Ullman & Lovelett, 2018, p. 41; also, Knowlton & Moody, 2008). Weber, Christiansen, Petersson, Indefrey and Hagoort (2016) demonstrate that brain ‘regions known to be involved in syntactic processing’ (i.e. the procedural memory system) show fMRI syntactic repetition effects, one of which is repetition enhancement (an increase in neural activity when stimuli are repeated) to infrequent unfamiliar structures (p. 6877).However, a crucial point to keep in mind about procedural memory system involvement in this case (where sentences are repeated but no novel sentences are created) is that it is likely to be very limited to perhaps acquiring a few syntactic features, if at all. Therefore, it should not be construed as being sufficient enough to enable the users of the products mentioned in this paper to acquire a foreign language at higher levels of proficiency.

13 Duolingo is a trademark of Duolingo, Inc. Babbel is a trademark operated by Lesson Nine GmbH. Kahoot! is a trademark of Kahoot! Pimsleur is a trademark of Beverly Pimsleur, used by Simon and Schuster under exclusive license. Quizlet is a trademark of Quizlet, Inc. Rosetta Stone is a trademark of Rosetta Stone, Ltd. We, the authors, have no commercial or proprietary interest in any of these companies or the products mentioned in this article. Also, we are not affiliated with any organizations or companies having a direct financial interest in the materials or products discussed in this article.

14 Turkish was chosen as the language of study in Duolingo because one of the authors of this paper is a native speaker of Turkish and this gave us a native-speaker intuition advantage when it came to acceptability judgement of sentences being presented or asked as part of an activity. Also, the same author is a subject matter expert in Turkish and has the academic background to evaluate the grammar explanations provided therein or the explanations given by other users through Duolingo’s online discussion platform.

15 The exception to that is the option to have artificial-intelligence powered simple conversations with Duolingo Bots. However, these conversations are never user-generated, and the context is limited.

16 For ILR language proficiency level descriptions, please see ILR (2011).

Additional information

Notes on contributors

M. Ali Bolgün

M. Ali Bolgün holds a PhD in Applied Linguistics and currently works as a language curriculum development (CD) advisor. His research interests include neurolinguistics, corpus linguistics, discourse analysis, and language technology and its impact on language education.

Tatiana McCaw

Tatiana McCaw holds an EdD in Instructional Leadership and currently oversees a faculty development program. Her research interests include neurolinguistics, learning strategies, critical thinking, and technology integration in foreign language teaching.