Conceptualizing and Operationalizing Second Language Speaking Assessment: Updating the Construct for a New Century

Speaking has been a component of second language proficiency tests for at least 100 years (Weir, Vidakovic, & Galaczi, 2013). Over the course of a century, ideas about the nature of speaking ability and best practice in language assessment have evolved. Early tests that assessed phonetics knowledge, reading aloud, and dictation have been followed by more communicatively oriented paired and group format tests. In addition to better-researched task types (Taylor & Wigglesworth, 2009; Young & He, 1998), tests nowadays also have more clearly defined constructs, empirically validated assessment criteria (Fulcher, 1996; Galaczi, Ffrench, Hubbard, & Green, 2011), and provisions in place for assessor training and effects (Reed & Cohen, 2001; Winke, 2012).

Indeed, speaking contexts and second language speaking assessment have continued to evolve as the use of technology has become more widespread. Technology has made face-to-face conversations between people thousands of miles apart—once by definition a contradiction in terms—not only possible but increasingly commonplace, giving rise to learners who are “digital natives” (Prensky, 2001). Automated technologies are also beginning to be applied more widely to the delivery and scoring of speaking tests (Bernstein, 2012; Xi, Higgins, Zechner, & Williamson, 2012). It is interesting that the rise of computer-delivered tests has seen the revival of dictation and reading-aloud tasks from a century ago. As with previous periods of change, these developments have fundamental implications for how the speaking construct is conceptualized and assessed and will play a role in its future development.

Therefore, at this juncture it seems opportune to take a look at where second language speaking assessment is at present and where it might be in the future. To this end the articles in this special issue consider the state of the art for select aspects of the speaking construct and some relevant speech technologies. Galaczi and Taylor consider the notion of interactional competence, while Xu proposes a construct he calls spoken collocational competence. De Jong discusses the topic of fluency and how research in applied linguistics more broadly might benefit language assessment practice, while Isaacs discusses work on pronunciation in historical and social context and their implications not just for testing but also for teaching. Finally, Litman, Strik, and Lim provide an overview of developments in automatic speech recognition and in spoken dialogue systems, two technologies that have the potential to be useful for second language speaking assessment in the future.

While the authors were each asked to focus on a particular aspect of the speaking construct, the essential interrelatedness of these can be seen in the resulting articles here. Isaacs’s discussion of pronunciation moves into a discussion of comprehensibility, which is affected by a lexico-grammatical factor, the topic for Xu’s article. While the focus for Xu was supposed to be lexis, a major component of his construct of spoken collocational competence is fluency. De Jong’s topic of fluency discussed pauses as a feature of interaction, which is the topic of Galaczi and Taylor’s contribution. And the assessment of interactive speech is what technologies like spoken dialogue systems in Litman et al. hope to assess. There is a refusal to be constrained to particular, narrow units of analysis, which in and of itself says something about what these authors think of the speaking construct.

Another insight that emerges clearly from these articles is that speaking constructs—as reflected in rating scales and descriptors, which are the embodiment of a test’s construct (Weigle, 2002)—have not been conceptualized carefully enough. For example, (De Jong, this issue) points out that disfluencies in speech can be deliberate to achieve particular communicative purposes (e.g., to signal upcoming complex speech, for turn taking, and for floor holding), but are generally only conceived of in present-day rating scales as a deficit, even though such pragmatic competence is central to and predictive of high-level language ability (Grabowski, 2009). Few scales reflect the rich research that looks at lexis as collocations, formulaic sequences, and lexico-grammar (Xu, this issue). Interactional competence is reflected in some scales but not represented comprehensively (Galaczi & Taylor, this issue). And Isaacs notes that accent continues to appear as a criterion, even though it does not necessarily have a negative impact on intelligibility. Thus, while the development of assessment criteria has become more empirical (Fulcher, 1996), there is probably an even greater need for them to be conceptually driven, for clear thinking about constructs to be reflected in future assessment materials and practices.

In discussions about the future of speaking assessment, it is difficult not to talk about the use of technology. The discourse in this regard has often tended toward the negative, focusing on the ways it can result in the narrowing of the construct. There is of course truth in that; many present-day computer-delivered speaking tests tend to elicit relatively short, monologic speech samples, so it can be more difficult to make a validity argument for and inferences about spoken interactional ability. Spoken dialogue systems, as Litman, Strik, and Lim (this issue) discuss, are not yet mature enough to simulate spontaneous conversation, and it remains to be seen whether it ever will be. However, it should be pointed out that interactive speech in some contexts is naturally more restricted, such as in aviation English (Moder, 2012), and the technology may well be ready to deliver interactive assessments in those domains.

What can be missed in those discussions are the ways in which technology can also result in the broadening of constructs. One way might simply be by provoking discussion about the topic. Isaacs (this issue) suggests that the emphasis placed on pronunciation by automated scoring algorithms is part of the reason why there is a resurgence of interest in the topic recently. Another way is simply by newer technologies being able to do what older technologies cannot or find more difficult. Rating scales, for example, which are the traditional “technology” for scoring, are most of the time limited to four or five criteria due to the conscious attentional limitations of human raters. That these scales take the form of a grid and typically have the same number of score points across different criteria is also known to be problematic, because raters can distinguish more or fewer levels, depending on the criteria being assessed (Galaczi & Taylor, this issue; Isaacs, Trofimovich, Yu, & Muñoz Chereau, 2015). By contrast, automated marking systems are able to pick up myriad predictive features (e.g., Zechner, Higgins, & Xi, 2007)—just as human raters actually do, if not consciously or in ways that they can articulate (Lumley, 2005). Scoring is based not on an arbitrary number of levels but on relevant features being applied and weighted according to their ability to predict levels of performance, leading to finer-grain evaluation and rank ordering. Being able to separate out these criteria and applying them accordingly means more clearly described constructs and therefore more valid and defensible test outcomes.

From the foregoing it should be apparent that all technologies have their own affordances and limitations and can broaden and narrow the construct in different ways. That being the case, it is perhaps more fruitful to consider how new and emerging technologies can be used or developed in ways that are fit for purpose, to operationalize particular constructs. This way of thinking makes apparent that while spoken dialogue systems may not be close to ready for some uses, they may be for others. It helped make clear that automated speech recognition based on L1 speech is not appropriate for L2 speech, leading to further development in that area (Litman et al., this issue). This way of thinking can reveal that this century’s read-aloud task differs from last century’s read-aloud task, even if they are apparently identical on the surface, because they operationalize different constructs (Van Moere, 2012). It can potentially lead to the myriad features that automated evaluation systems pick up becoming rich feedback for learners, helping to operationalize learning-oriented assessment (Carless, 2007; Jones & Saville, 2016; Turner & Purpura, 2015). For that matter, humans and technologies can jointly be used to deliver assessment: face-to-face speaking tests can be delivered via videoconferencing technology (Nakatsuhara, Inoue, Berry, & Galaczi, 2017); or humans and computers can each score those criteria they are best at scoring (Isaacs, this issue).

The developments in technology make this an interesting and exciting time to be engaged in speaking assessment, holding the promise that speaking assessment a century hence will look very different. At the same time the current emphasis in the field on validity and validation is vitally important. It should help to ensure that technology and construct develop in tandem, rather than one outstripping the other. Indeed, however the technology develops, there is one thing that it will not be able to help with, which is determining the reasons why we would assess in the first place (McNamara & Roever, 2006). The purposes are many, and the stakes are often high. For good or for ill, in the new century as in the old, speaking remains ultimately a human enterprise.