Validating an L2 Academic Group Oral Assessment: Insights From a Spoken Learner Corpus

ABSTRACT

This study determines the fine-grained bottom-up linguistic features involved in successful second language (L2) English academic group oral tutorial discussion through the use of a spoken learner corpus composed of more than 20 hrs of L2 production. Student performances were graded by teacher-raters using a can-do rating scale, which assessed students’ ability to participate in a group academic oral discussion. The performances were transcribed and annotated for linguistic features such as L2 errors, a range of interactive and interpersonal metadiscourse features, and a range of temporal, prosodic, lexical, and syntactic markers (or “fluencemes”) of (dis)fluency. The results of the corpus study suggest that frequent use of metadiscourse is the primary indicator of raters’ positive evaluation of student performance in L2 academic tutorial discussion, alongside frequent use of discourse markers, filled pauses, and a high speech rate per minute as fluencemes. Most L2 error types and syntactic fluencemes did not particularly feature in raters’ positive (or negative) evaluations. The detailed cross-sectional data afforded by this corpus analysis serve as quantitative evidence of the linguistic features accompanying each grade awarded across the rating scale and contributing to the construct validity of the assessment.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

¹ At least, in terms of what an abstract, conceptual “native speaker” would consider as a feature facilitating the perception of fluency following Götz’s (2013) approach.

² -2LL = 2 log-likelihood statistic, χ2 = chi squared statistic.

³ β = expected regression value, Exp(B) = odds ratio.

⁴ The Pearson goodness-of-fit test should be nonsignificant if the model is to be useful.

⁵ The Negelkerk r² value is a measure of the model’s predictive power.

⁶ Tables 6 and 7, Calculated via the sum total of infinitive clauses, that relative clauses in object position, that relative clauses in subject position, past participle clauses, pied-piping relative clauses, present participle clauses, sentence relatives, split infinitives, subordinator that deletion, wh-clauses, wh-relatives in object position, wh-relatives in subject position, past participle deletion relatives and present participle deletion relatives, as tagged by the Nini (2015) MAT tagger.

⁷ Tables 6 & 7, Calculated via the sum total of infinitive clauses, that relative clauses in object position, that relative clauses in subject position, past participle clauses, pied-piping relative clauses, present participle clauses, sentence relatives, split infinitives, subordinator that deletion, wh-clauses, wh-relatives in object position, wh-relatives in subject position, past participle deletion relatives and present participle deletion relatives, as tagged by the Nini (2015) MAT tagger.