Text Authenticity in Listening Assessment: Can Item Writers Be Trained to Produce Authentic-sounding Texts?

ABSTRACT

A long-standing debate in the testing of listening concerns the authenticity of the listening input. On the one hand, listening texts produced by item writers often lack spoken language characteristics. On the other hand, real-life recordings are often too context-specific to stand alone, or not suitable for item generation. In this study, we explored the effectiveness of an existing item-writing training course to produce authentic-sounding listening texts within the constraints of test specifications. Twenty-five trainees took an online item-writing course including training on creating authentic-sounding listening texts. Prior to and after the course, they developed a listening task. The resulting listening texts were judged on authenticity by three professional item reviewers and analysed linguistically by the researchers. Additionally, we interviewed the trainees following each item writing event and analysed their online discussions from during the course. Statistical comparison of the pre-and post-course authenticity scores revealed a positive effect of the training on item-writers’ ability to produce authentic-sounding listening texts, while the linguistic analysis demonstrated that the texts produced after the training contained more instances of spoken language. The interviews and discussions revealed that item writers’ awareness of spoken language features and their text production techniques influenced their ability to develop authentic-sounding texts.

Acknowledgements

We would like to thank the item-writing trainees and item reviewers who took part in our study, as well as the assistant tutor on the item-writing training course. We also thank the anonymous reviewers of this article for their supportive feedback.

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Notes

¹ Ambiguity exists in the literature regarding listening text development terminology. What Alderson et al. (2006) call ‘adapted/simplified’ texts, Field (2019a) terms ‘semi-scripted’ – a name that Buck (2001) uses for another text development technique. What Field (2019a) calls ‘improvised’ texts, Wagner and Toth (2014) call ‘unscripted’. Throughout this article, the terms ‘adapted’, ‘improvised’, ‘semi-scripted’, ‘scripted’, and ‘authenticated’ are used as defined in this section.

Additional information

Funding

The authors acknowledge the role of the British Council in making this study possible. The British Council provided a research grant which enabled the first author to conduct part of the study under the Assessment Research Awards and Grants programme 2018. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the British Council, its related bodies or its partners.