Situating Standard Setting within Argument-Based Validity

ABSTRACT

Although there has been substantial work on argument-based approaches to validation as well as standard-setting methodologies, it might not always be clear how standard setting fits into argument-based validity. The purpose of this article is to address this lack in the literature, with a specific focus on topics related to argument-based approaches to validation in language assessment contexts. We first argue that standard setting is an essential part of test development and validation because of the important consequences cut scores might have for decision-making. We then present the Assessment Use Argument (AUA) framework and explain how evidence from standard setting can support claims about consequences, decisions, and interpretations. We finally identify several challenges in setting cut scores in relation to the levels of the Common European Framework of Reference (CEFR) and argue that despite these challenges, standard setting is a critical component of any claim focusing on the interpretation and use of test scores in relation to the CEFR levels. We conclude that standard setting should be an integral part of the validity argument supporting score use and interpretation and should not be treated as an isolated event between the completion of test development and the reporting of scores.

Acknowledgment

The authors thank the editor and the anonymous reviewers of Language Assessment Quarterly for their feedback. The authors also thank their colleagues Don Powers, Xiaoming Xi, Gary Ockey, and Jonathan Schmidgall for their useful comments on an earlier version of this article. The authors are responsible for any errors in this publication. Any opinions expressed in this article are those of the authors and not necessarily of Educational Testing Service.

Notes

¹ We use the term “CEFR scales” (plural) to refer to what the CEFR calls “illustrative scales” (of descriptors). These scales describe various language activities and aspects of language competence across the six main levels (A1-C2) in a table format (Council of Europe, 2001, pp. 217-225). As Little (2007) points out, the CEFR contains 34 illustrative scales for language activities and 13 illustrative scales for language competences, which are summarized in a global scale; the global scale briefly describes overall communicative proficiency at each of the six levels (Council of Europe, 2001, p. 24).