Standard Setting to an International Reference Framework: Implications for Theory and Practice

Abstract

Standard setting theory has largely developed with reference to a typical situation, determining a level or levels of performance for one exam for one context. However, standard setting is now being used with international reference frameworks, where some parameters and assumptions of classical standard setting do not hold. We consider the challenges standard setting poses to reference frameworks and vice versa, focusing on the acceptance within standard setting theory of divergent outcomes. We argue that the justification for it does not hold in the context of reference frameworks; convergent outcomes should be expected and divergences investigated. The argument is illustrated using work relating the International English Language Testing System, an English language proficiency examination, to the Common European Framework of Reference for Languages (CEFR), a reference framework of language ability. We describe a standard setting study and a criterion validation study, show how their results agree, and reconcile findings with those from other studies. Implications for standard setting and for the CEFR are discussed.

Keywords:

• Common European Framework of Reference
• IELTS
• language testing
• standard setting
• validation
• validity