Exploring the Construct Validity of the ECCE: Latent Structure of a CEFR-Based High-Intermediate Level English Language Proficiency Test

ABSTRACT

The Common European Framework of Reference for Languages Companion Volume (CEFR CV) emphasizes macro-functions of language (i.e., reception, production, interaction, and mediation). However, there seems to be little consensus on whether the macro-functions are commensurable with CEFR-based proficiency tests. This paper focuses on the Examination for the Certificate of Competency in English (ECCE), which is based on the CEFR and assesses high-intermediate level English proficiency. Specifically, the study explores the latent structure of the ECCE and its generalizability across groups (i.e., gender, age, and first language [L1]) to examine its construct validity, dimensionality of language proficiency, and commensurability with the CEFR macro-functions. The latent structure was examined through confirmatory factor analysis using performance scores from 9,700 test-takers. The results indicated that test-takers’ performance on the ECCE could be best represented by a correlated three-factor model (i.e., reading/listening/lexico-grammar, writing, and speaking abilities). The correlated three-factor model also held irrespective of gender, age, and L1 (with the exception of vocabulary scores). Overall, the findings indicate that the correlated three-factor model is consistent with the constructs that the ECCE proposes to measure, is in line with the current multi-componential view of language proficiency, and is partly commensurate with the CEFR macro-functions.

Acknowledgments

This paper reports on research funded through Michigan Language Assessment’s Spaan Research Grant Program, 2017. We are grateful to Michigan Language Assessment for their support and feedback. We are also greatly indebted to the anonymous reviewers and the editor Dr. Constant Leung for their helpful comments and suggestions.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplementary material

Supplemental data for this article can be accessed on the publisher’s website.

Notes

¹ In this article, an L2 is used as a broad term which is referred to as a language that is not a native language, including nth (e.g., second and third) languages and foreign languages.

² The majority of test-takers of the ECCE are adolescent EFL learners. However, preteens and adults also take the ECCE for various reasons.

³ In this study, we do not cover the mode of mediation because the ECCE, as in many other tests, does not include mediation as a testing component.

⁴ A majority of ECCE test-takers are Greek-speaking. An increasing number of EFL adolescent learners from Greece take the ECCE because in Greece, English became the primary foreign language throughout formal schooling after Greece joined the European Union, and the Greece government prioritizes improving English proficiency for promoting economic competitiveness (British Council, 2018).

⁵ Michigan Language Assessment submitted test specifications that support the claim that the ECCE assesses the CEFR B2 level to the Council of Europe (Michigan Language Assessment, personal communication, 2019). However, it should be noted that the CEFR is not a standardized scale, and, thus, no institution monitors or coordinates its use (Council of Europe, 2018, p. 26).

⁶ The speaking rating rubrics are available on the Michigan Language Assessment website at http://michiganassessment.org/wp-content/uploads/2014/11/ECCE-Rating-Scale-Speaking-20140220.pdf.

⁷ The writing rating rubrics are available on the Michigan Language Assessment website at http://michiganassessment.org/wp-content/uploads/2014/11/ECCE-Rating-Scale-Writing-20140220.pdf.

⁸ Generally, a minimum of 100 observations is recommended to construct a latent variable model (Loehlin, 1992). Thus, the other L1 groups which included less than 100 test-takers were not used for testing measurement invariance.

⁹ We tested another model based on a reviewer’s suggestion which consisted of two factors: one factor included language use and control (i.e., speaking, writing, grammar, and vocabulary), and a second factor which included receptive language skills (i.e., listening and reading). The model fit was bad (SBχ²[64] = 27059.509, CFI =.652, RMSEA =.209, SRMR =.161, AIC = 413255.246, and BIC = 413449.103), and, thus, this model was not considered further.

¹⁰ To complement the results of confirmatory factor analysis, exploratory factor analysis was also conducted. For the exploratory factor analysis, a Promax rotation method (i.e., correlated solution) was used. Results indicated that the first three factors showed eigenvalues greater than 1 and explained 76.469% of the shared variance in the ECCE data. In addition, the scree plot showed noticeable declines in the eigenvalues until the third component. Thus, a three-factor solution was considered appropriate. These first three factors were the same as those of the correlated three-factor model resulting from confirmatory factor analysis (i.e., listening/reading/lexico-grammar, writing, and speaking). Thus, the results of the exploratory factor analysis support the results of the confirmatory factor analysis.

¹¹ It is possible that the test format effect could be the primary reason for the combined listening/reading/lexico-grammar factor. However, the effect of the test format cannot be examined in this study because each language skill/knowledge was measured by one response type only, not allowing for an analysis on the effect of different test formats. In addition, previous studies on the latent structure of language tests have reported that multiple-choice items loaded on different factors (e.g. Sawaki & Sinharay, 2013). Thus, using multiple choice items does not always lead to their loading on the same factor.

Additional information

Funding

This work was supported by the Michigan Language Assessment [N/A].