Predicting the Difficulty of EFL Tests Based on Corpus Linguistic Features and Expert Judgment

ABSTRACT

This study examines the relationships among various major factors that may affect the difficulty level of language tests in an attempt to enhance the robustness of item difficulty estimation, which constitutes a crucial factor ensuring the equivalency of high-stakes tests. The observed difficulties of the reading and listening sections of two EFL tests were compared using corpus linguistic features and expert judgments, i.e., native and nonnative speakers’ perceived difficulty of the test items. The research findings are as follows: Some corpus features and the predicted difficulties demonstrated a moderate to high correlation with the test sections’ observed difficulty. The native and nonnative speakers’ predicted difficulties significantly explained the observed difficulty of the test sections, where the nonnative speakers’ predicted difficulty explained a similar variance. When entered separately, the corpus features showed a stronger explanatory power than the predicted difficulties. The corpus features and predicted difficulty together accounted for the largest variance, which was more than half of the variance of the test sections. The current study suggests that corpus features and expert judgment capture different aspects of item difficulty and future research in this area needs to consider how these two can be combined for robust item difficulty estimation.

Disclosure statement

No potential conflict of interest was reported by the authors.