Developing and Validating a Computerized Adaptive Testing System for Measuring the English Proficiency of Taiwanese EFL University Students

ABSTRACT

Prompted by Taiwanese university students’ increasing demand for English proficiency assessment, the absence of a test designed specifically for this demographic subgroup, and the lack of a localized and freely-accessible proficiency measure, this project set out to develop and validate a computerized adaptive English proficiency testing (E-CAT) system for Taiwanese EFL university students. Drawing on the guidelines posited by L2 testing specialists, we devised and followed a six-stage procedure to develop this E-CAT system: determining the test purpose, defining the construct, developing the test items, designing the administration processes, performing the field testing, and constructing the E-CAT system. Upon its completion, we performed two validation studies on the simulated data and as such offered the backing for the generalization inference and the explanation inference, which combined to lend support for the validity argument for the E-CAT score-based interpretations and uses. This project highlighted the importance of test localization, foregrounded the utility of computerized adaptive testing and item response theory in language test development and validation, and generated a localized, free-of-charge English proficiency test for Taiwanese university students to satisfy the graduation benchmark requirement and/or to demonstrate their English proficiency levels when job hunting.

本研究計劃運用試題反應理論及電腦適性測驗的技術, 來建置一項針對臺灣大學生所設計之電腦化英語適性測驗 (E-CAT), 並進一步為其成績推論進行效度驗證。透過參考語言測驗學者之建議, 我們首先採取6項步驟來建置E-CAT:確立測驗目標, 定義測驗構念, 設計測驗題項, 定調施測流程, 進行題項預試以及建置E-CAT系統。接著, 我們透過模擬研究來檢測其效度論證之概化推論及解釋推論是否成立, 因而藉此為E-CAT成績推論提出了部分效度證據。本研究計劃除了突顯測驗在地化對測驗效度之重要性, 也示範了試題反應理論及電腦適性測驗在語言測驗上之應用, 並提供臺灣大學生一項免費且具效度之測驗來作為其展示英語能力之工具

KEYWORDS:

• Language proficiency assessment
• test localization
• computerized adaptive testing
• computerized adaptive testing

 :

• 語言能力測驗
• 測驗在地化
• 電腦適性測驗
• 試題反應理論

Acknowledgments

The authors wish to express their gratitude to the Ministry of Education in Taiwan and National Sun Yat-sen University for their sponsorship for this E-CAT development and validation project.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been republished with minor changes. These changes do not impact the academic content of the article.

Notes

¹ Although the GEPT has been developed as a localized English proficiency assessment battery that caters to the test population in Taiwan, it does not target specifically the needs of university students, and it charges test fees to the examinees.

² We derived the warrants for these inferences from those by Chapelle et al. (2008) since this research project also targeted English proficiency assessment.

³ This research project has been sponsored jointly by the KTP Regional Resource Center for Teaching and Learning and National Sun Yat-sen University in Taiwan for the ongoing development and maintenance of the E-CAT system. However, the budget has currently been earmarked only for the development and validation of the reading and listening sub-tests.

Additional information

Funding

This work was supported by the Ministry of Education of Taiwan.