Predicting the Difficulty of EFL Tests Based on Corpus Linguistic Features and Expert Judgment

References

• Ai, H., & Lu, X. (2010, June 8–12). A web-based system for automatic measurement of lexical complexity. Paper presented at the 27th Annual Symposium of the Computer-Assisted Language Consortium (CALICO-10). Amherst, MA.
   Google Scholar
• Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508−600). Washington, DC: American Council on Education.
   Google Scholar
• Anthony, L. (2014). AntWordProfiler (Version 1.4.1) [computer software]. Tokyo, Japan: Waseda University.
   Google Scholar
• Bachman, L. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
   Google Scholar
• Bachman, L., Davidson, F., Ryan, K., & Choi, I.-C. (1995). Studies in language testing 1: An investigation into the comparability of two tests of English as a Foreign Language. Cambridge: Cambridge University Press.
   Google Scholar
• Bachman, L., & Palmer, A. (1996). Language testing in practice. Oxford: Oxford University Press.
   Google Scholar
• Beinborn, L., Zesch, T., & Gurevych, I. (2014). Predicting the difficulty of language proficiency tests. Transactions of the Association of Computational Linguistics, 2(1), 517−529. doi:10.1162/tacl_a_00200
   Google Scholar
• Blau, E. K. (1990). The effect of syntax, speed, and pauses on listening comprehension. TESOL Quarterly, 24(4), 746–753. doi:10.2307/3587129
   Web of Science ®Google Scholar
• Choi, I. C. (1994). Content and construct validation of a criterion-referenced English proficiency test. English Teaching, 48, 311−348.
   Google Scholar
• Crossley, S. A., Greenfield, J., & McNamara, D. S. (2008). Assessing text readability using cognitively based indices. TESOL Quarterly, 42(3), 475−493. doi:10.1002/j.1545-7249.2008.tb00142.x
   Web of Science ®Google Scholar
• Crossley, S. A., Skalicky, S., Dascalu, M., McNamara, D. S., & Kyle, K. (2017). Predicting text comprehension, processing, and familiarity in adult readers: New approaches to readability formulas. Discourse Processes, 54(5−6), 340−359. doi:10.1080/0163853X.2017.1296264
   Web of Science ®Google Scholar
• Dorans, N. J., Moses, T. P., & Eignor, D. R. (2010). Principles and practices of test score equating (TOEFL Research Report 10-29). Princeton, NJ: Educational Testing Service.
   Google Scholar
• Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing, 25(2), 155−185. doi:10.1177/0265532207086780
   Web of Science ®Google Scholar
• Elder, C., Iwashita, N., & McNamara, T. (2002). Estimating the difficulty of oral proficiency tasks: What does the test-taker have to offer? Language Testing, 19(4), 347−368. doi:10.1191/0265532202lt235oa
   Google Scholar
• Freedle, R., & Kostin, I. (1991). The prediction of SAT reading comprehension item difficulty for expository prose passages (TOEFL Research Report 91-29). Princeton, NJ: Educational Testing Service.
   Google Scholar
• Freedle, R., & Kostin, I. (1993a). The prediction of TOEFL reading comprehension item difficulty for expository prose passages for three item types: Main idea, inference, and supporting idea items (TOEFL Research Report 44). Princeton, NJ: Educational Testing Service.
   Google Scholar
• Freedle, R., & Kostin, I. (1993b). The prediction of TOEFL reading item difficulty: Implications for construct validity. Language Testing, 10(2), 133−170. doi:10.1177/026553229301000203
   Google Scholar
• Freedle, R., & Kostin, I. (1996). The prediction of TOEFL listening comprehension item difficulty for minitalk passages: Implications for construct validity. TOEFL Research Report 56. Princeton, NJ: Educational Testing Service.
   Google Scholar
• Grabe, W. (2009). Reading in a second language: Moving from theory to practice. Cambridge: Cambridge University Press.
   Google Scholar
• Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2), 193−202. doi:10.3758/BF03195564
   PubMedGoogle Scholar
• Hale, G. A., Rock, D. A., & Jirele, T. (1989). Confirmatory factor analysis of the test of English as a Foreign Language. TOEFL Research Report 32. Princeton, NJ: Educational Testing Service.
   Google Scholar
• Hamada, A. (2015). Linguistic variables determining the difficulty of Eiken reading passages. JLTA Journal, 18, 57−77. doi:10.20622/jltajournal.18.0_57
   Google Scholar
• Hamp-Lyons, L., & Mathias, S. P. (1994). Examining expert judgments of task difficulty on essay tests. Journal of Second Language Writing, 3(1), 49–68. doi:10.1016/1060-3743(94)90005-1
   Google Scholar
• Harley, B., & King, M. L. (1989). Verb lexis in the written compositions of young L2 learners. Studies in Second Language Acquisition, 11, 415–440. doi:10.1017/S0272263100008421
   Google Scholar
• Hoshino, A., & Nakagawa, H. (2010). Predicting the difficulty of multiple-choice close questions for computer-adaptive testing. Special Issue: Natural Language Processing and Its Applications, 279−291.
   Google Scholar
• Kolen, M., & Brennan, R. (2014). Test equating, scaling, and linking: Methods and practices (3rd ed. ed.). New York, NY: Springer.
   Google Scholar
• Laufer, B. (1994). The lexical profile of second language writing: Does it change over time? RELC Journal, 25(2), 21–33. doi:10.1177/003368829402500202
   Google Scholar
• Lewis, D. M., Mitzel, H. C., & Green, D. R. (1996). Standard setting: A Bookmark approach. In D. R. Green (Chair), IRT-based standard setting procedures utilizing behavioral anchoring. Symposium conducted at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.
   Google Scholar
• Loukina, A., Yoon, S. Y., Sakano, J., Wei, Y., & Sheehan, K. (2016). Textual complexity as a predictor of difficulty of listening items in language proficiency tests. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers (pp. 3245−3253). Osaka, Japan.
   Google Scholar
• Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4), 474−496. doi:10.1075/ijcl.15.4.02lu
   Web of Science ®Google Scholar
• Lu, X. (2012). The relationship of lexical richness to the quality of ESL learners’ oral narratives. The Modern Language Journal, 96(2), 190−208. doi:10.1111/j.1540-4781.2011.01232_1.x
   Web of Science ®Google Scholar
• Nation, I. S. P. (2013). Learning vocabulary in another language (2nd ed.). Cambridge: Cambridge University Press.
   Google Scholar
• Nelson, J., Perfetti, C., Liben, D., & Liben, M. (2012). Measures of text difficulty: Testing their predictive value for grade levels and student performance. New York, NY: Student Achievement Partners.
   Google Scholar
• Perkins, K., Gupta, L., & Tammana, R. (1995). Predicting item difficulty in a reading comprehension test with an artificial neural network. Language Testing, 12(1), 34−53. doi:10.1177/026553229501200103
   Google Scholar
• Rupp, A. A., Garcia, P., & Jamieson, J. (2001). Combining multiple regression and CART to understand difficulty in second language reading and listening comprehension test items. International Journal of Testing, 1(3−4), 185−216.
   Google Scholar
• Sheehan, K. M., Kostin, I., Futagi, Y., & Flor, M. (2010). Generating automated text complexity classifications that are aligned with targeted text complexity standards. ETS Research Report Series, 2010(2). doi:10.1002/j.2333-8504.2010.tb02235.x
   Google Scholar
• Sydorenko, T. (2011). Item writer judgments of item difficulty versus actual item difficulty: A case study. Language Assessment Quarterly, 8(1), 34−52. doi:10.1080/15434303.2010.536924
   Web of Science ®Google Scholar
• Templin, M. (1957). Certain language skills in children: Their development and interrelationships. Minneapolis: The University of Minnesota Press.
   Google Scholar