A Cognitive Diagnostic Assessment Study of the Listening Test of the Singapore–Cambridge General Certificate of Education O-Level: Application of DINA, DINO, G-DINA, HO-DINA, and RRUM

References

• Ackerman, T. A., Gierl, M. J., & Walker, C. M. (2003). Using multidimensional item response theory to evaluation educational and psychological tests. Educational Measurement, Issues, and Practice, 22, 37–50. doi:10.1111/j.1745-3992.2003.tb00136.x
   Google Scholar
• Aryadoust, V. (2018a). An integrated cognitive model of comprehension. International Journal of Listening. doi:10.1080/10904018.2017.1397519
   Google Scholar
• Aryadoust, V. (2018b, July). Dynamics of reading strategies in computerized listening tests: An eye-tracking study. Paper presented at the Pacific Rim objective measurement symposium, Fudan University, China.
   Google Scholar
• Bachman, L. F. (1990). Fundamental considerations in language testing. Oxford: Oxford University Press.
   Google Scholar
• Banerjee, J., & Papageorgiou, S. (2016). What’s in a topic? Exploring the interaction between test-taker age and item content in high-stakes testing. International Journal of Listening, 30, 8–24. doi:10.1080/10904018.2015.1056876
   Google Scholar
• Bejar, I., Douglas, D., Jamieson, J., Nissan, S., & Turner, J. (2000). TOEFL 2000 listening framework: A working paper (TOEFL Monograph Series No. MS-19). Princeton, NJ: ETS.
   Google Scholar
• Buck, G. (2001). Assessing listening. Cambridge, England: Cambridge University Press.
   Google Scholar
• Buck, G., & Tatsuoka, L. (1998). Application of the rule-space procedure to language testing examining attributes of a free response listening test. Language Testing, 15, 119–157. doi:10.1177/026553229801500201
   Google Scholar
• Chen, H. L., & Chen, J. S. (2016). Retrofitting non-cognitive-diagnostic reading assessment under the generalized DINA model framework. Language Assessment Quarterly, 13, 218–230. doi:10.1080/15434303.2016.1210610
   Web of Science ®Google Scholar
• Chen, W. H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265–289. doi:10.3102/10769986022003265
   Web of Science ®Google Scholar
• Core Team, R. (2015). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. ISBN 3-900051-07-0. Retrieved from http://www.R-project.org
   Google Scholar
• Cui, Y., Gierl, M. J., & Chang, H. H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49, 19–38. doi:10.1111/j.1745-3984.2011.00158.x
   Web of Science ®Google Scholar
• Cumming, A. (2015). Design in four diagnostic language assessments. Language Testing, 32, 407–416. doi:10.1177/0265532214559115
   Web of Science ®Google Scholar
• de la Torre, J. (2008). An empirically based method of Q-matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45, 343–362. doi:10.1111/j.1745-3984.2008.00069.x
   Web of Science ®Google Scholar
• de la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33, 163–183. doi:10.1177/0146621608320523
   Web of Science ®Google Scholar
• de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76, 179–199. doi:10.1007/s11336-011-9207-7
   Web of Science ®Google Scholar
• de la Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69, 333–353. doi:10.1007/BF02295640
   Web of Science ®Google Scholar
• de la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20, 89–97. doi:10.1016/j.pse.2014.11.001
   Google Scholar
• DeCarlo, L. T. (2010). On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models. Journal of Mathematical Psychology, 54, 304–313. doi:10.1016/j.jmp.2010.01.001
   Web of Science ®Google Scholar
• DiBello, L. V., Roussos, L. A., & Stout, W. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. Handbook of Statistics, 26, 1–49. doi:10.1016/S0169-7161(06)26031-0
   Google Scholar
• DiBello, L. V., Stout, W. F., & Roussos, L. A. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In P. D. Nichols, S. F. Chipman, & R. L. Brennan (Eds.), Cognitive diagnostic assessment (pp. 361–389). Mahwah, NJ: Erlbaum.
   Google Scholar
• Dunkel, P., Henning, G., & Chaudron, C. (1993). The assessment of an L2 listening comprehension construct: A tentative model for test specification and development. Modern Language Journal, 77, 180–191. doi:10.1111/j.1540-4781.1993.tb01962.x
   Web of Science ®Google Scholar
• George, A. C., & Robitzsch, A. (2015). Cognitive Diagnosis Models in R: A didactic. The Quantitative Methods for Psychology, 11, 189–205. doi:10.20982/tqmp.11.3.p189
   Web of Science ®Google Scholar
• George, A. C., Robitzsch, A., Kiefer, T., Groß, J., & Ünlü, A. (2016). The R package CDM for cognitive diagnosis modeling. Journal of Statistical Software, 74. doi:10.18637/jss.v074.i02
   Web of Science ®Google Scholar
• Goh, C. (2008). Metacognitive instruction for second language listening development: Theory, practice and research implications. RELC Journal, 39, 188–213. doi:10.1177/0033688208092184
   Google Scholar
• Green, A., Unaldi, A., & Weir, C. J. (2010). Empiricism versus connoisseurship: Establishing the appropriacy of texts in tests of academic reading. Language Testing, 27, 191–211. doi:10.1177/0265532209349471
   Web of Science ®Google Scholar
• Haertel, E. H. (1989). Using restricted latent class models to map the skill structure of achievement items. Journal of Educational Measurement, 26, 301–321. doi:10.1111/j.1745-3984.1989.tb00336.x
   Web of Science ®Google Scholar
• Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality. Dissertation Abstracts International: Section B: the Sciences and Engineering, 63(2–B), 864. Retrieved from http://psycnet.apa.org/record/2002-95016-234
   Google Scholar
• Hildyard, A., & Olson, D. (1978). Memory and inference in the comprehension of oral and written discourse. Discourse Processes, 1, 91–107. doi:10.1080/01638537809544431
   Google Scholar
• Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118
   Web of Science ®Google Scholar
• Jiang, H. (1996). Applications of computational statistics in cognitive diagnosis and IRT modeling ( Unpublished doctoral dissertation). University of Illinois, Champaign, IL.
   Google Scholar
• Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25, 258–272. doi:10.1177/01466210122032064
   Web of Science ®Google Scholar
• Kasai, M., & Saito, J. (1996, April). The rule space model applied to the reading comprehension section of the Test of English as a Foreign Language (TOEFL). Paper presented at the annual meeting of the national council in measurement in education, New York, NY.
   Google Scholar
• Kim, A. Y. A. (2015). Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing, 32, 227–258. doi:10.1177/0265532214558457
   Web of Science ®Google Scholar
• Kintsch, W. (1998). Comprehension: A paradigm for cognition. New York, NY: Cambridge University Press.
   Google Scholar
• Lee, W., Brennan, R. L., & Wan, L. (2009). Classification consistency and accuracy for complex assessments under the compound multinomial model. Applied Psychological Measurement, 33, 374–390. doi:10.1177/0146621608321759
   Web of Science ®Google Scholar
• Lee, Y. W., & Sawaki, Y. (2009). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6, 239–263. doi:10.1080/15434300903079562
   Web of Science ®Google Scholar
• Li, H. (2011). Evaluating language group differences in the subskills of reading using a cognitive diagnostic modeling and differential skill functioning approach ( Doctoral dissertation). Pennsylvania State University, State College, PA.
   Google Scholar
• Liao, Y. (2007). Investigating the construct validity of the grammar and vocabulary section and the listening section of the ECCE: Lexico-grammatical ability as a predictor of L2 listening ability. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 5, 37–78. Ann Arbor, MI: University of Michigan English Language Institute.
   Google Scholar
• Ockey, G., Papageorgiou, S., & French, R. (2016). Effects of strength of accent on an L2 interactive lecture listening comprehension test. International Journal of Listening, 30, 84–98. doi:10.1080/10904018.2015.1056877
   Google Scholar
• Oller, J. W., Jr. (1983). Evidence for a general proficiency factor: An expectancy grammar. In J. W. Oller Jr. (Ed.), Issues in language testing research (pp. 3–10). Rowley, MA: Newbury House Publishers.
   Google Scholar
• Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale, NJ: Erlbaum.
   Google Scholar
• Ranjbaran, F., & Alavi, S. M. (2017). Developing a reading comprehension test for cognitive diagnostic assessment: A RUM analysis. Studies in Educational Evaluation, 55, 167–179. doi:10.1016/j.stueduc.2017.10.007
   Web of Science ®Google Scholar
• Ravand, H. (2016). Application of a cognitive diagnostic model to a high-stakes reading comprehension test. Journal of Psychoeducational Assessment, 34, 782–799. doi:10.1177/0734282915623053
   Web of Science ®Google Scholar
• Revesz, A., & Brunfaut, T. (2013). Text characteristics of task input and difficulty in second language listening comprehension. Studies in Second Language Acquisition, 35, 31–65. doi:10.1017/S0272263112000678
   Web of Science ®Google Scholar
• Rost, M. (2016). Teaching and researching listening (3rd ed.). London, UK: Longman.
   Google Scholar
• Ruhm, R., Leitner-Jones, C., Kulmhofer, A., Kiefer, T., Mlakar, H., & Itzlinger-Bruneforth, U. (2016). Playing the recording once or twice: Effects on listening test performances. International Journal of Listening, 30, 67–83. doi:10.1080/10904018.2015.1104252
   Google Scholar
• Rupp, A., & Templin, J. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement: Interdisciplinary Research and Perspectives, 6, 219–262. doi:10.1080/15366360802490866
   Google Scholar
• Sawaki, Y., Kim, H.-J., & Gentile, C. (2009). Q-Matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments. Language Assessment Quarterly, 6, 190–209. doi:10.1080/15434300902801917
   Web of Science ®Google Scholar
• Seo, D., Taherbhai, H., & Frantz, R. (2016). Psychometric evaluation and discussions of English language learners’ listening comprehension. International Journal of Listening, 30, 47–66. doi:10.1080/10904018.2015.1065747
   Google Scholar
• Singapore Examinations and Assessment Board (SEAB) (2017). GCE O-Level: General information. Retrieved from http://www.seab.gov.sg/pages/nationalExaminations/GOL/general.asp
   Google Scholar
• Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11, 287–305. doi:10.1037/1082-989X.11.3.287
   PubMed Web of Science ®Google Scholar
• Vandergrift, L. (2007). Recent developments in second and foreign language listening comprehension research. Language Teaching, 40, 191–210. doi:10.1017/S0261444807004338
   Google Scholar
• Vandergrift, L., & Goh, C. C. M. (2012). Teaching and learning second language listening: Metacognition in action. New York, NY: Routledge.
   Google Scholar
• Von Davier, M. (2005). A general diagnostic model applied to language testing data (ETS Research Rep. No. RR-05-16). Princeton, NJ: ETS.
   Google Scholar
• Wagner, E. (2004). A construct validation study of the extended listening sections of the ECPE and MELAB. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 2, 1–23. Ann Arbor, MI: University of Michigan English Language Institute.
   Google Scholar
• Wang, L., Song, L., Chen, P., Meng, Y., & Ding, S. (2015). Attribute-level and pattern-level classification consistency and accuracy indices for cognitive diagnostic assessment. Journal of Educational Measurement, 52, 457–476. doi:10.1111/jedm.12096/full
   Web of Science ®Google Scholar
• Wolfgramm, C., Suter, N., & Göksel, E. (2016). Examining the role of concentration, vocabulary and self-concept in listening and reading comprehension. International Journal of Listening, 30, 25–46. doi:10.1080/10904018.2015.1065746
   Google Scholar
• Yamaguchi, K., & Okada, K. (2018). Comparison among cognitive diagnostic models for the TIMSS 2007 fourth grade mathematics assessment. PLoS ONE, 13(2), e0188691. doi:10.1371/journal.pone.0188691
   PubMed Web of Science ®Google Scholar
• Yeldham, M. (2016). Second language listening instruction: Comparing a strategies-based approach with an interactive, strategies/bottom-up skills approach. TESOL Quarterly, 50, 394–420. doi:10.1002/tesq.233
   Web of Science ®Google Scholar
• Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125–145. doi:10.1177/014662168400800201
   Web of Science ®Google Scholar
• Yi, Y. S. (2017). In search of optimal cognitive diagnostic model(s) for ESL grammar test data. Applied Measurement in Education, 30, 82–101. doi:10.1080/08957347.2017.1283314
   Web of Science ®Google Scholar