Constructing and Validating a Q-Matrix for Cognitive Diagnostic Analyses of a Reading Test

REFERENCES

• Alderson , J. C. 2000 . Assessing reading Cambridge , , UK : Cambridge University Press. .
   Google Scholar
• Alderson , J. C. 2010 . “Cognitive diagnostic and Q-matrices in language assessment”: A commentary . Language Assessment Quarterly , 7 : 96 – 103 .
   Web of Science ®Google Scholar
• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education . 1999 . Standards for educational and psychological testing Washington , DC : American Psychological Association. .
   Google Scholar
• Bernhardt , E. B. 2011 . Understanding advanced second-language reading New York , NY : Routledge. .
   Google Scholar
• Bolt , D. , Chen , H. , DiBello , L. , Hartz , S. , Henson , R. , Roussos , L. and Templin , J. 2008 . The Arpeggio Suite: Software for Cognitive Skills Diagnostic Assessment [Computer software and manual] St. Paul , MN : Assessment Systems. .
   Google Scholar
• Buck , G. , VanEssen , T. , Tatsuoka , K. , Kostin , I. , Lutz , D. and Phelps , M. 1998 . Development, selection and validation of a set of cognitive and linguistic attributes for the SAT I Verbal: Analogy section Princeton , NJ : Educational Testing Service. . (Research Rep. No. RR-98–19)
   Google Scholar
• Close , C. N. , Davison , M. L. and Davenport , E. C. An exploratory technique for finding the Q-matrix in cognitive diagnostic assessment: Combining theory with data . Paper presented at the Annual Meeting of the National Council on Measurement in Education . Vancouver, British Columbia , Canada.
   Google Scholar
• de la Torre , J. 2009 . DINA model and parameter estimation: A didactic . Journal of Educational and Behavioral Statistics , 34 : 115 – 130 .
   Web of Science ®Google Scholar
• DiBello , L. V. , Roussos , L. A. and Stout , W. 2007 . “ Review of cognitively diagnostic assessment and a summary of psychometric models. ” . In Handbook of statistics Edited by: Rao , C. V. and Sinharay , S. Vol. 26 , 979 – 1027 . Amsterdam , , the Netherlands : Elsevier. . Psychometrics
   Google Scholar
• DiBello , L. V. and Stout , W. 2007 . Guest editors' introduction and overview: IRT-based cognitive diagnostic models and related methods . Journal of Educational Measurement , 44 : 285 – 291 .
   Google Scholar
• DiBello , L. V. , Stout , W. F. and Roussos , L. 1995 . “ Unified cognitive psychometric assessment likelihood-based classification techniques. ” . In Cognitively diagnostic assessment Edited by: Nichols , P. D. , Chipman , S. F. and Brennan , R. L. 361 – 390 . Hillsdale , NJ : Erlbaum. .
   Google Scholar
• ELI-UM . 2003 . The MELAB technical manual Retrieved from http://www.cambridgemichigan.org/sites/default/files/resources/MELAB_TechManual_2002.pdf
   Google Scholar
• Ericsson , K. A. and Simon , H. A. 1993 . Protocol analysis: Verbal reports as data Cambridge , MA : MIT Press. .
   Google Scholar
• Fleiss , J. L. 1971 . Measuring nominal scale agreement among many raters . Psychological Bulletin , 76 : 378 – 382 .
   Web of Science ®Google Scholar
• Frederickson , J. R. and Collins , A. 1989 . A systems approach to educational testing . Educational Researcher , 18 : 27 – 32 .
   Google Scholar
• Fu , J. and Li , Y. An integrated review of cognitively diagnostic psychometric models . Paper presented at the annual meeting of the National Council on Measurement in Education . Chicago , IL .
   Google Scholar
• Gao , L. and Rogers , W. T. 2010 . Use of tree-based regression in the analyses of L2 reading test items . Language Testing , 28 ( 2 ) : 1 – 28 .
   Google Scholar
• Gierl , M. J. 1997 . Comparing the cognitive representations of test developers and students on a mathematics achievement test using Bloom's taxonomy . Journal of Educational Research , 91 : 26 – 32 .
   Web of Science ®Google Scholar
• Gierl , M. J. , Alves , C. , Roberts , M. and Gotzmann , A. Using judgments from content specialists to develop cognitive models for diagnostic assessments . Paper presented at the 2009 annual meeting of the National Council on Measurement in Education . San Diego , CA .
   Google Scholar
• Gierl , M. J. and Cui , Y. 2008 . Defining characteristics of diagnostic classification models and the problem of retrofitting in cognitive diagnostic assessment . Measurement: Interdisciplinary Research & Perspective , 6 : 263 – 268 .
   Google Scholar
• Hartz , S. M. 2002 . A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality Urbana-Champaign : University of Illinois. . (Unpublished doctoral dissertation)
   Google Scholar
• Jang , E. E. 2005 . A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG-TOEFL Urbana-Champaign : University of Illinois. . (Unpublished doctoral dissertation)
   Google Scholar
• Jang , E. E. 2009 . Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for applying Fusion Model to LanguEdge assessment . Language Testing , 26 : 31 – 73 .
   Web of Science ®Google Scholar
• Kane , M. T. 2006 . “ Validation. ” . In Educational measurement, , 4th ed. Edited by: Brennan , R. L. 17 – 64 . Westport , CT : American Council on Education and Praeger. .
   Google Scholar
• Karelitz , T. M. 2008 . How binary skills obscure the transition from non-mastery to mastery . Measurement: Interdisciplinary Research & Perspective , 6 : 268 – 272 .
   Google Scholar
• Landis , J. R. and Koch , G. G. 1977 . The measurement of observer agreement for categorical data . Biometrics , 33 : 159 – 174 .
   PubMed Web of Science ®Google Scholar
• Lee , Y-W. and Sawaki , Y. 2009 . Cognitive diagnosis approaches to language assessment: An overview . Language Assessment Quarterly , 6 : 172 – 189 .
   Web of Science ®Google Scholar
• Leighton , J. P. and Gierl , M. J. 2007 . Defining and evaluating models of cognition used in educational measurement to make inferences about examinees' thinking processes . Educational Measurement: Issues and Practice , 26 : 3 – 16 .
   Google Scholar
• Leighton , J. P. , Gierl , M. J. and Hunka , S. M. 2004 . The attribute hierarchy model for cognitive assessment: A variation on Tatsuoka's rule-space approach . Journal of Educational Measurement , 41 : 205 – 237 .
   Web of Science ®Google Scholar
• Lord , F. M. and Novick , M. R. 1968 . Statistical theories of mental test scores Reading , MA : Addison-Wesley. .
   Google Scholar
• Messick , S. 1989 . “ Validity. ” . In Educational measurement, , 3rd ed. Edited by: Linn , R. 13 – 103 . Washington , DC : American Council on Education. .
   Google Scholar
• Messick , S. 1996 . Validity and washback in language testing . Language Testing , 13 : 241 – 256 .
   Google Scholar
• Moss , P. A. 2008 . Reconstructing validity . Educational Researcher , 36 : 470 – 476 .
   Google Scholar
• Ntzoufras , I. 2009 . Bayesian modeling using WinBUGS Hoboken , NJ : Wiley. .
   Google Scholar
• Patz , R. J. and Junker , B. W. 1999 . Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses . Journal of Educational and Behavioral Statistics , 24 : 342 – 366 .
   Web of Science ®Google Scholar
• Pressley , M. and Afflerbach , P. 1995 . Verbal protocols of reading: The nature of constructively responsive reading. Hillsdale , NJ : Erlbaum. .
   Google Scholar
• Román , A. I. S. 2009 . Fitting cognitive diagnostic assessment to the cognitive assessment tool for statistics Lafayette , OH : Purdue University. . (Unpublished doctoral dissertation)
   Google Scholar
• Roussos , L. A. , DiBello , L. V. , Stout , W. F. , Hartz , S. M. , Henson , R. A. and Templin , J. H. 2007 . “ The fusion model skills diagnostic system. ” . In Cognitive diagnostic assessment for education: Theory and applications Edited by: Leighton , J. and Gierl , M. 275 – 318 . New York , NY : Cambridge University Press. .
   Google Scholar
• Roussos , L. A. , Templin , J. L. and Henson , R. A. 2007 . Skills diagnosis using IRT-based latent class models . Journal of Educational Measurement , 44 : 293 – 311 .
   Web of Science ®Google Scholar
• Rupp , A. A. and Templin , J. L. 2008 . Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art . Measurement: Interdisciplinary Research and Perspectives , 6 : 219 – 262 .
   Google Scholar
• Sawaki , Y. , Kim , H. J. and Gentile , C. 2009 . Q-Matrix construction: Defining the link between constructs and test items in large-scale reading and listening comprehension assessments . Language Assessment Quarterly , 6 : 190 – 209 .
   Web of Science ®Google Scholar
• Sinharay , S. 2004 . Experiences with Markov Chain Monte Carlo convergence assessment in two psychometric examples . Journal of Educational and Behavioral Statistics , 29 : 461 – 488 .
   Web of Science ®Google Scholar
• Tatsuoka , K. K. 1983 . Rule-space: An approach for dealing with misconceptions based on item response theory . Journal of Educational Measurement , 20 : 345 – 354 .
   Web of Science ®Google Scholar
• Walker , C. M. , Azen , R. and Schmitt , T. 2006 . Statistical versus substantive dimensionality: The effect of distributional differences on dimensionality assessment using DIMTEST . Educational and Psychological Measurement , 66 : 721 – 738 .
   Web of Science ®Google Scholar
• Wang , C. and Gierl , M. J. 2011 . Using the Attribute Hierarchy Method to make diagnostic inferences about examinees' cognitive skills in critical reading . Journal of Educational Measurement , 48 : 165 – 187 .
   Web of Science ®Google Scholar
• Xu , X. and von Davier , M. 2008 . Fitting the structured General Diagnostic Model to NAEP data Princeton , NJ : Educational Testing Service. . (Research Rep. No. RR-08–27)
   Google Scholar
• Zappe , S. 2007 . Response process validation of equivalent test forms: How qualitative data can support the construct validity of multiple test forms The Pennsylvania State University, State College. . (Unpublished doctoral dissertation)
   Google Scholar