Diagnostic Classification Models: Recent Developments, Practical Issues, and Prospects

REFERENCES

• American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
   Google Scholar
• Baghaei, P. (2012). The application of multidimensional Rasch models in large scale assessment and validation: An empirical example. Electronic Journal of Research in Educational Psychology, 10, 233–252.
   Google Scholar
• Barnes, T. (2010). Novel derivation and application of skill matrices: The Q-matrix method. In C. Ramero, S. Vemtora, M. Pechemizkiy, & R. S. J. de Baker (Eds.), Handbook of educational data mining (pp. 159–172). Boca Raton, FL: Chapman & Hall.
   Google Scholar
• Bolt, D., Chen, H., DiBello, L., Hartz, S., Henson, R. A., & Templin, J. L. (2008). The Arpeggio Suite: Software for cognitive skills diagnostic assessment {computer software and manual}. St. Paul, MN: Assessment Systems.
   Google Scholar
• Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123–140. doi:10.1111/j.1745-3984.2012.00185.x
   Web of Science ®Google Scholar
• Chen, Y., Li, X., Liu, J., & Ying, Z. (2017). Regularized latent class analysis with application in cognitive diagnosis. Psychometrika, 82(3), 660–692. doi:10.1007/s11336-016-9545-6
   Web of Science ®Google Scholar
• Chen, Y., Liu, J., Xu, G., & Ying, Z. (2015). Statistical analysis of Q-matrix based diagnostic classification models. Journal of the American Statistical Association, 110(510), 850–866. doi:10.1080/01621459.2014.934827
   PubMed Web of Science ®Google Scholar
• Chen, W. H., & Thissen, D. (1997). Local dependence indexes for item pairs using item response theory. Journal of Educational and Behavioral Statistics, 22, 265–289. doi:10.3102/10769986022003265
   Web of Science ®Google Scholar
• Chiu, C.-Y. (2013). Statistical refinement of the Q-matrix in cognitive diagnosis. Applied Psychological Measurement, 37(8), 598–618. doi:10.1177/0146621613488436
   Web of Science ®Google Scholar
• Chiu, C.-Y., Douglas, J. A., & Li, X. (2009). Cluster analysis for cognitive diagnosis: Theory and applications. Psychometrika, 74(4), 633–665. doi:10.1007/s11336-009-9125-0
   Web of Science ®Google Scholar
• Collins, L. M., & Wugalter, S. E. (1992). Latent class models for stage-sequential dynamic latent variables. Multivariate Behavioral Research, 27(1), 131–157. doi:10.1207/s15327906mbr2701_8
   Web of Science ®Google Scholar
• Cui, Y., Gierl, M. J., & Chang, H. H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49(1), 19–38. doi:10.1111/j.1745-3984.2011.00158.x
   Web of Science ®Google Scholar
• De Carlo, L. T. (2012). Recognizing uncertainty in the Q-matrix via a Bayesian extension of the DINA model. Applied Psychological Measurement, 36(6), 447–468.
   Web of Science ®Google Scholar
• de la Torre, J. (2008). An empirically based method of Q‐matrix validation for the DINA model: Development and applications. Journal of Educational Measurement, 45(4), 343–362. doi:10.1111/j.1745-3984.2008.00069.x
   Web of Science ®Google Scholar
• de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179–199.
   Web of Science ®Google Scholar
• de la Torre, J., & Chiu, C. -Y. (2016). General method of empirical Q-matrix validation. Psychometrika, 81(2), 253–273. doi:10.1007/s11336-015-9467-8
   PubMed Web of Science ®Google Scholar
• de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595. doi:10.1007/s11336-008-9063-2
   Web of Science ®Google Scholar
• de la Torre, J., & Lee, Y. S. (2013). Evaluating the Wald test for item‐level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50(4), 355–373. doi:10.1111/jedm.12022
   Web of Science ®Google Scholar
• de la Torre, J., & Minchen, N. (2014). Cognitively diagnostic assessments and the cognitive diagnosis model framework. Psicología Educativa, 20(2), 89–97. doi:10.1016/j.pse.2014.11.001
   Google Scholar
• Desmarais, M., & Naceur, R. (2013). A matrix factorization method for mapping items to skills and for enhancing expert-based q-matrices. In Lane, H. C., Yacef, K., Mostow, J., & Pavlik, P. (Eds.), Proceedigns of the 6th International Conference on Artificial Intelligence in Education (pp. 441–450). Heidelberg: Springer.
   Google Scholar
• DiBello, L. V., Roussos, L. A., & Stout, W. F. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In Rao, C. R., Sinharay, S. (Eds.), Handbook of statistics, vol. 26: Psychometrics (pp. 979–1030). Amsterdam, Netherlands: Elsevier.
   Google Scholar
• DiBello, L. V., Stout, W., & Roussos, L. (1995). Unified cognitive/psychometric diagnostic assessment likelihood-based classification techniques. In P. Nichols, S. Chipman, & R. Brennan (Eds.), Cognitively diagnostic assessment (pp. 361–390). Hillsdale, NJ: Lawrence Erlbaum.
   Google Scholar
• Embretson, S. E., & Yang, X. (2013). A multicomponent latent trait model for diagnosis. Psychometrika, 78(1), 14–36. doi:10.1007/s11336-012-9296-y
   PubMed Web of Science ®Google Scholar
• Galeshi, R., & Skaggs, G. (2016). Parameter recovery of a cognitive diagnostic model: Evidence from a simulation study. International Journal of Quantitative Research in Education, 3(4), 223–241. doi:10.1504/IJQRE.2016.082386
   Google Scholar
• Gierl, M. J., & Cui, Y. (2008). Defining characteristics of diagnostic classification models and the problem of retrofitting in cognitive diagnostic assessment. Measurement: Interdisciplinary Research and Perspectives, 6(4), 263–268. doi:10.1080/15366360802497762
   Google Scholar
• Gierl, M. J., Leighton, J. P., & Hunka, S. M. (2007). Using the attribute hierarchy method to make diagnostic inferences about examinees' cognitive skills. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 242–274). New York, NY: Cambridge University Press
   Google Scholar
• Gorin, J. S. (2009). Diagnostic classification models: Are they necessary? Commentary on Rupp and Templin. Measurement: Interdisciplinary Research and Perspectives, 7(1), 30–33. doi:10.1080/15366360802715387
   Google Scholar
• Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilites: Blending theory with practicality. Doctoral dissertation. University of Illinois at Urbana-Champaign.
   Google Scholar
• Henson, R. A. (2009). Diagnostic classification models: Thoughts and future directions. Measurement: Interdisciplinary Research and Perspectives, 7(1), 34–36. doi:10.1080/15366360802715395
   Google Scholar
• Henson, R. A., Templin, J. L., & Willse, J. T. (2009). Defining a family of cognitive diagnosis models using log-linear models with latent variables. Psychometrika, 74(2), 191–210. doi:10.1007/s11336-008-9089-5
   Web of Science ®Google Scholar
• Hou, L., la Torre, J. D., & Nandakumar, R. (2014). Differential item functioning assessment in cognitive diagnostic modeling: Application of the Wald test to investigate DIF in the DINA model. Journal of Educational Measurement, 51(1), 98–125. doi:10.1111/jedm.12036
   Web of Science ®Google Scholar
• Hu, J., Miller, M. D., Huggins-Manley, A. C., & Chen, Y. -H. (2016). Evaluation of model fit in cognitive diagnosis models. International Journal of Testing, 16(2), 119–141. doi:10.1080/15305058.2015.1133627
   Web of Science ®Google Scholar
• Huo, Y., & de la Torre, J. (2014). Estimating a cognitive diagnostic model for multiple strategies via the EM algorithm. Applied Psychological Measurement, 38(6), 464–485. doi:10.1177/0146621614533986
   Web of Science ®Google Scholar
• Jang, E. E. (2009). Cognitive diagnostic assessment of L2 reading comprehension ability: Validity arguments for fusion model application to LanguEdge assessment. Language Testing, 26(1), 031–073. doi:10.1177/0265532208097336
   Web of Science ®Google Scholar
• Jiao, H. (2009). Diagnostic classification models: Which one should I use? Measurement: Interdisciplinary Research and Perspectives, 7(1), 65–67. doi:10.1080/15366360902799869
   Google Scholar
• Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258–272. doi:10.1177/01466210122032064
   Web of Science ®Google Scholar
• Jurich, D. P., & Bradshaw, L. P. (2014). An illustration of diagnostic classification modeling in student learning outcomes assessment. International Journal of Testing, 14(1), 49–72. doi:10.1080/15305058.2013.835728
   Google Scholar
• Kaya, Y., & Leite, W. L. (2017). Assessing change in latent skills across time with longitudinal cognitive diagnosis modeling: An evaluation of model performance. Educational and Psychological Measurement, 77(3), 369–388. doi:10.1177/0013164416659314
   PubMed Web of Science ®Google Scholar
• Kim, A.-Y. (2015). Exploring ways to provide diagnostic feedback with an ESL placement test: Cognitive diagnostic assessment of L2 reading ability. Language Testing, 32(2), 227–258. doi:10.1177/0265532214558457
   Web of Science ®Google Scholar
• Kim, Y. H. (2011). Diagnosing EAP writing ability using the Reduced Reparameterized Unified Model. Language Testing, 28(4), 509–541. doi:10.1177/0265532211400860.
   Web of Science ®Google Scholar
• Kunina‐Habenicht, O., Rupp, A. A., & Wilhelm, O. (2012). The impact of model misspecification on parameter estimation and item‐fit assessment in log‐linear diagnostic classification models. Journal of Educational Measurement, 49(1), 59–81. doi:10.1111/j.1745-3984.2011.00160.x
   Web of Science ®Google Scholar
• Lee, Y.-W., & Sawaki, Y. (2009). Cognitive diagnosis approaches to language assessment: An overview. Language Assessment Quarterly, 6(3), 172–189. doi:10.1080/15434300902985108
   Web of Science ®Google Scholar
• Lei, P.-W., & Li, H. (2016). Performance of fit indices in choosing correct cognitive diagnostic models and Q-matrices. Applied Psychological Measurement, 40(6), 405–417. doi:10.1177/0146621616647954
   PubMed Web of Science ®Google Scholar
• Li, H. (2011). A cognitive diagnostic analysis of the MELAB reading test. Spaan Fellow, 9, 17–46.
   Google Scholar
• Li, F., Cohen, A., Bottge, B., & Templin, J. (2016). A latent transition analysis model for assessing change in cognitive skills. Educational and Psychological Measurement, 76(2), 181–204. doi:10.1177/0013164415588946
   PubMed Web of Science ®Google Scholar
• Li, H., Hunter, C. V., & Lei, P. W. (2015). The selection of cognitive diagnostic models for a reading comprehension test. Language Testing, 33, 391–409. doi:10.1177/0265532215590848
   Web of Science ®Google Scholar
• Liu, Y., Douglas, J. A., & Henson, R. A. (2009). Testing person fit in cognitive diagnosis. Applied Psychological Measurement, 33(8), 579–598. doi:10.1177/0146621609331960
   Web of Science ®Google Scholar
• Liu, R., Huggins-Manley, A. C., & Bulut, O. (2018). Retrofitting diagnostic classification models to responses from IRT-based assessment forms. Educational and Psychological Measurement, 78(3), 357–383. doi:10.1177/0013164416685599
   PubMed Web of Science ®Google Scholar
• Liu, J., Xu, G., & Ying, Z. (2012). Data-driven learning of Q-matrix. Applied Psychological Measurement, 36(7), 548–564. doi:10.1177/0146621612456591
   PubMed Web of Science ®Google Scholar
• Ma, W. & de la Torre, J. (2018). GDINA: The generalized DINA model framework. R package version 2.1. Retrieved from https://CRAN.R-project.org/package¼GDINA
   Google Scholar
• Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40(3), 200–217. doi:10.1177/0146621615621717
   Google Scholar
• Madison, M. J., & Bradshaw, L. P. (2015). The effects of Q-matrix design on classification accuracy in the log-linear cognitive diagnosis model. Educational and Psychological Measurement, 75(3), 491–511. doi:10.1177/0013164414539162
   PubMed Web of Science ®Google Scholar
• Madison, M. J., & Bradshaw, L. P. (2018). Assessing growth in a diagnostic classification model framework. Psychometrika, 83(4), 963–990. doi:10.1007/s11336-018-9638-5
   PubMed Web of Science ®Google Scholar
• Maris, E. (1999). Estimating multiple classification latent class models. Psychometrika, 64(2), 187–212. doi:10.1007/BF02294535
   Web of Science ®Google Scholar
• Maydeu-Olivares, A. (2013). Goodness-of-fit assessment of item response theory models. Measurement: Interdisciplinary Research and Perspectives, 11(3), 71–101. doi:10.1080/15366367.2013.831680
   Google Scholar
• McDonald, R. P., & Mok, M. M.-C. (1995). Goodness of fit in item response models. Multivariate Behavioral Research, 30(1), 23–40. doi:10.1207/s15327906mbr3001_2
   PubMed Web of Science ®Google Scholar
• Morrison, D. F. (1967). Multivariate statistical methods. New York, NY: McGraw-Hill.
   Google Scholar
• Ravand, H. (2016). Application of a cognitive diagnostic model to a high-stakes reading comprehension test. Journal of Psychoeducational Assessment, 34, 782–799. doi:10.1177/0734282915623053
   Web of Science ®Google Scholar
• Ravand, H. (2019). Hierarchical diagnostic classification models in assessing reading comprehension. In V. Aryadoust & M. Raquel (Eds.), Quantitative data analysis for language assessment II: Advanced methods (pp. xxx). New York, NY: Routledge.
   Google Scholar
• Ravand, H., & Baghaei, P. (2016). Partial least squares structural equation modeling with R. Practical Assessment, Research & Evaluation, 21(11). Retrieved from http://pareonline.net/getvn.asp?v=21&n=11.
   Google Scholar
• Ravand, H., & Robitzsch, A. (2015). Cognitive diagnostic modeling using R. Practical Assessment, Research & Evaluation, 20, 112.
   Google Scholar
• Ravand, H. & Robitzsch, A. (2018). Cognitive diagnostic model of best choice: A study of reading comprehension. Educational Psychology, 38(10), 1255–1277. doi:10.1080/01443410.2018.1489524
   Web of Science ®Google Scholar
• Robitzsch, A., Kiefer, T., George, A., & Uenlue, A. (2017). CDM: Cognitive diagnosis modeling. R package version 3.1-14: Retrieved from the Comprehensive R Archive Network [CRAN] website http://CRAN.R-project.org/package=CDM.
   Google Scholar
• Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement, 6(4), 219–262. doi:10.1080/15366360802490866
   Google Scholar
• Rupp, A., & Templin, J. (2009). The (un)usual suspects? A community in search of its identity. Measurement, 7,115–121. doi:10.1080/15366360903187700
   Google Scholar
• Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.
   Google Scholar
• Rupp, A. A., & van Rijn, P. W. (2018). GDINA and CDM packages in R. Measurement: Interdisciplinary Research and Perspectives, 16(1), 71–77. doi:10.1080/15366367.2018.1437243
   Web of Science ®Google Scholar
• Sinharay, S., & Haberman, S. J. (2009). How much can we reliably know about what examinees know? Measurement: Interdisciplinary Research and Perspectives, 7, 46–49. doi:10.1080/15366360802715486
   Google Scholar
• Skaggs, G., Hein, S. F., & Wilkins, J. L. (2016). Diagnostic profiles: A standard setting method for use with a cognitive diagnostic model. Journal of Educational Measurement, 53(4), 448–458. doi:10.1111/jedm.12125
   Web of Science ®Google Scholar
• Sorrel, M. A., Abad, F. J., Olea, J., de la Torre, J., & Barrada, J. R. (2017). Inferential item-fit evaluation in cognitive diagnosis modeling. Applied Psychological Measurement, 41(8), 614–631. doi:10.1177/0146621617707510
   PubMed Web of Science ®Google Scholar
• Tatsuoka, C. (2009). Diagnostic models as partially ordered sets. Measurement: Interdisciplinary Research and Perspectives, 7(1), 49–53. doi:10.1080/15366360802715510
   Google Scholar
• Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20(4), 345–354. doi:10.1111/j.1745-3984.1983.tb00212.x
   Web of Science ®Google Scholar
• Tatsuoka, K. K. (1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational Statistics, 10(1), 55–73. doi:10.3102/10769986010001055
   Google Scholar
• Templin, J. (2009). On the origin of species: The evolution of diagnostic modeling within the psychometric taxonomy (Powerpoint). International meeting of Psychometric Society pre-conference workshop, Cambridge University, England.
   Google Scholar
• Templin, J., & Bradshaw, L. (2013). Measuring the reliability of diagnostic classification model examinee estimates. Journal of Classification, 30(2), 251–275. doi:10.1007/s00357-013-9129-4
   Web of Science ®Google Scholar
• Templin, J. L., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3):287–305. doi:10.1037/1082-989X.11.3.287
   PubMed Web of Science ®Google Scholar
• von Davier, M. (2005). A general diagnostic model applied to language testing data. ETS Research Report Series, 2005(2). doi:10.1002/j.2333-8504.2005.tb01993.x
   Google Scholar
• Wang, S., Yang, Y., Culpepper, S. A., & Douglas, J. A. (2018). Tracking skill acquisition with cognitive diagnosis models: A higher-order, hidden markov model with covariates. Journal of Educational and Behavioral Statistics, 43(1), 57–87. doi:10.3102/1076998617719727
   Web of Science ®Google Scholar
• Wilhelm, O., & Robitzsch, A. (2009). Have cognitive diagnostic models delivered their goods? Some substantial and methodological concerns. Measurement: Interdisciplinary Research and Perspectives, 7(1), 46–49.
   Google Scholar
• Xu, G. and Zhang, S. (2016). Identifiability of diagnostic classification models. Psychometrika, 81(3), 625–649 doi:10.1007/s11336-015-9471-z
   PubMed Web of Science ®Google Scholar
• Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8(2), 125–145. doi:10.1177/014662168400800201
   Web of Science ®Google Scholar
• Yi, Y. (2012). Implementing a cognitive diagnostic assessment in an institutional test: a new networking model in language testing and experiment with a new psychometric model and task type. Unpublished doctoral dissertation. University of Illinois at Urbana Champaign, Urbana-Champaign, IL.
   Google Scholar
• Yi, Y.-S. (2017). In search of optimal cognitive diagnostic model(s) for ESL grammar test data. Applied Measurement in Education, 30(2), 82–101. doi:10.1080/08957347.2017.1283314
   Web of Science ®Google Scholar