Beyond correctness: development and validation of concept-based categorical scoring rubrics for diagnostic purposes

References

• Abdi, H., & Valentin, D. (2007). Multiple correspondence analysis. In N. J. Salkind (Ed.), Encyclopedia of measurement and statistics (pp. 651–657). Thousand Oaks, CA: Sage.
   Google Scholar
• Almond, R. G., DiBello, L. V., Moulder, B., & Zapata-Rivera, J.-D. (2007). Modeling diagnostic assessments with Bayesian networks. Journal of Educational Measurement, 44, 341–359.10.1111/jedm.2007.44.issue-4
   Web of Science ®Google Scholar
• Arieli-Attali, M. (2014, April). Towards meaningful scores: Linking student responses to learning progression. Paper presented at the annual meeting of the National Council on Measurement in Education, Philadelphia, PA.
   Google Scholar
• Arieli-Attali, M., Wylie, E. C., & Bauer, I. M. (2012, April). The use of three learning progressions in supporting formative assessment in middle school mathematics. Paper presented in the annual meeting of the American Educational Research Association, Vancouver, Canada.
   Google Scholar
• Bennett, R. E. (2010). Cognitively based assessment of, for, and as learning: A preliminary theory of action for summative and formative assessment. Measurement: Interdisciplinary Research and Perspectives, 8, 70–91.
   Google Scholar
• Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education: Principles, Policy & Practice, 18, 5–25.
   Google Scholar
• Bennett, R. E., & Gitomer, D. H. (2009). Transforming K-12 assessment: Integrating accountability testing, formative assessment, and professional support. In C. Wyatt-Smith & J. Cumming (Eds.), Educational assessment in the 21st century (pp. 43–61). New York, NY: Springer.10.1007/978-1-4020-9964-9
   Google Scholar
• Birenbaum, M., & Tatsuoka, K. K. (1993). Applying an IRT-based cognitive diagnostic model to diagnose students’ knowledge states in multiplication and division with exponents. Applied Measurement in Education, 6, 255–268.10.1207/s15324818ame0604_1
   Google Scholar
• Blasius, J., & Thiessen, V. (2001). Methodological artifacts in measures of political efficacy and trust: A multiple correspondence analysis. Political Analysis, 9, 1–20.10.1093/oxfordjournals.pan.a004862
   Google Scholar
• Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29–51.10.1007/BF02291411
   Web of Science ®Google Scholar
• Bradshaw, L., & Templin, J. (2014). Combining item response theory and diagnostic classification models: A psychometric model for scaling ability and diagnosing misconceptions. Psychometrika, 79, 403–425.
   PubMed Web of Science ®Google Scholar
• Carlson, M., Oehrtman, M., & Thompson, P. W. (2005). Key aspects of knowing and learning the concept of function. Mathematical Association of America, Research Sampler (9).
   Google Scholar
• Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1981). Categorization and representation of physics problems by experts and novices. Cognitive Science, 5, 121–152.10.1207/s15516709cog0502_2
   Web of Science ®Google Scholar
• Confrey, J. (1990). A review of research on student conceptions in mathematics, science and programming. In C. Cazden (Ed.), Review research in education (Vol. 16, pp. 3–56). Washington, DC: American Educational Research Association.
   Google Scholar
• Craik, F. I., & Lockhart, R. S. (1972). Levels of processing: A framework for memory research. Journal of Verbal Learning and Verbal Behavior, 11, 671–684.10.1016/S0022-5371(72)80001-X
   Google Scholar
• de la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple-choice options. Applied Psychological Measurement, 33, 163–183.10.1177/0146621608320523
   Web of Science ®Google Scholar
• Dugdale, S. (1993). Functions and graphs: Perspectives on student thinking. In T. A. Romberg, E. Fennema, & T. P. Carpenter (Eds.), Integrating research on the graphical representation of functions (pp. 101–130). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Embretson, S. (1984). A general latent trait model for response processes. Psychometrika, 49, 175–186.10.1007/BF02294171
   Web of Science ®Google Scholar
• Embretson, S. E. (1998). A cognitive design system approach to generating valid tests: Application to abstract reasoning. Psychological Methods, 3, 380–396.10.1037/1082-989X.3.3.380
   Web of Science ®Google Scholar
• Embretson, S., & Gorin, J. (2001). Improving construct validity with cognitive psychology principles. Journal of Educational Measurement, 38, 343–368.10.1111/jedm.2001.38.issue-4
   Web of Science ®Google Scholar
• Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York, NY: John Wiley. ISBN 0-471-26370-2.
   Google Scholar
• Gitomer, D. H., & Yamamoto, K. (1991). Performance modeling that integrates latent trait and class theory. Journal of Educational Measurement, 28, 173–189.10.1111/jedm.1991.28.issue-2
   Web of Science ®Google Scholar
• Graf, E. A., & Arieli-Attali, M. (in press). Designing, developing, and validating assessments of complex thinking in mathematics for the middle grades. Theory into Practice Journal.
   Google Scholar
• Greenacre, M. (2007). Correspondence analysis in practice (2nd ed.). London: Chapman & Hall/CRC.
   Google Scholar
• Greenacre, M., & Blasius, J. (Eds.). (2006). Multiple correspondence analysis and related methods. London: Chapman & Hall/CRC.
   Google Scholar
• Guinot, C., Latreille, J., Malvy, D., Preziosi, P., Galan, P., Hercberg, S., & Tenenhaus, M. (2001). Use of multiple correspondence analysis and cluster analysis to study dietary behaviour: Food consumption questionnaire in the SU. VI. MAX. cohort. European Journal of Epidemiology, 17, 505–516.10.1023/A:1014586129113
   PubMed Web of Science ®Google Scholar
• Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2, 130–144.10.1016/j.edurev.2007.05.002
   Google Scholar
• Kaciak, E., & Louviere, J. (1990). Multiple correspondence analysis of multiple choice experiment data. Journal of Marketing Research, 27, 455–465.10.2307/3172630
   Web of Science ®Google Scholar
• Kalchman, M., & Koedinger, K. (2005). Teaching and learning functions. In M. S. Donovan & J. D. Bransford (Eds.), How students learn: History, mathematics and science in the classroom (pp. 351–396). Washington, DC: National Academies Press.
   Google Scholar
• Kalchman, M., Moss, J., & Case, R. (2001). Psychological models for development of mathematical understanding: Rational numbers and functions. In S. M. Carver & D. Klahr (Eds.), Cognition and instruction: Twenty-five years of progress (pp. 1–38). Mahwah, NJ: Erlbaum.
   Google Scholar
• Kieran, C. (1993). Functions, graphing, and technology: Integrating research on learning and instruction. In T. A. Romberg, E. Fennema, & T. P. Carpenter (Eds.), Integrating research on the graphical representation of functions (pp. 189–237). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Le Roux, B., & Rouanet, H. (2004). Geometric data analysis, from correspondence analysis to structured data analysis. Dordrecht: Kluwer.
   Google Scholar
• Lee, J., & Corter, J. E. (2011). Diagnosis of subtraction bugs using Bayesian networks. Applied Psychological Measurement, 35, 27–47.10.1177/0146621610377079
   Web of Science ®Google Scholar
• Leighton, J. P., & Gierl, M. J. (2007). Cognitive diagnostic assessment for education. New York, NY: Cambridge University Press.10.1017/CBO9780511611186
   Google Scholar
• Nenadic, O., & Greenacre, M. (2007). Correspondence analysis in R, with two- and three-dimensional graphics: The ca package. Journal of Statistical Software, 20, 1–13. Retrieved from http://www.jstatsoft.org/v20/i03/
   Web of Science ®Google Scholar
• Oehrtman, M. C., Carlson, M. P., & Thompson, P. W. (2008). Foundational reasoning abilities that promote coherence in students’ understandings of function. In M. P. Carlson & C. Rasmussen (Eds.), Making the connection (pp. 27–42). Washington, DC: Mathematical Association of America.10.5948/UPO9780883859759
   Google Scholar
• Pellegrino, J. W., Baxter, G. P., & Glaser, R. (1999). Addressing the ‘two disciplines’ problem: Linking theories of cognition and learning with assessment and instructional practice. Review of Research in Education, 24, 307–353.
   Web of Science ®Google Scholar
• Popham, W. J. (1997). What’s wrong and what’s right with rubrics. Educational Leadership, 55, 72–75.
   Web of Science ®Google Scholar
• Radatz, H. (1979). Error analysis in mathematics education. Journal for Research in Mathematics Education, 10, 163–172.10.2307/748804
   Google Scholar
• R Development Core Team. (2008). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.
   Google Scholar
• Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35, 435–448.
   Web of Science ®Google Scholar
• Rupp, A. A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.
   Google Scholar
• Samejima, F. (1969). Estimation of Latent Ability Using a Response Pattern of Graded Scores (Psychometric Monograph No. 17). Richmond, VA: Psychometric Society. Retrieved from http://www.psychometrika.org/journal/online/MN17.pdf
   Google Scholar
• Samejima, F. (1997). Graded response model. In W. J. van der Linden & R. K. Hambleton (Eds.), Handbook of modern item response theory (pp. 85–100). New York, NY: Springer.10.1007/978-1-4757-2691-6
   Google Scholar
• Sfard, A. (1992). Operational origins of mathematical objects and the quandary of reification: The case of function. In G. Harel & E. Dubinsky (Eds.), The concept of function: Aspects of epistemology and pedagogy. Mathematical Association of America Note (Vol. 25, pp. 59–84). Washington, DC: Mathematical Association of America.
   Google Scholar
• Shih, S. C., & Kuo, B. C. (2005). Using Bayesian networks for modeling students’ learning bugs and sub-skills. In D. Hutchison, T. Kanade, & J. Kittler (Eds.), Konwledge-based intelligent information and engineering systems (pp. 69–75). Berlin: Springer-Verlag.10.1007/11552413
   Google Scholar
• Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345–354.10.1111/jedm.1983.20.issue-4
   Web of Science ®Google Scholar
• Tatsuoka, K. K. (1985). A probabilistic model for diagnosing misconceptions by the pattern classification approach. Journal of Educational and Behavioral Statistics, 10, 55–73.
   Google Scholar
• van Rijn, P., Wise, M., Yoo, H., & Chung, S. (2014). Statistical report: Summary statistics, local dependence and differential item functioning in the CBAL mathematics 2012 study. Unpublished manuscript. Princeton, NJ: Educational Testing Service.
   Google Scholar
• Vinner, S., & Dreyfus, T. (1989). Images and definitions for the concept of function. Journal for Research in Mathematics Education, 20, 356–366.10.2307/749441
   Web of Science ®Google Scholar
• Wilson, M., & Sloane, K. (2000). From principles to practice: An embedded assessment system. Applied Measurement in Education, 13, 181–208.10.1207/S15324818AME1302_4
   Web of Science ®Google Scholar
• Yamamoto, K. (1989). Hybrid model of IRT and latent class models. Research Report No. 89–41. Princeton, NJ: Educational Testing Service.
   Google Scholar