Beyond correctness: development and validation of concept-based categorical scoring rubrics for diagnostic purposes

Abstract

Diagnostic assessment approaches intend to provide fine-grained reports of what students know and can do, focusing on their areas of strengths and weaknesses. However, current application of such diagnostic approaches is limited by the scoring method for item responses; important diagnostic information, such as type of errors and strategy use is lost or neglected when item-level responses are scored only for correctness. The purpose of this paper is to introduce a scoring method that classifies responses into concept-based nominal categories. In this paper, we describe the methodology of developing concept-based rubrics, and present an empirical examination of the construct validity of the scores in comparison to the scores based on conventional correctness rubrics. Utilising Multiple Correspondence Analysis, the results show that the categorical scores do not only reflect what the correctness scores capture, but also provide additional information that communicates aspects of student performance crucial for learning but neglected by conventional scoring methods.

Keywords:

• Constructed-response items
• scoring rubrics
• diagnostic assessment
• correspondence analysis
• educational measurement

Disclosure statement

No potential conflict of interest was reported by the authors.

Acknowledgements

The authors would like to thank David Budescu, Randy Bennett and Peter van Rijn for the helpful comments on earlier draft of the paper, as well as the anonymous reviewers who provided insightful suggestions. We would also like to thank Sarah Ohls for help with data collection, and the CBAL research initiative at ETS for partial support of this study.