Reporting Diagnostic Scores in Educational Testing: Temptations, Pitfalls, and Some Solutions

Abstract

Diagnostic scores are of increasing interest in educational testing due to their potential remedial and instructional benefit. Naturally, the number of educational tests that report diagnostic scores is on the rise, as are the number of research publications on such scores. This article provides a critical evaluation of diagnostic score reporting in educational testing. The existing methods for diagnostic score reporting are discussed. A recent method (Haberman, 2008a) that examines if diagnostic scores are worth reporting is reviewed. It is demonstrated, using results from operational and simulated data, that diagnostic scores have to be based on a sufficient number of items and have to be sufficiently distinct from each other to be worth reporting and that several operationally reported subscores are actually not worth reporting. Several recommendations are made for those interested to report diagnostic scores for educational tests.

Notes

¹A larger PRMSE is equivalent to a smaller mean squared error in estimating the true subscore and hence is desirable.

²Disattenuated correlation between two subscores is equal to the simple correlation between them divided by the square root of the product of the reliabilities of the two subscores.

³Note that although the table reports the averages to summarize a lot of information in a compendious manner, for some of these tests, the lengths, reliabilities, and correlations of the subscores are substantially unequal.

⁴Changing 0.01 to other small values such as 0.02 or 0.03 did not affect the conclusions much.

⁵In other words, there is an interaction between the two factors average length (or average reliability) and average disattenuated correlation.