An Application of Multidimensional Vertical Scaling

ABSTRACT

Vertical scales are widely used in educational assessment as a basis for considering grade-to-grade changes in student performance. Typically, the underlying construct is assumed to be essentially unidimensional; however, if there is a change in the measured construct across grades, this assumption may be untenable. Developing a multidimensional vertical scale in these instances provides a potential solution to this problem. This paper uses empirical data from four parallel forms of a test designed to measure six foundational reading skills–administered to students in grades 6–9–to address issues in the development of a multidimensional vertical scale. The defensibility of the multidimensional structure, value-added subscores, and the stability of the scale are considered. Student growth based on unidimensional versus multidimensional estimates of ability is also presented with particular attention to implications associated with potential construct shift.

KEYWORDS:

• Multidimensional vertical scaling
• value-added subscores
• reading comprehension

Acknowledgement

The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305G04065 and Grant R305F100005 to the Educational Testing Service (ETS) as part of the Reading for Understanding Research Initiative. The opinions expressed are those of the authors and do not represent views of the institute or the U.S. Department of Education. We are extremely grateful to the Institute of Education Sciences and ETS for sponsoring and supporting this research.

Notes

^1. A third approach based on a weighted averaged of the total score and subscores is also presented in Haberman (2008). This approach is associated with augmented subscores.

^2. Weeks and Sabatini (2017) have conducted a preliminary examination of these profiles.