Efficient Matrix Sampling Instruments for Correlated Latent Traits: Examples from the National Assessment of Educational Progress

Abstract

We study the efficiency of administering different subsets of a large collection of items in a sample survey to different people to estimate the distribution of several correlated traits. The designs are motivated by examples from the National Assessment of Educational Progress, an ongoing survey of U.S. students in the fourth, eighth, and twelfth grades. In this survey the traits represent proficiency in different subjects. For example, the Mathematics assessment estimates proficiency with Numbers and Operations, Measurement, and several other traits. Time constraints and concerns about student motivation limit the number of items that can be administered to a small subset of the items defining each trait. We find that effective matrix designs assign some items to measure each trait, even if the resulting number of items assigned to each trait must be small. Efficient allocations of items are ones that produce measurement error variances for a given trait that are similar for each sampled student. These allocations can be substantially better than designs that split the sample and measure traits on different subsets of students. Our results are consistent with recent research on efficient survey instrument design, but lead to substantially different recommendations due to the errors in the measurement of the traits.

Key Words:

• Efficient design
• Item response models
• Item sampling
• Measurement error
• NAEP