Modeling Local Item Dependence Due to Common Test Format With a Multidimensional Rasch Model

Abstract

Research shows that test method can exert a significant impact on test takers’ performance and thereby contaminate test scores. We argue that common test method can exert the same effect as common stimuli and violate the conditional independence assumption of item response theory models because, in general, subsets of items which have a shared feature are a source of response dependence (Marais & Andrich, 2008). In this study, we use the Rasch testlet model (Wang & Wilson, 2005a) to examine the effect of test method on violating the unidimensionality assumption of the Rasch model. Results show that test formats can introduce small to large construct-irrelevant variance, contaminate test scores, and lead to the violation of the conditional independence assumption. Our findings further suggest that the degree of construct-irrelevant variance exerted by test method could be a function of test format familiarity.

Keywords:

• conditional independence
• Rasch testlet model
• test method

Notes

Wu and Adams (2013) argued that to set realistic critical values for residual-based fit statistics, the distributional properties of the mean square statistics should be known. The null distribution of fit MNSQ values depends on several factors, the main factor being the sample size. Therefore, a single acceptable fit range cannot be used for all data sets and situations. When the data fit the Rasch model, infit and outfit MNSQ values are expected to be equal to unity. For unidimensional models, Wu and Adams (2013) suggested approximating the null distribution of outfit MNSQ values by deriving the asymptotic variance of the null distribution as . This approach offers an important advantage: rather than simulating numerous datasets, to specify “the null distribution” of the MNSQ for each dataset, MNSQ values should be presumed to follow “a scaled chi-square distribution with variance that is approximately equal to ” (Wu & Adams, 2013, p. 343). Therefore, 95% confidence intervals around outfit MNSQ values can be approximated by 1± 2, where N is the sample size. Wu and Adams compared their approach with the simulation studies of Wang and Chen (2005) and showed that when tests comprise 40 items, standard deviation for the null distribution of outfit MNSQs based on simulations and their simple method are virtually equal.