Item Parameter Drift in a Time-Varying Predictor

ABSTRACT

The current simulation study examined the effects of Item Parameter Drift (IPD) occurring in a short scale on parameter estimates in multilevel models where scores from a scale were employed as a time-varying predictor to account for outcome scores. Five factors, including three decisions about IPD, were considered for simulation conditions. It was revealed that IPD occurring in a relatively shorter scale led to a substantial increase in the amount of relative bias in parameter estimates. The bias was more prominent in the estimates of level-2 time-varying predictors relative to those of level-1 time-varying predictors. Regarding the decisions about IPD, keeping items exhibiting IPD was more appropriate than removing them based on the results from relative bias of standard errors of estimates. Based on the findings, it can be concluded that removing items exhibiting IPD may lead to an increase of Type II errors due to the underestimation of parameter estimates and overestimation of standard errors. The applied example showed findings consistent with those in the simulation study.

Notes

¹ The TIMSS uses the term “context questionnaire scale” for the short questionnaires based on the fact that the purpose of the questionnaires is to examine the home, classroom, and school contexts where students learn mathematics and science. Also, items in each scale were developed to measure a single underlying latent construct, and the scales were constructed using the Rasch partial credit model. Martin et al. (2016) may be referred to for creating and interpreting the TIMSS context questionnaire scales.

² See author’s notes 1.

³ See author’s notes 2.

1. To decide whether a Likert-scale item exhibiting IPD should be removed, Li (2012) employed the Weighted Root Mean Squared Difference (WRMSD) between item response functions from two time points. According to Li, an item displaying a WRMSD value larger than 0.15 can be considered for the removal from a test.

in the equation is the expected scale score for each item (j) in a scale administered at time point 2 (t₂) and is the expected scale score for each item in a scale administered at time point 1 (t₁). k is the scale score on mj categories of each item for k = 0, 1, … mj -1, and n is the number of latent proficiency groups (e.g., n =801 for the theta range of -4 to +4 with a 0.01 interval). Wproficiency is the weight for each latent proficiency group, and the sum of the weights is 1. The weights are based on a standard normal distribution of latent proficiency.

2. IPD affects linking constants. For example, under the condition of the 9-item scale with 3 items exhibiting IPD, a student‘s scale score was -0.71 when the items exhibiting IPD were kept, and it was -0.63 when the items were removed only for linking. The school mean for the estimated scale scores were 0.003 and 0.083 in each condition, respectively. However, after centering the time-varying predictors (see pp. 8- 9 for the equations of the multilevel model employed in the current study), the centered values became identical (-0.713) in both conditions (keeping the items [-0.71 – 0.003] and removing only for linking [-0.63 – 0.083]). Because the linear transformation process based on different linking constants does not change the actual distance between student‘s estimated scale score from the school mean score, when centered values were used in multilevel models, both conditions (keeping the items exhibiting IPD and removing the items only for linking) produce the same parameter estimates, resulting in mathematically identical outcomes for relative bias. Related to centering in multilevel modeling, predictors are usually centered (e.g., grand mean or group mean centering) before the estimation of parameters to remove variability in each level and to avoid biased estimation (Hofmann and Gavin [1998] and Kreft, Leeuw, and Aiken [1995] may be referred to for the decisions about centering and their impacts on parameter estimation).