Estimating New Quantities from Longitudinal Test Scores to Improve Forecasts of Future Performance

Abstract

Psychometric models for longitudinal test scores typically estimate quantities associated with single-administration tests, like ability at each time-point. However, models for longitudinal tests have not considered opportunities to estimate new quantities that are unavailable from single-administration tests. Specifically, we discuss dynamic measurement models – which combine aspects of longitudinal IRT, nonlinear growth models, and dynamic assessment – to directly estimate capacity, defined as the expected future score once the construct has fully developed. After discussing the history and connecting these areas into a single framework, we apply the model to verbal test scores from the Intergenerational Studies, which follow 494 people from 3 to 72 years old. The goal is to predict adult verbal scores (Age ≥ 34) from adolescent scores (Age ≤ 20). We held-out the adult data for prediction and compared predictions from traditional longitudinal IRT ability scores and proposed dynamic measurement capacity scores from models fit to the adolescent data. Results showed that the R² from capacity scores were 2.5 times larger than the R² from longitudinal IRT ability scores (43% vs. 16%), providing some evidence that exploring new quantities available from longitudinal testing could be worthwhile when an interest in testing is forecasting future performance.

Keywords:

• Multilevel modeling
• nonlinear growth models
• longitudinal data analysis
• longitudinal psychometrics
• dynamic measurement modeling

Notes

1 Vertical scaling is a method for scoring tests that places the scores of two or more tests onto a single scale when the tests are on the same topic but vary in difficulty. For example, math tests intended for 2nd grade and 3rd grade students might each have separate scale scores that range from 100 to 400. This makes it easy to compare within a grade but makes comparisons across grades difficult (e.g., a 300 in 2nd grade is not the same as a 300 in 3rd grade). The tests can be vertically scaled so that scores from both tests are on a single scale, perhaps ranging from 100 to 500 (e.g., a score of 300 means the same thing whether the student is in 2nd or 3rd grade). By vertically scaling, it is possible to compare an advanced 2nd grade student to students in 3rd grade or inspect how a single student improved between 2nd and 3rd grade.

2 Anonymized data, SAS code, and R code used for the analysis are included on the first author’s Open Science Framework page, https://osf.io/d49kb

3 These correlations are quite close for the models featuring other types of growth trajectories. The Michaelis-Menten DMM capacity correlations with Asymptotic ability were the largest and exceeded .70 in some analyses. See the supplemental material for complete results.

4 Using a Cholesky decomposition of the random effect covariance matrix, the logistic model with all four random effects did converge (Cholesky decompositions did not converge for any other trajectory with all random effects). The fit was slightly better (BIC = 5,537 vs. 5,650) and the marginal reliability of the capacities was a little higher (.80 vs. .74). Nonetheless, we proceeded with the version with two random effects (a) to better compare the different competing trajectories, which could only be fit with two random effects and (b) to better contextualize the ensuing sensitivity analysis because the Cholesky decomposition did not converge with different cut-offs, which required fitting models with only two random effects. The results were not appreciably different with a Cholesky decomposition and these results are reported in Appendix E of the supplemental material