A Comparison of Methodologies for Scaling Longitudinal Social-Emotional Survey Responses

ABSTRACT

Much of what we know about how children develop is based on survey data. In order to estimate growth across time and, thereby, better understand that development, short survey scales are typically administered at repeated timepoints. Before estimating growth, those repeated measures must be put onto the same scale. Yet, little research examines how scaling decisions affect comparisons of growth derived from survey item responses. In this study, we use a sample of 174,669 students in grades 7 through 12 who took the same self-efficacy and social awareness surveys for four years. We use those survey item responses to construct scales using different approaches, then compare the resultant scores to see how inferences about changes over time during adolescence might shift dependent on scaling. While we find that conclusions about average trends are quite consistent by scaling approach, specific quantifications of change like effect sizes can differ by scaling method.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/10627197.2023.2213432

Notes

¹ This contention is based on the fact that standardized testing has been at the heart of No Child Left Behind and the Every Student Succeeds Act, whereas student SE and psychological outcomes have not been the primary focus.

² These analyses were conducted testing for longitudinal invariance separately for two different cohorts followed over time such that each timepoint includes only one grade.

³ Because we are only reporting means over time, we do not have an estimation option like Full Information Maximum Likelihood (FIML) to address missingness in the data. However, results are consistent when using an intact cohort versus what we report in the study, namely including anyone with at least one available score.

⁴ We also produced these ESs in ways that account for differing sample sizes in the pooled SD. However, substantive conclusions remained unchanged.

⁵ Means and ESs that include standard errors can be found in Appendix D. Except in cases where means/ESs are themselves extremely close to zero, virtually all of the estimates are distinguishable from zero.