The Importance of Modeling Method Effects: Resolving the (Uni)Dimensionality of the Loneliness Questionnaire

Abstract

This study sought to resolve the dimensionality of the Loneliness Questionnaire (LQ; Asher, Hymel, & Renshaw, 1984) by applying recommended confirmatory factor analytic procedures that control for method effects (Brown, 2003). This study was needed given that inconsistent findings have been reported recently regarding the structure of this instrument (Bagner, Storch, & Roberti, 2004) and all models to date have not accounted for method effects due to the non-reversed-worded and reversed-worded items of this instrument. Using a large sample of youth in Grades 2 through 12 (N = 11,725), we compared the previously reported 1- and 2-factor models with a newly posited 1-factor model that incorporated correlated error terms to account for method effects. We found that the 1-factor model that included correlated error terms fit the data best, and that this factor structure evidenced measurement invariance across boys and girls in childhood, but not in adolescence. The meaning of the LQ indicators was also consistent for boys across development, but evidenced differences for girls in childhood versus adolescence. More generally, it was demonstrated that modeling method effects is vital to accurately understanding the dimensionality of loneliness when reversed-worded and non-reversed-worded items are used as indicators. The measurement and clinical implications of these findings are discussed.

Acknowledgments

Chad Ebesutani is now in the Department of Psychology at Yonsei University, Seoul, South Korea.

Notes

The WLSMV has some noted advantages over other estimators for categorical (ordinal) data, such as the weighted least squares (WLS) estimator. Specifically, WLSMV estimates parameters using robust standard errors, a mean- and variance-adjusted χ² test statistic, and also a diagonal weight matrix (which is not inverted during parameter estimation, thereby not requiring data matrices to be positive definite, unlike WLS, which does require positive definite matrices). WLSMV also does not require sample sizes as large as WLS to obtain reliable and unbiased parameter estimates (Flora & Curran, 2004).

Stated yet another way, factor loading invariance is similar to having parallel regression slopes in a regression framework; factor loading invariance across boys and girls would mean that a one-unit change in the latent construct of loneliness is associated with the same statistically significant changes in the observed factor indicators across boys and girls.

We divided our sample based on grade, as opposed to age, given that the anonymous nature of our IRB-approved data collection procedures prevented us from collecting information related to actual age. We included second graders in the child cohort given that children ages 5 and 6 years old have evidenced at least a basic understanding of the concept of loneliness (Cassidy & Asher, 1992).

Given the possibility that these significant χ² difference test results were due to the large sample size of this study, we reran these analyses based on a random subsample of 500 youths. Results were consistent, χ² _diff (9) = 37.12, p < .001, demonstrating that the improved model fit was related to significantly improved fit due to the one-factor model with correlated errors accounting for the noted method factor, as opposed to the large sample size.

To obtain adequate model fit, we added additional correlated error terms between LQ9 (“I feel alone”) and LQ21 (“I’m lonely”), LQ16 (“I get along with other kids”) and LQ20 (“I don't get along with other children”), LQ1 (“It's easy for me to make new friends at school”) and LQ6 (“It's hard for me to make friends”), and LQ9 (“I feel alone”) and LQ17 (“I feel left out of things”) based on the largest modification indexes that identified items with highly similar item content, thereby warranting correlated residuals between these highly synonymous items.

Byrne et al. (1989) first introduced the concept of partial measurement invariance to determine, for example, whether a subset of parameters is invariant across groups (i.e., testing whether some, but not all, parameters are equivalent across groups). Importantly, Byrne et al. noted that invariance evaluations can continue when an omnibus test of invariance is not supported if partial measurement invariance is supported. This is important given that full measurement invariance is unlikely to hold for all parameters in practice.

Again, to obtain adequate model fit with this female sample, we needed to add additional correlated error terms between LQ9 (“I feel alone”) and LQ21 (“I’m lonely”), LQ16 (“I get along with other kids”) and LQ20 (“I don't get along with other children”), LQ1 (“It's easy for me to make new friends at school”) and LQ6 (“It's hard for me to make friends”), and LQ9 (“I feel alone”) and LQ17 (“I feel left out of things”) based on the largest modification indexes that identified items with highly similar item content.