Detecting Individual Differences in Change: Methods and Comparisons

Abstract

This study examined and compared various statistical methods for detecting individual differences in change. Considering 3 issues including test forms (specific vs. generalized), estimation procedures (constrained vs. unconstrained), and nonnormality, we evaluated 4 variance tests including the specific Wald variance test, the generalized Wald variance test, the specific likelihood ratio (LR) variance test, and the generalized LR variance test under both constrained and unconstrained estimation for both normal and nonnormal data. For the constrained estimation procedure, both the mixture distribution approach and the alpha correction approach were evaluated for their performance in dealing with the boundary problem. To deal with the nonnormality issue, we used the sandwich standard error (SE) estimator for the Wald tests and the Satorra–Bentler scaling correction for the LR tests. Simulation results revealed that testing a variance parameter and the associated covariances (generalized) had higher power than testing the variance solely (specific), unless the true covariances were zero. In addition, the variance tests under constrained estimation outperformed those under unconstrained estimation in terms of higher empirical power and better control of Type I error rates. Among all the studied tests, for both normal and nonnormal data, the robust generalized LR and Wald variance tests with the constrained estimation procedure were generally more powerful and had better Type I error rates for testing variance components than the other tests. Results from the comparisons between specific and generalized variance tests and between constrained and unconstrained estimation were discussed.

Keywords:

• constrained vs. unconstrained estimation
• growth curve modeling
• nonnormality
• specific vs. generalized tests
• variance tests

Acknowledgments

The authors are grateful for the helpful comments from Scott E. Maxwell, Ke-Hai Yuan, and the graduate students in the study group led by Scott E. Maxwell and Lijuan (Peggy) Wang.

Notes

¹ An alternative way to specify the LRS test is to use $M_0:\mu =\mu \left(\theta \right)$, $\mathbf{\Sigma }=\mathbf{\Sigma }\left(\theta \right)$ with ${\sigma }_S^2=0$ and freely estimated ${\sigma }_{IS}$ versus $M_1:\mu =\mu \left(\theta \right)$, $\mathbf{\Sigma }=\mathbf{\Sigma }\left(\theta \right)$ with freely estimated ${\sigma }_S^2$ and ${\sigma }_{IS}$. However, this approach is not conceptually logical because when the variance is zero, the covariance should be zero theoretically.

² Another form of the WS test is $\frac{{\hat{\sigma }}_S^2}{S.E.\left({\hat{\sigma }}_S^2\right)}$ where $S.E.\left({\hat{\sigma }}_S^2\right)$ is the square root of ${\widehat{\mathit{\text{C}}}}_{{\sigma }_S^2}$, which is compared with the standard normal distribution. The two forms of the WS test are essentially equivalent and lead to identical results. We use $\frac{{\hat{\sigma }}_S^4}{{\widehat{\mathit{\text{C}}}}_{{\sigma }_S^2}}$ here only for the purpose of easing the comparison with the other methods.

³ When data do not follow normal distributions, the variation of parameter estimates around the population values can be asymptotically described by the so-called sandwich covariance matrix. The sandwich covariance matrix has the form of ABA, where the outer layer, A, equals the inverse of the Hessian matrix and the inner part, B, is the correction matrix. A consistent estimator of the sandwich covariance matrix is able to correct the impact of distributional violations because it can balance information implied by the model (the outer layer) and the data (the inner part involves higher order moments of the data). Please refer to Yuan and Hayashi (2006) for detailed mathematical treatment.

⁴ The sampling distribution of the LR test statistic after Satorra–Bentler scaling correction is generally unknown for nonnormal data (e.g., Yuan, 2005).

⁵ The generated data were centered so that the theoretical mean equals 0.

⁶ The GCR is defined as $\mathrm{G}\mathrm{C}\mathrm{R}=\left({\sigma }_{Yt}^2-{\sigma }_{ϵ}^2\right)/{\sigma }_{Yt}^2$ (Hertzog et al., 2008). We scale GCR in this study at the first time point. Note that there are different definitions of reliability of longitudinal data. One commonly used reliability of change is defined as $\frac{{\sigma }_S^2}{{\sigma }_S^2+\frac{{\sigma }_{ϵ}^2}{\sum {\left(K_i-\bar{K}\right)}^2}}$ where K_i is the time indicator and $\bar{K}$ is the average time (Rast & Hofer, 2014; Willett, 1989). We did not use this reliability definition because when ${\sigma }_S^2=0$, we cannot manipulate the reliability. However, we computed the reliability of change for our simulation conditions based on Willett’s formula and the values nicely ranged from 0 to .96 across conditions.

⁷ The degree of freedom of the chi-square distribution depends on the test. If it is a specific variance test, the degree of freedom equals 1, whereas if it is a generalized variance test, it becomes 2.

⁸ For the sake of saving space, we have the results available on request.

⁹ For the robust WG tests, constrained estimation produced higher power than unconstrained estimation in 81.5% of the conditions that carried low to moderate GCRs. For the robust LRG tests, constrained estimation produced higher power than unconstrained estimation in 73.8% of the conditions that carried low to moderate GCRs.

¹⁰ Elimination and duplication matrices are used to perform the transformations between two vectors vec(M) and vech(M): vech (M) = Lvec (M); vec (M) = Dvech (M). Please refer to Magnus and Neudecker (1999) for formal definitions.

¹¹ t can be viewed as the mean or average of the transformed data, ${\left({\mathbf{y}}_i^{{\hspace{0.17em}}^{\prime }},{\mathrm{v}\mathrm{e}\mathrm{c}\mathrm{h}}^{{\hspace{0.17em}}^{\prime }}\left(\left({\mathbf{y}}_i-\bar{y}\right){\left({\mathbf{y}}_i-\bar{y}\right)}^{{\hspace{0.17em}}^{\prime }}\right)\right)}^{{\hspace{0.17em}}^{\prime }}$ with i = 1, … , N, and $\hat{\mathbf{\Gamma }}$ is the sample covariance matrix of the transformed data. By central limit theorem and Slutskys theorem, $\hat{\mathbf{\Gamma }}$ is a consistent estimator of $\mathbf{\Gamma }$ (see the proof on http://www3.nd.edu/˜lwang4/testsofvariances/).

¹² When the constrained estimation procedure was performed, ${\sigma }_S^2$ was constrained to be always nonnegative.