Level-Specific Evaluation of Model Fit in Multilevel Structural Equation Modeling

Abstract

In multilevel structural equation modeling, the “standard” approach to evaluating the goodness of model fit has a potential limitation in detecting the lack of fit at the higher level. Level-specific model fit evaluation can address this limitation and is more informative in locating the source of lack of model fit. We proposed level-specific test statistics for the test of overall model fit, comparative fit index, and root mean squared error of approximation using partially saturated models, and we also considered another level-specific approach proposed by Yuan and Bentler (2007). A simulation study showed that the standard approach failed to detect the lack of fit at the group level. The fit indexes produced by the level-specific approaches both successfully detected the lack of model fit at each level. There were only minor differences in the performance of the 2 level-specific approaches.

Notes

¹Many applications of SEM consider covariance structure only. Mean structure is of interest only in particular cases such as multiple group analysis and longitudinal growth modeling. This study considers models only for covariance structure analysis.

²Any kind of sampling units with a nested structure can form multilevel data (e.g., students within classrooms, children within families, and repeated measurements within individuals). We use the term individual to represent a Level 1 unit and group to represent a Level 2 unit.

³In this article we do not make an explicit distinction between individual-level and group-level variables. In this formulation, the group-level variables can be considered as special cases in which the within-group components are zero.

⁴In this article, we use ML estimation to represent normal theory ML estimation.

⁵The likelihood ratio test of exact fit is commonly referred to by researchers and SEM packages as the chi-square test of exact fit. However, the test of exact fit follows a chi-square distribution only when its assumptions (described later in this section) are met.

⁶The independence model is a model in which (a) the variances are estimated with no constraint and (b) all the covariances are constrained to be zero. Typically, the independence model is nested within the hypothesized model. In some applications, the independence model is not nested within the hypothesized model. Use of an alternative null model is suggested in these cases (see Widaman & Thompson, 2003).

⁷The level-specific approach to evaluating model fit does not necessarily provide a solution to the third limitation of the standard approach. Yuan and Bentler's (2007) approach might be a potential solution because the parameter estimates, as well as model fit assessment, are obtained by separate analysis of between-group and within-group models. However, the properties of parameter estimates and comparability to those produced by multilevel SEM have not been studied.

⁸They also obtained the estimate of the asymptotic covariance matrix of the elements in the estimated saturated covariance matrix and constructed alternative test statistics other than ML (e.g., rescaled ML, residual-based ADF) using the asymptotic covariance matrix. However, test statistics and fit indexes based on alternative estimation procedures are beyond the scope of this article, which only considers the normal theory ML method.