An Evaluation of Information Criteria Use for Correct Cross-Classified Random Effects Model Selection

Abstract

The authors assessed correct model identification rates of Akaike's information criterion (AIC), corrected criterion (AICC), consistent AIC (CAIC), Hannon and Quinn's information criterion (HQIC), and Bayesian information criterion (BIC) for selecting among cross-classified random effects models. Performance of default values for the 5 indices used by SAS PROC MIXED for estimating a 2-level cross-classified random effects model were compared with modifications to the sample size used in the AICC, CAIC, HQIC, and BIC formulations. The sample sizes explored included the number of level 1 units (N), the average number of classification units (m), and the number of nonempty classification cells (c). The authors also assessed performance of the χ² _diff test for testing the difference in fit between 2 nested cross-classified random effects models. The χ² _diff exhibited a slightly inflated Type I error rate with high power. The modified information criteria performed better than did the default values. Pairing of N with the HQIC, BIC, and CAIC and of m with the AICC worked best. Results and suggestions for future research are discussed.

Keywords:

• cross-classified model
• information criteria
• model fit indices
• multilevel model

Acknowledgments

A previous version of this article was presented at the 2010 annual meeting of the American Educational Research Association in Denver, Colorado.

Notes

This value corresponds to the number of “effective subjects” as appears under the DIMENSIONS table in the PROC MIXED output when two (crossed) random effects are specified in the SUBJECT = statement in two RANDOM statements.

Use of “one” as N* in the HQIC formula (see Equation 6) results in an incalculable second term and zeroes out the second term in the BIC formula (Equation 4). Thus, SAS PROC MIXED sets HQIC and BIC to −2LL when estimating CCREMs.