Using Information Criteria Under Missing Data: Full Information Maximum Likelihood Versus Two-Stage Estimation

ABSTRACT

The full information maximum likelihood (FIML) and the two-stage (TS) procedure are two popular likelihood-based approaches to SEM model estimation with missing data. After model estimation, one often needs to choose the best model from a group of candidate models. A popular type of model selection tools is information criteria. Because FIML and TS both give consistent model parameter estimates, it is tempting to assume both FIML-based and TS-based information criteria are appropriate and useful. However, in this paper we show FIML and TS do not both give appropriate information criteria, and model selection results may be different from those under complete data, even in large samples. We first analytically study the implications of missing (completely) at random data for information criteria. Next, we conduct simulations to verify our theoretical proof and understand the empirical performance of information criteria. Our conclusions apply to AIC, BIC, and their variants.

KEYWORDS:

• Information criteria
• missing data
• full information maximum likelihood
• two-stage estimation

Notes

¹ The main conclusion in Huang (2017) about AIC and BIC has three parts. First, if Models A and B have different population fit function values, then in large samples both AIC and BIC will select the model with smaller population fit function value, regardless of model parsimony. Second, if Models A and B have the same population fit function value and A is nested within B, then in large samples BIC (but not AIC) will always favor Model A. Third, if Models A and B have the same population fit function value but they are nonnested, then neither BIC nor AIC can eventually settle on a single candidate model.

² When comparing BIC and AIC, it is tempting to calculate their difference directly using $BIC-AIC=[ln(n)-2]q$ and accordingly argue that BIC and AIC will have different performance in large samples (as their difference seems to approach infinity). The problem with this method is that ${lim}_{n\to \mathrm{\infty }}BIC$ and ${lim}_{n\to \mathrm{\infty }}AIC$ do not exist in the first place (because their main component ${\sum }_{i=1}^n{u}_i$ will approach infinity in the population), and thus ${lim}_{n\to \mathrm{\infty }}(BIC-AIC)\ne ({lim}_{n\to \mathrm{\infty }}BIC)-({lim}_{n\to \mathrm{\infty }}AIC)$. The knowledge of $(BIC-AIC)$ in large samples cannot be used to study how AIC in large samples differs from BIC in large samples.

³ The missing data mechanism in D3 and D4 is a relatively strong type of MAR, in that all the individuals who satisfy a condition (say $X_1>z_{0.75}$) will lose the score on the corresponding variable ($X_2$ in this example). That is, $P(X_2\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g})=1$ if $X_1>z_{0.75}$, and $P(X_2\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g})=0$ if $X_1\le z_{0.75}$. It is also possible to have a smoother function for $P(X_2\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g})$. For example, if $P(X_2\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g})=exp(X_1)/[1+exp(X_1)]$, then $X_2$ can be missing given any $X_1$ value, and $P(X_2\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g})$ can take on any value between 0 and 1. One advantage of the missing data mechanism we adopted for our simulation study is that the amount of observed data is relatively stable across replications (e.g., in every random sample there is about 25% missing observations on $X_2$).