Beyond Prediction: A Framework for Inference With Variational Approximations in Mixture Models

Abstract

Variational inference is a popular method for estimating model parameters and conditional distributions in hierarchical and mixed models, which arise frequently in many settings in the health, social, and biological sciences. Variational inference in a frequentist context works by approximating intractable conditional distributions with a tractable family and optimizing the resulting lower bound on the log-likelihood. The variational objective function is typically less computationally intensive to optimize than the true likelihood, enabling scientists to fit rich models even with extremely large datasets. Despite widespread use, little is known about the general theoretical properties of estimators arising from variational approximations to the log-likelihood, which hinders their use in inferential statistics. In this article, we connect such estimators to profile M-estimation, which enables us to provide regularity conditions for consistency and asymptotic normality of variational estimators. Our theory also motivates three methodological improvements to variational inference: estimation of the asymptotic model-robust covariance matrix, a one-step correction that improves estimator efficiency, and an empirical assessment of consistency. We evaluate the proposed results using simulation studies and data on marijuana use from the National Longitudinal Study of Youth. Supplementary materials for this article are available online.

Keywords:

• Generalized linear mixed models
• Profile M-estimation

Acknowledgments

The authors thank two referees, an associate editor, and the editor for providing constructive feedback that helped them improve this article.

Additional information

Funding

The authors also gratefully acknowledge grant 62389-CS-YIP from the United States Army Research Office, grants SES-1559778 and DMS-1737673 from the National Science Foundation, a grant from the National Institute of Child Health and Human Development (NICHD): NIH funding: K01 HD078452.