Bayesian Dimensionality Reduction With PCA Using Penalized Semi-Integrated Likelihood

ABSTRACT

We discuss the problem of estimating the number of principal components in principal components analysis (PCA). Despite the importance of the problem and the multitude of solutions proposed in literature, it comes as a surprise that there does not exist a coherent asymptotic framework, which would justify different approaches depending on the actual size of the dataset. In this article, we address this issue by presenting an approximate Bayesian approach based on Laplace approximation and introducing a general method of developing criteria for model selection, called PEnalized SEmi-integrated Likelihood (PESEL). Our general framework encompasses a variety of existing approaches based on probabilistic models, like the Bayesian Information Criterion for Probabilistic PCA (PPCA), and enables the construction of new criteria, depending on the size of the dataset at hand and additional prior information. Specifically, we apply PESEL to derive two new criteria for datasets where the number of variables substantially exceeds the number of observations, which is out of the scope of currently existing approaches. We also report results of extensive simulation studies and real data analysis, which illustrate the desirable properties of our proposed criteria as compared to state-of-the-art methods and very recent proposals. Specifically, these simulations show that PESEL-based criteria can be quite robust against deviations from the assumptions of a probabilistic model. Selected PESEL-based criteria for the estimation of the number of principal components are implemented in the R package pesel, which is available on github (https://github.com/psobczyk/pesel). Supplementary material for this article, with additional simulation results, is available online. The code to reproduce all simulations is available at https://github.com/psobczyk/pesel_simulations.

KEYWORDS:

• Bayesian model selection
• Dimension estimation
• Laplace approximation
• Principal component analysis

Acknowledgments

The authors thank the associate editor and anonymous reviewers for comments and suggestions that helped them to improve the presentation. The authors also thank professor Jean-Michel Marin from the University of Montpellier for assistance and comments that improved the article, professor Sandrine Lagarrigue for providing the mice data, and to professor David Ramsey and Artur Bogdan for a careful reading and many helpful suggestions.

Funding

PS and MB were supported by the European Union’s 7th Framework Programme for research, technological development and demonstration under Grant Agreement no. 602552, cofinanced by the Polish Ministry of Science and Higher Education under Grant Agreement 2932/7.PR/2013/2. Calculations were carried out in the Wroclaw Centre for Networking and Supercomputing (http://www.wcss.pl), grant no. 347.