Fast Bayesian Factor Analysis via Automatic Rotations to Sparsity

ABSTRACT

Rotational post hoc transformations have traditionally played a key role in enhancing the interpretability of factor analysis. Regularization methods also serve to achieve this goal by prioritizing sparse loading matrices. In this work, we bridge these two paradigms with a unifying Bayesian framework. Our approach deploys intermediate factor rotations throughout the learning process, greatly enhancing the effectiveness of sparsity inducing priors. These automatic rotations to sparsity are embedded within a PXL-EM algorithm, a Bayesian variant of parameter-expanded EM for posterior mode detection. By iterating between soft-thresholding of small factor loadings and transformations of the factor basis, we obtain (a) dramatic accelerations, (b) robustness against poor initializations, and (c) better oriented sparse solutions. To avoid the prespecification of the factor cardinality, we extend the loading matrix to have infinitely many columns with the Indian buffet process (IBP) prior. The factor dimensionality is learned from the posterior, which is shown to concentrate on sparse matrices. Our deployment of PXL-EM performs a dynamic posterior exploration, outputting a solution path indexed by a sequence of spike-and-slab priors. For accurate recovery of the factor loadings, we deploy the spike-and-slab LASSO prior, a two-component refinement of the Laplace prior. A companion criterion, motivated as an integral lower bound, is provided to effectively select the best recovery. The potential of the proposed procedure is demonstrated on both simulated and real high-dimensional data, which would render posterior simulation impractical. Supplementary materials for this article are available online.

KEYWORDS:

• EM algorithm
• Factor rotations
• Parameter expansion
• Sparsity
• Spike-and-Slab LASSO

Supplementary Materials

The supplementary materials contain further developments and proofs.

Acknowledgments

The authors would like to thank the Associate Editor and the anonymous referees for their insightful comments and useful suggestions. We would also like to thank Art Owen for kindly providing the AGEMAP dataset.

Funding

The work was supported by NSF grant DMS-1406563 and AHRQ Grant R21-HS021854.

Notes

1 Each equivalence class $\left[\Gamma \right]$ contains all matrices $\Gamma$ with the same left-ordered form, obtained by ordering the columns from left to right by their binary numbers.

2 Assumed to be left-ordered.

3 Under (D), $s_n^{\text{min}}$ converges to 1 a.s. as n → ∞, when ${\mathbf{B}}_{0n}$ are iid with mean 0 and variance 1 and G_n > K_0n.

4 Taking a submatrix S₁ of a symmetric positive-semi-definite matrix S₂, the smallest eigenvalue satisfies λ₁(S₁) ⩾ λ₁(S₂).

5 All 10 solutions recovered by dynamic posterior exploration are reported in Section F of the supplementary material.