Block-Wise Variable Selection for Clustering Via Latent States of Mixture Models

Abstract

Mixture modeling is a major paradigm for clustering in statistics. In this article, we develop a new block-wise variable selection method for clustering by exploiting the latent states of the hidden Markov model on variable blocks or the Gaussian mixture model. The variable blocks are formed by depth-first-search on a dendrogram created based on the mutual information between any pair of variables. It is demonstrated that the latent states of the variable blocks together with the mixture model parameters can represent the original data effectively and much more compactly. We thus cluster the data using the latent states and select variables according to the relationship between the states and the clusters. As true class labels are unknown in the unsupervised setting, we first generate more refined clusters, namely, semi-clusters, for variable selection and then determine the final clusters based on the dimension reduced data. Experiments on simulated and real data show that the new method is highly competitive in terms of clustering accuracy compared with several widely used methods. Supplementary materials for this article are available online.

KEYWORDS:

• Feature selection
• Semi-clusters
• Unsupervised learning
• Variable blocks

Supplementary Materials

Additional information and experimental results referenced in Sections 4–5 are available in the supplementary materials online. An R package implementing our method is available for download at https://github.com/seo-beomseok/HDclustVS.

Acknowledgment

We thank the reviewers and editors for detailed and constructive comments.

Funding

Additional information

Funding

The research is supported by the National Science Foundation under grant number DMS-2013905.