Epistatic Clustering: A Model-Based Approach for Identifying Links Between Clusters

Abstract

Most clustering methods assume that the data can be represented by mutually exclusive clusters, although this assumption may not be the case in practice. For example, in gene expression microarray studies, investigators have often found that a gene can play multiple functions in a cell and may, therefore, belong to more than one cluster simultaneously, and that gene clusters can be linked to each other in certain pathways. This article examines the effect of the above assumption on the likelihood of finding latent clusters using theoretical calculations and simulation studies, for which the epistatic structures were known in advance, and on real data analyses. To explore potential links between clusters, we introduce an epistatic mixture model which extends the Gaussian mixture by including epistatic terms. A generalized expectation-maximization (EM) algorithm is developed to compute the related maximum likelihood estimators. The Bayesian information criterion is then used to determine the order of the proposed model. A bootstrap test is proposed for testing whether the epistatic mixture model is a significantly better fit to the data than a standard mixture model in which each data point belongs to one cluster. The asymptotic properties of the proposed estimators are also investigated when the number of analysis units is large. The results demonstrate that the epistatic links between clusters do have a serious effect on the accuracy of clustering and that our epistatic approach can substantially reduce such an effect and improve the fit.

Keywords:

• Asymptotic property
• Bootstrap test
• Epistatic link
• Finite epistatic mixture
• Generalized EM algorithm
• Model-based epistatic clustering

SUPPLEMENTARY MATERIALS

The supplementary materials contain additional simulations, examples, and proofs from the main article.

We greatly appreciate the constructive and valuable comments from Co-Editor Jun Liu, an Associate Editor, two anonymous reviewers, and Professors Hal Stern and Qiwei Yao that have led to some significant improvements of the results and the presentation of the article.