Modeling Between-Study Heterogeneity for Improved Replicability in Gene Signature Selection and Clinical Prediction

ABSTRACT

In the genomic era, the identification of gene signatures associated with disease is of significant interest. Such signatures are often used to predict clinical outcomes in new patients and aid clinical decision-making. However, recent studies have shown that gene signatures are often not replicable. This occurrence has practical implications regarding the generalizability and clinical applicability of such signatures. To improve replicability, we introduce a novel approach to select gene signatures from multiple datasets whose effects are consistently nonzero and account for between-study heterogeneity. We build our model upon some rank-based quantities, facilitating integration over different genomic datasets. A high-dimensional penalized generalized linear mixed model is used to select gene signatures and address data heterogeneity. We compare our method to some commonly used strategies that select gene signatures ignoring between-study heterogeneity. We provide asymptotic results justifying the performance of our method and demonstrate its advantage in the presence of heterogeneity through thorough simulation studies. Lastly, we motivate our method through a case study subtyping pancreatic cancer patients from four gene expression studies. Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

KEYWORDS:

• Generalized linear mixed models
• Microarray
• Penalized likelihood
• Prediction
• RNA-seq

Supplementary Materials

The online supplementary materials contain details on the alternative TSP screening procedure described in Section 7, supplementary figures pertaining to Section 7, and additional proofs pertaining to Section 5. R and Rcpp Code is also provided to perform the simulations and real data analysis.

Additional information

Funding

This work was supported by National Cancer Institute (under grant numbers R01-CA199064, P01-CA142538, and R01-CA193650) and National Institute of General Medical Sciences (under grant numbers R01-GM070335 and R01-GM105785).