Bayesian Approximate Kernel Regression With Variable Selection

ABSTRACT

Nonlinear kernel regression models are often used in statistics and machine learning because they are more accurate than linear models. Variable selection for kernel regression models is a challenge partly because, unlike the linear regression setting, there is no clear concept of an effect size for regression coefficients. In this article, we propose a novel framework that provides an effect size analog for each explanatory variable in Bayesian kernel regression models when the kernel is shift-invariant—for example, the Gaussian kernel. We use function analytic properties of shift-invariant reproducing kernel Hilbert spaces (RKHS) to define a linear vector space that: (i) captures nonlinear structure, and (ii) can be projected onto the original explanatory variables. This projection onto the original explanatory variables serves as an analog of effect sizes. The specific function analytic property we use is that shift-invariant kernel functions can be approximated via random Fourier bases. Based on the random Fourier expansion, we propose a computationally efficient class of Bayesian approximate kernel regression (BAKR) models for both nonlinear regression and binary classification for which one can compute an analog of effect sizes. We illustrate the utility of BAKR by examining two important problems in statistical genetics: genomic selection (i.e., phenotypic prediction) and association mapping (i.e., inference of significant variants or loci). State-of-the-art methods for genomic selection and association mapping are based on kernel regression and linear models, respectively. BAKR is the first method that is competitive in both settings. Supplementary materials for this article are available online.

KEYWORDS:

• Effect size
• Epistasis
• Kernel regression
• Variable selection
• Random Fourier features
• Statistical genetics

Supplementary Materials

The supplementary materials contain referenced figures and tables, details on model extensions, all relevant theoretical proofs, additional simulation results, as well as extended real data analyses and interpretations.

Acknowledgments

LC, KCW, XZ, and SM thank Mike West, Elizabeth R. Hauser, and Jenny Tung for useful conversations and suggestions.

Additional information

Funding

During the formulation of this work, LC was supported by the National Science Foundation Graduate Research Program under Grant No. DGF-1106401. KCW acknowledges the support of the NIH BIRCWH Program, a V Scholar Award from the V Foundation for Cancer Research, a Liz Tilberis Early Career Award from the Ovarian Cancer Research Fund, a Lloyd Trust Translational Research Award, and a Stewart Trust Fellowship. XZ acknowledges the support of NIH Grants R01HG009124, R01GM126553, and NSF Grant DMS1712933. SM acknowledges the support of grants NSF IIS-1546331, NSF DMS-1418261, NSF IIS-1320357, NSF DMS-1045153, and NSF DMS-1613261. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of any of the funders. This study makes use of data generated by the Wellcome Trust Case Control Consortium (WTCCC). A full list of the investigators who contributed to the generation of the data is available from www.wtccc.org.uk. Funding for the WTCCC project was provided by the Wellcome Trust under award 076113 and 085475.