A Projection Approach to Local Regression with Variable-Dimension Covariates

Abstract

Incomplete covariate vectors are known to be problematic for estimation and inferences on model parameters, but their impact on prediction performance is less understood. We develop an imputation-free method that builds on a random partition model admitting variable-dimension covariates. Cluster-specific response models further incorporate covariates via linear predictors, facilitating estimation of smooth prediction surfaces with relatively few clusters. We exploit marginalization techniques of Gaussian kernels to analytically project response distributions according to any pattern of missing covariates, yielding a local regression with internally consistent uncertainty propagation that uses only one set of coefficients per cluster. Aggressive shrinkage of these coefficients regulates uncertainty due to missing covariates. The method allows in- and out-of-sample prediction for any missingness pattern, even if the pattern in a new subject’s incomplete covariate vector was not seen in the training data. We develop an MCMC algorithm for posterior sampling that improves a computationally expensive update for latent cluster allocation. Finally, we demonstrate the model’s effectiveness for nonlinear point and density prediction under various circumstances by comparing with other recent methods for regression of variable dimensions on synthetic and real data. Supplemental materials for this article are available online.

Keywords:

• Bayesian nonparametrics
• Clustering
• Dependent random partition models
• Indicator missing
• Pattern missing

Supplementary Materials

Appendix:(.pdf file) Illustration of marginalization behavior in VDLReg; fast screening tool for local linearity indicator, with simulations and data illustration; performance under different missingness mechanisms; cluster efficiency with locally linear prediction; role of the global shrinkage hyperparameter; full posterior; algorithm outline and computational complexity for allocation update; MCMC diagnostics; additional simulation results; an additional application; and additional details from the Old Faithful application.

VDLocalReg_examples:(zipped folder) R scripts that call the ProductPartitionModels Julia package to fit VDLReg models and recreate examples in the article.

Descriptions are contained in the README file.

Acknowledgments

The authors gratefully acknowledge helpful conversations with Peter Müller and Jyotishka Datta, as well as suggestions from an associate editor and two anonymous reviewers that significantly strengthened this work and its presentation. Figures were generated using ggplot2 (Wickham 2016).

Disclosure Statement

The authors report there are no competing interests to declare.

Additional information

Funding

The authors gratefully acknowledge partial funding from grant FONDECYT 1220017.