Comparing Methods for Multivariate Nonparametric Regression

Abstract

The ever-growing number of high-dimensional, superlarge databases requires effective analysis techniques to mine interesting information from the data. Development of new-wave methodologies for high-dimensional nonparametric regression has exploded over the last decade in an effort to meet these analysis demands. This article reports on an extensive simulation experiment that compares the performance of ten different, commonly-used regression techniques: linear regression, stepwise linear regression, additive models (AM), projection pursuit regression (PPR), recursive partitioning regression (RPR), multivariate adaptive regression splines (MARS), alternating conditional expectations (ACE), additivity and variance stabilization (AVAS), locally weighted regression (LOESS), and neural networks. Each regression technique was used to analyze multiple datasets each having a unique embedded structure; the accuracy of each technique was determined by its ability to correctly identify the embedded structure averaged over all the datasets. Datasets used in the experiment were constructed so as to have particular properties which varied across the datasets in order to determine each technique's accuracy within various environments. The dataset properties which were varied include dimension of the data, the true dimension of the embedded structure, the sample size, the amount of noise, and the complexity of the embedded structure. Analyses of the results show that all of these properties affect the accuracy of each regression technique under investigation. A mapping from data characteristics to the most effective regression technique(s) is suggested.

Keywords:

• Multivariate nonparametric regression
• Projection pursuit regression
• MARS
• ACE
• LOESS
• Neural networks

Acknowledgments

This research was supported by the National Science Foundation under contract IRI-9224544 and by the Defense Advanced Research Projects Agency (DARPA) under contracts F30602-96-1-0349 and F30602-99-2-0537. The views and conclusions are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the National Science Foundation, the Defense Advanced Research Projects Agency, or the U.S. government.