Visualizable and interpretable regression models with good prediction power

Abstract

Many methods can fit models with a higher prediction accuracy, on average, than the least squares linear regression technique. But the models, including linear regression, are typically impossible to interpret or visualize. We describe a tree-structured method that fits a simple but nontrivial model to each partition of the variable space. This ensures that each piece of the fitted regression function can be visualized with a graph or a contour plot. For maximum interpretability, our models are constructed with negligible variable selection bias and the tree structures are much more compact than piecewise-constant regression trees. We demonstrate, by means of a large empirical study involving 27 methods, that the average prediction accuracy of our models is almost as high as that of the most accurate “black-box” methods from the statistics and machine learning literature.

Keywords:

• Machine learning
• piecewise linear
• regression tree
• selection bias

Acknowledgements

The authors are grateful to two referees for their comments. Kim's research was partially supported by grant R01-2005-000-11057-0 from the Basic Research Program of the Korea Science and Engineering Foundation. Loh's research was partially supported by the National Science Foundation under grant DMS-0402470 and by the US Army Research Laboratory and the US Army Research Office under grant W911NF-05-1-0047. Shih's research was partially supported by a grant from the National Science Council of Taiwan.