Local Linear Forests

Abstract

Random forests are a powerful method for nonparametric regression, but are limited in their ability to fit smooth signals. Taking the perspective of random forests as an adaptive kernel method, we pair the forest kernel with a local linear regression adjustment to better capture smoothness. The resulting procedure, local linear forests, enables us to improve on asymptotic rates of convergence for random forests with smooth signals, and provides substantial gains in accuracy on both real and simulated data. We prove a central limit theorem valid under regularity conditions on the forest and smoothness constraints, and propose a computationally efficient construction for confidence intervals. Moving to a causal inference application, we discuss the merits of local regression adjustments for heterogeneous treatment effect estimation, and give an example on a dataset exploring the effect word choice has on attitudes to the social safety net. Last, we include simulation results on real and generated data. A software implementation is available in the R package grf. Supplementary materials for this article are available online.

Keywords:

• Asymptotic normality
• Heterogeneous treatment effect
• Smoothing and nonparametric regression

Supplementary Materials

Appendix: Supporting theoretical results and tables containing full simulation results.

R-package for local linear forests: The R-package grf contains code to run local linear forests and a folder with simulation replication code. Available via CRAN and Github.

Disclosure Statement

R.F. is currently at LinkedIn, and this article was included as part of her PhD dissertation at Stanford’s Statistics Department.

Acknowledgments

The authors would like to thank Guido Imbens, Art Owen, Evan Rosenman, and Steve Yadlowsky for useful comments and discussion.

Additional information

Funding

R.F. was supported by the DoD, Air Force Office of Scientific Research, National Defense Science and Engineering Graduate (NDSEG) Fellowship, 32 CFR 168a. The authors also gratefully acknowledge support by the Sloan Foundation, ONR grant N00014-17-1-2131, and NSF grant DMS-1916163. S.W. was supported by a Facebook Faculty Award.