A new kernel regression approach for robustified L₂ boosting

Abstract

We investigate L₂ boosting in the context of kernel regression. Kernel smoothers, in general, lack appealing traits like symmetry and positive definiteness, which are critical not only for understanding theoretical aspects but also for achieving good practical performance. We consider a projection-based smoother (Huang and Chen 2008) that is symmetric, positive-definite, and shrinking. Theoretical results based on the orthonormal decomposition of the smoother reveal additional insights into the boosting algorithm. In our asymptotic framework, we may replace the full-rank smoother with a low-rank approximation. We demonstrate that the smoother’s low-rank (d_n) is bounded above by O(h⁻¹), where h is the bandwidth. Our numerical findings show that, in terms of prediction accuracy, low-rank smoothers may outperform full-rank smoothers. Furthermore, we show that the boosting estimator with a low-rank smoother achieves the optimal convergence rate. Finally, to improve the performance of the boosting algorithm in the presence of outliers, we propose a novel robustified boosting algorithm that can be used with any smoother discussed in the study. We investigate the numerical performance of the proposed approaches using simulations and a real application.

Keywords:

• Eigenvalues
• reduced-rank
• pseudo data
• Huber loss

Acknowledgments

The author would like to thank Prof. Li-Shan Huang for introducing projection smoothers to him and Dr. Abhijit Mandal for suggesting an important reference on robust smoothing.

Disclosure statement

No potential conflict of interest was reported by the author(s).